Keeping secrets safe with YQL Storage

Just FYI, this is not a discussion on self-help to keep you from divulging your deep and dark secrets in real life. It’s about YQL and the various web secrets such as API keys, Consumer Keys and other key-secrets for personal use which you’d rather not give out to anyone visiting your web applications.

With that disclaimer, let me jump right into the scenario that I’m talking about. There are quite a few YQL Open Data Tables such as Netflix Catalog, NY Times Article search, Amazon Product Advertising table etc which require a user to register and enter an API key and/or Secret into the query. Issuing a query in YQL Console for testing purposes seems great. You can see the results and ensure that the query is working as you desire before adding it into your webpage/webapp.

This YQL query can then be run, either by issuing a HTTP GET request call from your website or by directly embedding it in your webpage as part of the Javascript. In the former case, since you own the website, there is no need to hide the Consumer-Key and Secret. In the later case, where the query is directly embedded in the client-side JS code running in the browser, you run the risk of exposing your ConsumerKey and Secret which can then to be used to run various other queries on your behalf. In this post, I show how one can hide the secrets from the user and also lock down the secrets to a single table.

Brief Introduction to ‘SET‘

‘SET’ statements in YQL let you decouple the constant keys from the varying keys in the where clause of the YQL statement. For example if I wanted only the PDF search results for a given search term, I would issue:

select * from search.web where query='yql' and type='pdf'

Try in console

Since the type=’pdf’ expression will be a constant key in all of my queries, I can decouple it by issuing the following query

SET type='pdf' on search.web;
select * from search.web where query='yql'

Try in console

In essence, the YQL compiler applies all the ‘SET’ clauses for a particular table to each compatible branch of the YQL statement and then executes the query.

Here is another example which shows the Amazon product listing api wrapped in YQL that requires your AWSAccessKeyId and the secret to be part of the query. To test this query I would use the entire query:

use 'http://nagiworld.net/yqldefs/yql/amazon.ecs.xml' as amazon.prodlist;
set AWSAccessKeyId='YOUR_KEY' on amazon.prodlist;
set secret='YOUR_SECRET' on amazon.prodlist;
set AssociateTag='YOUR_ASSOCIATE_TAG' on amazon.prodlist;

select * from amazon.prodlist where Title = 'panasonic' and SearchIndex="Electronics" and ResponseGroup='Images,ItemAttributes,SalesRank' and ItemLinks.ItemLink.Description='All Offers'

Storing your secrets

YQL provides a way to store data in Yahoo’s own Sherpa. For a deep dive into storing data with YQL you can checkout the Hosted Storage docs

Building on the ‘USE’ and ‘SET’ statements described earlier, we can easily decouple the query into two components. The first component has all the ‘SET’ statements whereas the second component has your query. Now to hide all the secret bits in YQL, all I have to do is insert the ‘SET’ statements into yahoo.storage with the following query

insert into yql.storage.admin (value) values ("use 'http://nagiworld.net/yqldefs/yql/amazon.ecs.xml' as amazon.prodlist;
set AWSAccessKeyId='YOUR_KEY' on amazon.prodlist;
set secret='YOUR_SECRET' on amazon.prodlist;
set AssociateTag='YOUR_ASSOCIATE_TAG' on amazon.prodlist;")

The response contains the various keys such as execute, select and update.

<results>
<inserted>
<execute>store://h0L6f1ecl32llKITPZEbOg </execute>
<select>store://xGnPqWdslS..sqvoOYObn </select>
<update>store://tiJkslt0pE..QEecsQyQn5 </update>
</inserted>
</results>

Each element described in the response has specific properties.

Given that we have the execute key, we can now provide the execute key to the query parameter called ‘env’. This tells the YQL Servers to pull the data from the store and prepend it to your query before actually running it. So in essence YQL does a read through execution of the query.

http://developer.yahoo.com/yql/console?env=store://h0L6f1ecl32llKITPZEbOg

Once you pass in the env, you can issue the following query:

select * from amazon.prodlist where Title = 'panasonic' and SearchIndex="Electronics" and ResponseGroup='Images,ItemAttributes,SalesRank' and ItemLinks.ItemLink.Description='All Offers'

Try it in Console

As you can be seen, the above query does not expose any of the your secrets specifically the amazon id, secret and your associate tag. None of the diagnostics or the debug information leak your URLs which at this point could contain the various keys that you’d like to keep secret.

Selecting an execute key does not show the secrets you put into the query

select * from yql.storage where name='store://h0L6f1ecl32llKITPZEbOg'

Try in Console

As expected this returns the following and thus Mission Accomplished! Secrets safe

<results>
<error>No select permission or doesn't exist</error>
</results>

Picture licensed from nostri-imago under Creative Commons

Related Posts

No related posts.