Keeping secrets safe with YQL Storage

DSCN1583

Just FYI, this is not a discussion on self-help to keep you from divulging your deep and dark secrets in real life. It’s about YQL and the various web secrets such as API keys, Consumer Keys and other key-secrets for personal use which you’d rather not give out to anyone visiting your web applications.

With that disclaimer, let me jump right into the scenario that I’m talking about. There are quite a few YQL Open Data Tables such as Netflix Catalog, NY Times Article search, Amazon Product Advertising table etc which require a user to register and enter an API key and/or Secret into the query. Issuing a query in YQL Console for testing purposes seems great. You can see the results and ensure that the query is working as you desire before adding it into your webpage/webapp.

This YQL query can then be run, either by issuing a HTTP GET request call from your website or by directly embedding it in your webpage as part of the Javascript. In the former case, since you own the website, there is no need to hide the Consumer-Key and Secret. In the later case, where the query is directly embedded in the client-side JS code running in the browser, you run the risk of exposing your ConsumerKey and Secret which can then to be used to run various other queries on your behalf. In this post, I show how one can hide the secrets from the user and also lock down the secrets to a single table.

Brief Introduction to ‘SET

‘SET’ statements in YQL let you decouple the constant keys from the varying keys in the where clause of the YQL statement. For example if I wanted only the PDF search results for a given search term, I would issue:

select * from search.web where query='yql' and type='pdf'

Try in console

Since the type=’pdf’ expression will be a constant key in all of my queries, I can decouple it by issuing the following query

SET type='pdf' on search.web;
select * from search.web where query='yql'

Try in console

In essence, the YQL compiler applies all the ‘SET’ clauses for a particular table to each compatible branch of the YQL statement and then executes the query.

Here is another example which shows the Amazon product listing api wrapped in YQL that requires your AWSAccessKeyId and the secret to be part of the query. To test this query I would use the entire query:

use 'http://nagiworld.net/yqldefs/yql/amazon.ecs.xml' as amazon.prodlist;
set AWSAccessKeyId='YOUR_KEY' on amazon.prodlist;
set secret='YOUR_SECRET' on amazon.prodlist;
set AssociateTag='YOUR_ASSOCIATE_TAG' on amazon.prodlist;

select * from amazon.prodlist where Title = 'panasonic' and SearchIndex="Electronics" and ResponseGroup='Images,ItemAttributes,SalesRank' and ItemLinks.ItemLink.Description='All Offers'

Storing your secrets

YQL provides a way to store data in Yahoo’s own Sherpa. For a deep dive into storing data with YQL you can checkout the Hosted Storage docs

Building on the ‘USE’ and ‘SET’ statements described earlier, we can easily decouple the query into two components. The first component has all the ‘SET’ statements whereas the second component has your query. Now to hide all the secret bits in YQL, all I have to do is insert the ‘SET’ statements into yahoo.storage with the following query

insert into yql.storage.admin (value) values ("use 'http://nagiworld.net/yqldefs/yql/amazon.ecs.xml' as amazon.prodlist;
set AWSAccessKeyId='YOUR_KEY' on amazon.prodlist;
set secret='YOUR_SECRET' on amazon.prodlist;
set AssociateTag='YOUR_ASSOCIATE_TAG' on amazon.prodlist;")

The response contains the various keys such as execute, select and update.

<results>
<inserted>
<execute>store://h0L6f1ecl32llKITPZEbOg </execute>
<select>store://xGnPqWdslS..sqvoOYObn </select>
<update>store://tiJkslt0pE..QEecsQyQn5 </update>
</inserted>
</results>

Each element described in the response has specific properties.

  • The execute key can only be used to execute a query. This implies that exposing an execute key does not expose the data stored within the entry.
  • The select key should be kept private since it can be used on the yql.storage table to view the contents of this particular entry.
  • The update key lets you update the contents that are seen via the select key.

Given that we have the execute key, we can now provide the execute key to the query parameter called ‘env’. This tells the YQL Servers to pull the data from the store and prepend it to your query before actually running it. So in essence YQL does a read through execution of the query.

http://developer.yahoo.com/yql/console?env=store://h0L6f1ecl32llKITPZEbOg

Once you pass in the env, you can issue the following query:

select * from amazon.prodlist where Title = 'panasonic' and SearchIndex="Electronics" and ResponseGroup='Images,ItemAttributes,SalesRank' and ItemLinks.ItemLink.Description='All Offers'

Try it in Console

As you can be seen, the above query does not expose any of the your secrets specifically the amazon id, secret and your associate tag. None of the diagnostics or the debug information leak your URLs which at this point could contain the various keys that you’d like to keep secret.

Selecting an execute key does not show the secrets you put into the query

select * from yql.storage where name='store://h0L6f1ecl32llKITPZEbOg'

Try in Console

As expected this returns the following and thus Mission Accomplished! Secrets safe

<results>
<error>No select permission or doesn't exist</error>
</results>

Picture licensed from nostri-imago under Creative Commons

Share
  • del.icio.us
  • Facebook
  • Digg
  • description

No related posts.

  • violetnataly

    A very interesting article. For sure there is information that you don't want anybody to know. Thanks for the tips!
    And the foto is super. It reminded me of the film The Bank Robbery 2(it is easy to find it at rapidshare SE http://rapidpedia.com ). I believe nobody will have a desire to break such a door.

  • ongamesite

    Hi every friends,

    Did you like play online flash games and nice VIDEO ^_^. OK ! If you are interesting with it.

    Just do a click http://www.ongamesite.com. Good luck.

    internet games,computer games,windows games,multiplayer games,fashion games,interactive games,top games,web games
    Best Luck Nice World!

  • RosanneJQuinones

    Niagara Falls is a well known site that is a popular tourist attraction. In addition to this, Niagara Falls is known for the hydroelectric power it provides, and the spot was chosen as potential source of power by Nikola first niagara bank Tesla, the famous inventor. A memorial has also been built for him at the site. The three waterfalls are shared by three cities, and these are Niagara Falls, Ontario, and the dual cities of Niagara Falls, New York. It is believed that Niagara Falls formed during a period that is called the Wisconsin glaciation, a period that ended over 9,000 years ago. The Wisconsin glaciation was a large glacier that connected the area with Canada.

  • ViolaBrown

    Thanks for the information. Nothing is gonna break this lock. It is Unbreakable (like in the song by Michael Jackson which can be downloaded from http://www.mp3hounddog.com )!

  • http://www.timacheson.com/ Tim Acheson

    Hi Nagesh, thanks, this is a useful tutorial, but it doesn't solve the biggest problem I have with needing to hide my API keys.

    I have a YQL query which returns JSON from a web service URL:

    SELECT
        *
    FROM
        json
    WHERE
        url="http://bungie.net/videos/{BungieAPIKey}/{user}/{page}"

    How can I use a YQL query to return the JSON from this URL, while making my API Key secret or less obvious? Is it impossible to do?

    I want to keep my API key “{BungieAPIKey}” secret, or at least make it a parameter “@BungieAPIKey” so I can hide it slightly within a query alias.

    I posted this question on the YDN/YQL forum, but I rarely get much of a response on the YDN forums.

  • http://nagiworld.net Nagesh Susarla

    Hi Tim,

    One way to accomplish this is by using the uritemplate table which lets you create arbitrary URLs from templates like the one you mention.

    example:

    select url from uritemplate where template='http://bungie.net/videos/{BungieAPIKey}/{user}/{page}' and BungieAPIKey='key' and user='foo' and page='bar'

    Once you have the URL you can use the JSON table to curl the URL

    select * from json where url in (select url from uritemplate where template='http://bungie.net/videos/{BungieAPIKey}/{user}/{page}' and BungieAPIKey='key' and user='foo' and page='bar')

    Now once you have this working, lets look at what needs to be done to make the API key a secret. To do that you could have to create a store table entry which contains the following

    “SET BungieAPIKey='secret' on uritemplate;”

    Insert this into the yql.storage as described in the docs or the link and use the execute key that is returned to run your query.

    http://developer.yahoo.com/yql/console?env=YOUR_STORE_ENTRY

    The query doesn't need the api key anymore so you can run

    select * from json where url in (select url from uritemplate where template='http://bungie.net/videos/{BungieAPIKey}/{user}/{page}' and user='foo' and page='bar')

  • http://www.timacheson.com/ Tim Acheson

    Thanks, that's fantastic!

    Is it enough to hide my API within a query alias? Or is YQL Storage the only way? If the latter is the case, can you explain why? It seems to me that, when the API key is in a query alias, other people can't see it. (E.g. if I set a hard-coded API key value in a query alias, with parameters for the other values, and then use the URL for that query alias. Then, the API key is not revealed in the URL.)

  • http://nagiworld.net Nagesh Susarla

    Hi Tim,

    That's indeed a very good question. Query aliases are only shortcuts. Besides that they do not prevent the user from seeing the actual URLs that you used in your query. The user of the alias can add a '&diagnostics=true' and see the entire URL that was used to make the call to the webservice.

    Let me take the following example:

    If I declared a query alias, such as http://query.yahooapis.com/v1/public/yql/nagesh/test2?foo=topstories which takes in a query parameter named 'foo' and the rest of the URL is hardcoded in my alias, I can simply append &diagnostics=true to see the following which gives away the URL that I used in this case.

    <url execution-time=”10″ >http://rss.news.yahoo.com/rss/topstories&lt;/url&gt;;

    Using stored secrets going one step ahead by obsuring all URLs that appear in the diagnostics to look something like http://domain... thus ensuring that any secret gives are not divulged to the user.

  • http://www.timacheson.com/ Tim Acheson

    Thanks, Nagesh! This all works beautifully. I've learned some very useful details about using YQL. I blogged my usage of YQL, in case you're interested!

    Halo Reach API demo — this would not be possible without YQL:

    http://www.timacheson.com/Blog/2010/oct/halo_reach_api_demo

  • http://www.timacheson.com/ Tim Acheson

    Thanks, that's fantastic!

    Is it enough to hide my API within a query alias? Or is YQL Storage the only way? If the latter is the case, can you explain why? It seems to me that, when the API key is in a query alias, other people can't see it. (E.g. if I set a hard-coded API key value in a query alias, with parameters for the other values, and then use the URL for that query alias. Then, the API key is not revealed in the URL.)

  • http://nagiworld.net Nagesh Susarla

    Hi Tim,

    That's indeed a very good question. Query aliases are only shortcuts. Besides that they do not prevent the user from seeing the actual URLs that you used in your query. The user of the alias can add a '&diagnostics=true' and see the entire URL that was used to make the call to the webservice.

    Let me take the following example:

    If I declared a query alias, such as http://query.yahooapis.com/v1/public/yql/nagesh/test2?foo=topstories which takes in a query parameter named 'foo' and the rest of the URL is hardcoded in my alias, I can simply append &diagnostics=true to see the following which gives away the URL that I used in this case.

    <url execution-time=”10″ >http://rss.news.yahoo.com/rss/topstories&lt;/url&gt;;

    Using stored secrets going one step ahead by obsuring all URLs that appear in the diagnostics to look something like http://domain... thus ensuring that any secret gives are not divulged to the user.

  • http://www.timacheson.com/ Tim Acheson

    Thanks, Nagesh! This all works beautifully. I've learned some very useful details about using YQL. I blogged my usage of YQL, in case you're interested!

    Halo Reach API demo — this would not be possible without YQL:

    http://www.timacheson.com/Blog/2010/oct/halo_reach_api_demo

  • http://blinklist.com/users/nerojohn91 Jane Deas

    This is nice YQL tutorial.  Thanks nagesh for providing such a great post.

  • http://www.wordsjournal.com/bookmarks/dorothysimmons Terry Little

    Excellent information it is definitely. Friend on mine continues to be seeking this tips.

  • Charlie Fuller

    Your online travel business sells the travel niagara tours from new york product directly to
    consumers online or wholesales it too other travel agencies, travel
    agents, tour operators and resellers.

  • Linda Lovelace

    The reason this is a top ten niagara tours from new york safe travel tip is obvious. The kids are our number one concern when it comes to traveling.

blog comments powered by Disqus