Sunday, February 22, 2015

Cookbook Caching [feedly]

Cookbook Caching
// Chef Blog

If you're configuring cookbook s3 URL TTL in your chef-server.rb configuration file (opscode_erchef['s3_url_ttl']), then you've been creating cookbook URLs that expire that many (28800 by default) seconds from "now" (i.e. the time of the request), which is great for Chef Client runs, but it's terrible for caching!

Each signed URL has a query string with an expiration time, which means that every time a signed URL is generated, it's unique. If you want to cache URLs, we need URLs that don't change as frequently.

Today in Chef Server 12.0.4, we're introducing a new setting: opscode_erchef['s3_url_expiry_window_size']. Don't need it? Just set it to :off and close this browser tab (you have too many open, tbqh). Actually, it's :off by default, so you don't have to do anything.

If you want less uniqueness in URLs, you can set s3_url_expiry_window_size to be the length of time for which a URL should be unique. For example, let's set it to "15m". Now, every URL generated in a 15 minute window will be the same. The price we pay is that the link will actually take a little longer to expire than you've configured in s3_url_ttl, but no more than 15 minutes longer.

You can also set s3_url_expiry_window_size to be a percentage of the s3_url_ttl. The default of 28800 (or 8 hours), and a s3_url_expiry_window_size value of "10%" would mean a window size of 48 minutes.

Here's a walkthrough of the "15m" setting:

Screenshot 2015-02-18 15.00.25

Each letter represents a unique URL, with the capital letter being the first time that URL is seen. Look at the URL A generated at 1:03. Until 1:15 they'll all be set to expire at 2:15 indicated by the lower case as.

Once it's 1:15, we're in a new interval, but we don't try getting a URL until 1:25. We set that one to expire at 2:30 (1hr + the remaining time in the interval).

Nobody asks for a URL until after 1:30, so the URL B is only used once and is never asked for again. Oh well. We played the odds and lost this time. It's not the end of the world.

At 1:33, the URL C is generated and this interval gets used alot, so it's good that we have this feature.

You get the idea. Over the course of the day we will only ever generate 96 unique expiration times, as opposed to a new expiration time for every URL requested.

Now, 15m may not be the optimal window size. If we went with "60m" then we'd only generate 24 unique URLs per day. That's why we've made it configurable.

If you've got an F5 load balancer or even if you want nginx to serve up cached cookbook content instead of hitting s3 or Bookshelf, well, now you can!

After you've enabled the s3_url_expiry_window_size, you have another choice to make. If you're using nginx to cache cookbooks:

opscode_erchef['nginx_bookshelf_caching'] = :on

Then nginx will serve up the cached content instead of forwarding the request to s3 or Bookshelf. If you're using an F5 or other load balancer, turn that setting off like this:

opscode_erchef['nginx_bookshelf_caching'] = :off

and your load balancer will take care of serving up cached content.


Shared via my feedly reader

Sent from my iPhone