Why does the API lack (proper) HTTP cache control headers?

Question

I've just read Kevin's answer to Do I need API key to use StackAuth API? and been quite surprised that the API doesn't seem to facilitate proper HTTP caching already:

You really shouldn't be hitting it more than once a day though, for site information, and in accordance with general API guidelines for associated users.

That's understandable and perfectly fine with me, but why don't you tell me (and especially my toolchain) this via the well established HTTP cache control headers then?

Both of which can be safely cached quite heavily.

Indeed, and I'd happily do so out of own interest, but preferably without the need to reinvent the wheel on my part for this well explored use case - if proper HTTP cache control headers would be served, any decent toolchain component wouldn't even think about violating your API guidelines/policies.

Granted, caching might not provide much benefit (if at all) for many volatile API routes and/or be not exactly easy to implement on the server side in some cases, but still it would for others, and the StackAuth routes are a good example, e.g. for /sites you could do:

• Expiration-based caching:
  Send an HTTP 1.1 Cache-Control: max-age=... header for one day into the future, if that's what the API guidelines ask for right now.
  • Of course you could get even fancy here in case you actually happen to know when you are going to update the sites, e.g. if you'd only ever publish a new site on a Friday you might as well indicate this right away.
  • See HTTP 1.0 Expires: ... header too for the complete picture.
• Validation-based caching:
  Send an HTTP 1.1 ETag: ... header computed from a hash of the result.
  • This might not seem to be too useful with the short list of sites right now (though it would still reduce bandwidth considerably), but this is planned to change, isn't it ;) Even more important though is that you are still communicating valuable information, because otherwise I need to compare the current and former result locally myself in order to know whether there has been any change in this resource.
  • See HTTP 1.0 Last Modified: ... header too for the complete picture.

Or any variation thereof - the chosen cache strategy largely depends on the particular route of course, but some basics applied would get us a long way without risking premature optimization, IMHO.

I'm reluctantly marking this as a feature request rather than a bug, but please consider applying these RESTful principles in order to take the cache control burden off the Stack Apps developer community shoulders and onto those of the Stack Exchange giants ;)

---

Further Reading

• HTTP caching options - more editorial like overview in the context of a RESTful architecture
• HTTP Caching - concise tutorial with good contrasting of expiration/validation
• How To Optimize Your Site With HTTP Caching - concise tutorial with good illustrations
• Caching Tutorial - thorough tutorial often cited/referenced elsewhere
• Caching in HTTP - the authoritative HTTP/1.1 reference

Answer 1

There are two, rather divergent, answers to this question.

First and foremost, its not in the API because nobody (us included, woops!) thought of it before the API interface was frozen... so it didn't get implemented, even as a trial feature. Perhaps we'll revisit it for a subsequent version.

---

Alright, onto the real meat of the issue.

Headers _{(Unintentionally)} Suck

The sad truth of the internet today, especially for sites with as much uptake amongst corporate (firewall-ed) users as StackOverflow and ServerFault, is that you really can't rely on HTTP Headers. There's a reason we treat Accept-Encoding as a suggestion, at best.

Basically, anything that is HTTP/1.1 specific has a good^* chance of not making it through a proxy; so Cache-Control and ETag are right out. This sucks and we know it, but its also beyond our power to change.

I'm being optimistic and hoping that HTTP/1.0 "compliant" proxies actually pass all relevant headers through, but I wouldn't be at all surprised to find a popular counterpoint.

The Difference Between the Sites and the API

The API could go the route of the sites themselves, and make use of features we know won't actually be available to everyone; but consider the difference in "targets".

The sites only care about the handful of popular browsers out there, all of which have extensive testing regiments and often times histories extending back into HTTP/1.0 days. We can be certain that if you're using IE, Firefox, Chrome, Safari, Opera or whatever that the sudden disappearance of - say - Cache-Control isn't going to cause a crash in some seldom used code path.

The API, however, is catering to a potentially much larger and more diverse set of [app]s and [library]s. Most of which are products of hobbyists with limited resources, and naturally the code-bases themselves are very young. Instructing developers they can make use of things that will often - but not always - be there is just asking for all sorts of nasty crashes in untested code paths.

Once you consider that most [app] developers probably aren't behind proxies when developing their [app]s that path is even less attractive given the implied difficulty in properly testing an [app] under such a scheme.

As I noted above, we'll revisit this - maybe not this particular solution, but something similar in purpose anyway - in future versions of the API. Making it easier for developers to not spam us with spurious requests is very attractive, naturally.

_{*I don't have hard numbers to throw around, but we've seen it in closed beta's which are pretty small. I'd use ~1% as a planning number, maybe higher.}

Answer 2

Perhaps I can offer an opinion on cacheability of the API. Please don't downvote before reading through.

Caching is hard

Caching and cache invalidation is hard. Often, if not always, you need to intimately know the data to be able to cache it effectively.

In order to enable a generic Etag caching ability the server will need to execute the request and then get a hash for it to compare with the requested hash. So, from server's point of view, this kind of caching is pointless.

Take a look, for example, at this request (which is the heaviest I could think of - requesting 100 answers with body):

http://api.stackoverflow.com/0.9/questions/9033/answers?pageSize=100&body=true

Answers request http://img641.imageshack.us/img641/6236/answerrequest.png

It takes the server nearly twice the transfer time to process the request. So, a trivial implementation of Etag caching would be at most 50% efficient.

Validation using time

You could possibly have an even more trivial caching that simply caches by URL and, for example, if the URL matches exactly something the server did before than it would either serve cached data or reply with Not Modified. However, I would think that cache hits on exact API URLs would be pretty rare.

Expiration-based caching

Again, without actually getting the data from the DB, the best the server can do is give you an estimate of how long it thinks the data will be valid for. This is the kind of thing that an app/wrapper can actually do for itself. E.g. do not request the answers for a particular question more than once a minute. Which brings me to the next point:

Wouldn't it be easier for the API to provide data "freshness" intervals for different kinds of requests as an API call?

This way the app has the option of requesting data more frequently than the server thinks it's valid for and giving the app the ability to cache with possibly better freshness confidence than the app would guess.

Regardless,

Caching at the application level trumps all

Let me give you a half example, half plug. Stack Tagz does two levels of caching.

One is completed timeseries - this is the graph that you see. I consider that questions don't really get retagged all that often, so once it calculates a timeseries, it's persisted to the DB. Requesting a graph for Jon Skeet takes about 400 requests (which would be smaller if the vectorized requests are fixed, hint hint). It'd be crazy to make those requests every time someone wants to look at his graph, especially considering it won't look any different next week.

Another is caching of individual questions. There is significant overlap if multiple people answer the same question, no good reason to request it again. Even if a question is retagged/deleted, it's OK, there are plenty of other data points for it to be statistically significant.

So, I guess what I am trying to say, is that different apps have different tolerances for the data freshness, and hence can improve on the most pessimistic guarantee that the API can provide. Individual app makers should really think about caching, not delegate it all to the API.