Conscientious use of the API

Question

What constitutes abuse?

Making an undue number of requests in a short span of time, or regularly exceeding a request quota during "normal" use.

Guidelines for polling

We realize that a great many applications will need to regularly make calls to see if some event has occurred on one of the Trilogy sites.

Examples:

A user gained/lost reputation
A question was answered
New questions - in general, or within a tag or a search query - were posted
... and many more

However, polling can incur considerable overhead especially if done too aggressively. So, to lay down some hard guidelines...

If you're application is making identical requests more than once a minute it is probably going to be considered abusive.

Another way to think of it, if you're using more than 15% of your daily request quota simply polling for new activity you're being needlessly aggressive.

What constitutes an "identical" request?

Exactly the same method and query parameters. You don't need to wait a minute before fetching the 2nd page of a request, for example, but you do need to wait a minute between calls to say the /questions method.

What if the requests are only triggered by user activity? For example: a refresh button for /questions results. Is allowing a manual refresh considered abusive? I can imagine some users tapping it way more than once per minute. Do we need to put a throttle on these types of things? — TM., Jun 27 '10 at 22:14
@T M - yes, you should throttle those requests. If a user is going out of there way to defeat your throttling, that's a different matter. — Kevin Montrose, Jun 27 '10 at 23:01
@T M - perhaps a caching buffer proxy class in between the generation of the URL and the actual request would help. when a request is made it is passed to the buffer, buffer checks the cached for a similar request less than 1 minute old and returns it if found, otherwise make the request and cache the results before returning it. — Sky Sanders, Jun 29 '10 at 01:25

score 11 · Answer 1 · answered Jun 06 '10 at 13:25

Another scenario to consider is testing. Granted tests shouldn't really be hitting the service that hard but I know for my library I have at least 50 integration test that when run all at once would it the /questions method with the same parameters many times in a minute. These requests are done anonymously (with no api key) so I am not sure if you treat those differently.

One thing you didn't cover in your post is what the penalty is for being too aggressive or if any action is taken at all against such apps.

Well, I ran into the problem tonight of having my IP run out of anonymous request due to integration testing with my library. The only solution I have is to include an API key as part of my libraries tests. — lfoust, Jun 17 '10 at 04:07

Tyler Carter · Answer 2 · 2010-06-06T06:57:07.217

Another good way to be conscientious to the API is think about timing. If your app has certain peaks hours, slow down on the calls after hours (If 75% of users are US, run at 1 call/min from 9-5, and 1 call/5 min the rest of the time)

The same goes vice-versa. If your application isn't completely reliant on up-to-minute data during the day, slow down on the requests during the day and then do your heavy lifting during off-hours. That way it leaves room for applications that are reliant on up-to-minute data. (Statistics Applications, as well as other apps that generalize data come to mind)

As Code Poet pointed out, technically users are using the site 24/7 because the site is fairly global, but realistically speaking, there are definitely off-peak hours. Maybe at some point the devs can share those off-hours with us so we can adjust accordingly to spread out the load.

score 4 · Answer 3 · answered Jun 06 '10 at 10:22

Obvious, but perhaps worth stating:

An easy way for desktop apps to reduce API usage is to detect when the screensaver is active or the workstation is locked, and throttle or pause API calls.
Use separate polling intervals for different types of data: you may need questions every minute, but can live with reputation updates every 5 minutes.
Favor fewer API calls over smaller payloads when choosing which method to call. E.g., if you want to know the questions tagged foo, bar, and baz in the last minute, retrieve all questions and filter in your application, rather than doing separate calls for each tag.

score 0 · Answer 4 · answered Jun 06 '10 at 13:32

0

For a JavaScript library that uses the API to dynamically render content to the end-user of a website, making identical requests often may not be avoidable. With StackUnderflow.js, for example, each user to my blog will make a request to display questions related to the blog entry they are looking at. There's no way to cache that across all users unless I involve the server somehow, and the whole point of the library is to enable people to StackExchange-ify their websites by doing nothing more than including some script (often, the only thing they can do, such as on hosted blog sites).

If I get about 1000 hits to my blog a day, there'd be at least that many requests, many of which will be the same. I don't see a way around that without proxying through some server-side service, which is a very heavy 'ask' for a simple javascript library for only the purpose of reducing duplicate requests (and also makes the library dependent on this proprietary service).

answered Jun 06 '10 at 13:32

InfinitiesLoop

411
3
7

3

Remember that all throttling/request discussion is done with respect to IPs. Your blog will be fine provided a single page load doesn't result in many repeated requests in rapid succession. – Kevin Montrose Jun 06 '10 at 13:38
Although if there are a lot of readers behind a proxy.... – InfinitiesLoop Jun 06 '10 at 14:27
3

proxies break the internet. Sad fact of life, and one that's difficult to engineer around. – Kevin Montrose Jun 06 '10 at 14:43
No argument here. – InfinitiesLoop Jun 06 '10 at 14:54

score 0 · Answer 5 · edited Apr 13 '17 at 12:25

This is a good point. For my application, it does an update of the statistics every day at midnight UTC. I've modified it now so that it does one user every 15 seconds instead of just doing them all as fast as possible. That means for 40 users, it could take up to 10 minutes to update all the days statistics, but that really is not a big deal.

Also, I wouldn't mind running the statistics update at a different time, say to coincide with a "low period" for the server - if such a time exists in such an international site... is there a better time of the day to run statistics updates or does it really not matter that much?

Conscientious use of the API

What constitutes abuse?

Guidelines for polling

What constitutes an "identical" request?

5 Answers5

Linked