1

My node server does a cache warmup on start up. The data is requested from an external webservice.

All of this is done asynchronously, which means that (currently) all requests (about 300) are done almost instantly (as quickly as node can loop), and then each response is handled when it arrives (as per what I understand the node async paradigm to be).

This causes a problem because it triggers the DDoS protection on the external webservice.

So I somehow need to limit the frequency with which requests are sent out during this phase (when the server is running, requests are not done this often and never in this quick succession).

But: How do I reconcile this with the async programming that node "needs"? If I use setTimeout it will just delay the requests to a later time, but because it's done in a loop, they will still happen almost in the same instant.

Using await so each call is only triggered when the previous one was received feels too much of a limit. I can do parallel requests, just not that many in that short of a timespan.

How can I get around this?

F.P
  • 16,592
  • 33
  • 120
  • 188
  • 2
    Here's something that does exactly this. [`async.parallelLimit()`](https://caolan.github.io/async/docs.html#parallelLimit). – Patrick Roberts Jan 15 '18 at 06:51
  • 1
    Perhaps this will help: [Make several requests to an API that can only handle 20 request a minute](https://stackoverflow.com/questions/33378923/make-several-requests-to-an-api-that-can-only-handle-20-request-a-minute/33379149#33379149). – jfriend00 Jan 15 '18 at 06:51
  • 1
    This one has a function already in it that you could use for controlling `requestsPerSec`: [Choose proper async method for batch processing for max requests/sec](https://stackoverflow.com/questions/36730745/choose-proper-async-method-for-batch-processing-for-max-requests-sec/36736593#36736593) – jfriend00 Jan 15 '18 at 06:56
  • 1
    @PatrickRoberts - The problem with `parallelLimit` is that it only controls how many requests are in flight at the same time. Rate limiting is often implemented on servers as requests per second from a given source so a scheme that controls requests per second directly can generally be more useful. – jfriend00 Jan 15 '18 at 06:57
  • 1
    @jfriend00 Oh, I agree, promises are a much more direct and robust approach to rate-limiting, but to a beginner, I believe that using `async` is probably a bit easier to figure out. Maybe if they're still having errors with outgoing requests then they might consider a more direct approach like yours. – Patrick Roberts Jan 15 '18 at 14:15
  • @PatrickRoberts - Promises are the core tool in Javascript for managing asynchronous operations. There is NO reason to postpone learning how to use them. They are NOT more difficult to use than plain callbacks. Yes there are a few things to learn to use them appropriately, but once learned they are a far, far superior tool than the `async` library. I would not personally recommend anyone spend one second learning the `async` library. If you want a library of functions to help you, then get the Bluebird promise library and use that. You will at least be building on modern tools. – jfriend00 Jan 15 '18 at 19:18
  • @PatrickRoberts - And, I'm not sure you understand my actual objection to your `async.parallelLimit()` suggestion. I wasn't even objecting to the async library itself (though I do think promises are the more modern tool). I was objecting to the fact that it does not control requests per second. It controls how many requests are in flight at the same time which is usually not what rate limiting is set to control. – jfriend00 Jan 15 '18 at 19:20
  • @jfriend00 I understood your argument. I think you're the one misunderstanding mine on multiple points. I wasn't saying that promises are harder to learn than `async`, nor was I discouraging the use of promises in general. I was saying that directly controlling requests per second using promises would be harder to figure out than a drop-in function like `async.parallelLimit()` or Bluebird's `Promise.map()` with `concurrent` parameter that causes similar throttling and could very well be "good enough" for the asker's purposes. – Patrick Roberts Jan 15 '18 at 19:33
  • @jfriend00 While directly rating-limiting vs. batching are not equivalent, it's much easier to call a pre-existing function than to try and understand or hand-write a monstrosity like the one [in your answer](https://stackoverflow.com/a/36736593/1541563). – Patrick Roberts Jan 15 '18 at 19:33

0 Answers0