0

When I use a local Tor proxy and start many head requests, my local proxy dies immediately (and needs to be restarted) without any outcome.

I used those suggested pieces of code: concurrent.futures combined with requests session requests session.

When I request one after another, it works fine. But that is too time consuming.

idonthaveanyplan
  • 56
  • 4
  • maybe you need not to send so many requests at once? there are lots of ways to limit the no. of concurrent requests: do some searches (for e.g. 'python limit concurrent requests') and pick an implementation. start with 2 concurrent requests and see how many you can do before it dies. What do you want to do a zillion head requests for anyhow? – 2e0byo Nov 12 '21 at 12:19
  • I limited the amount of concurrent requests to 10, but that is still too much. With 2 requests at the same time and a sufficient timeout the time explodes. Its for research purposes. – idonthaveanyplan Nov 12 '21 at 12:44
  • thus you need to keep the concurrent requests < 10, sadly, with your current setup. This isn't really a python problem: it's just your tor proxy which can't hanle it. You can look at seeing why *that* is, or replacing the tor proxy with another, or using multiple tor proxies at once, but none of these are really issues from the python end of things. I suspect tor is quite careful with what it lets you do: it's a free service anyhow, and has to accomodate a lot of use – 2e0byo Nov 12 '21 at 14:00
  • It seems a bit odd that 10 concurrent requests would cause problems but the issue may be related to the fact that each connection will use the same Tor circuit. You might try [stream isolation](https://stackoverflow.com/a/56828482/892493) and have each request use a different circuit. – drew010 Nov 18 '21 at 18:37

0 Answers0