How can I force urllib2 to time out?

Question

I want to to test my application's handling of timeouts when grabbing data via urllib2, and I want to have some way to force the request to timeout.

Short of finding a very very slow internet connection, what method can I use?

I seem to remember an interesting application/suite for simulating these sorts of things. Maybe someone knows the link?

here's a [code example of `slow_http_server.py`](http://stackoverflow.com/a/32685765/4279) — jfs, Sep 21 '15 at 01:12

score 10 · Accepted Answer · answered Nov 15 '10 at 21:02

10

I usually use netcat to listen on port 80 of my local machine:

nc -l 80

Then I use http://localhost/ as the request URL in my application. Netcat will answer at the http port but won't ever give a response, so the request is guaranteed to time out provided that you have specified a timeout in your urllib2.urlopen() call or by calling socket.setdefaulttimeout().

answered Nov 15 '10 at 21:02

ʇsәɹoɈ

21,294
7
52
59

very nice. never thought of that! – jcomeau_ictx Nov 15 '10 at 21:06
I had to use: nc -l -p 80 Great idea though! Thanks. – mhost Nov 08 '12 at 20:17
Yeah, older versions of netcat have slightly different command line options, IIRC. Glad it worked! – ʇsәɹoɈ Nov 09 '12 at 02:32

score 6 · Answer 2 · answered Nov 15 '10 at 21:06

You could set the default timeout as shown above, but you could use a mix of both since Python 2.6 in there is a timeout option in the urlopen method:

import urllib2
import socket

try:
    response = urllib2.urlopen("http://google.com", None, 2.5)
except URLError, e:
    print "Oops, timed out?"
except socket.timeout:
    print "Timed out!"

The default timeout for urllib2 is infinite, and importing socket ensures you that you'll catch the timeout as socket.timeout exception

score 3 · Answer 3 · answered Nov 15 '10 at 21:00

3

import socket 

socket.setdefaulttimeout(2) # set time out to 2 second.

If you want to set the timeout for each request you can use the timeout argument for urlopen

answered Nov 15 '10 at 21:00

mouad

63,741
17
112
105

One thing to keep in mind is the timeout argument for urlopen was only added in Python 2.6 – mhost Nov 08 '12 at 20:18

score 0 · Answer 4 · answered Nov 15 '10 at 21:00

0

why not write a very simple CGI script in bash that just sleeps for the required timeout period?

answered Nov 15 '10 at 21:00

jcomeau_ictx

36,402
6
91
101

score 0 · Answer 5 · answered Nov 15 '10 at 21:15

If you're running on a Mac, speedlimit is very cool.

There's also dummynet. It's a lot more hardcore, but it also lets you do some vastly more interesting things. Here's a pre-configured VM image.

If you're running on a Linux box already, there's netem.

I believe I've heard of a Windows-based tool called TrafficShaper, but that one I haven't verified.

How can I force urllib2 to time out?

5 Answers5

Linked