19

In python 2, it was possible to get debug output from urllib by doing

import httplib
import urllib
httplib.HTTPConnection.debuglevel = 1
response = urllib.urlopen('http://example.com').read()

However, in python 3 it looks like this has been moved to 

http.client.HTTPConnection.set_debuglevel(level)

However, I'm using urllib not http.client directly. How can I set it up so that my http request display debugging information in this way?

Here's what I"m using so far. What's the best way to proceed if I want to be able to get debug information?

#Request Login page
cookiejar = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookiejar))
request = urllib.request.Request(options.uri)
add_std_headers(request)
response = opener.open(request)
response_string = response.read().decode("utf8")
# ...
Martin Tournoij
  • 24,971
  • 24
  • 101
  • 136
Zxaos
  • 7,391
  • 12
  • 46
  • 61

2 Answers2

19

You were right the first time. You can simply add the line http.client.HTTPConnection.debuglevel = 1 at the start of your file to turn on HTTP debugging application-wide. urllib.request still uses http.client.

It seems that there's also a way to set the debuglevel for a single handler (by creating urllib.request.HTTPHandler(debuglevel=1) and building an opener with that), but on my installation of Python3 (3.0b3) it's not actually implemented. I imagine that's changed in more recent versions!

PAG
  • 1,756
  • 1
  • 15
  • 18
0

In https://github.com/urllib3/urllib3/issues/107, SmithSamuelM recommends:

if self.debug:
            import httplib
            httplib.HTTPConnection.debuglevel = 5
            requests.packages.urllib3.add_stderr_logger()
serv-inc
  • 32,612
  • 9
  • 143
  • 165
  • That's for Python 2 though, isn't it. – Nickolay Oct 21 '21 at 00:32
  • @Nickolay: https://docs.python.org/2/library/httplib.html says "Note The httplib module has been renamed to http.client in Python 3. The 2to3 tool will automatically adapt imports when converting your sources to Python 3. " – serv-inc Oct 21 '21 at 08:14