17
# -*- coding: utf-8 -*- 
# Python3
import urllib
import urllib.request as url_req
opener = url_req.build_opener()
url='http://zh.wikipedia.org/wiki/'+"毛泽东"
opener.open(url).read()
# opener.open(url.encode("utf-8")).read()
# # doesn't work either

When I run it, it complains that:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 10-12: ordinal not in range(128)

But I can't use .encode() either as it will complain:

Traceback (most recent call last):
  File "t.py", line 8, in <module>
    opener.open(url.encode("utf-8")).read()
  File "/usr/local/Cellar/python3/3.2.2/lib/python3.2/urllib/request.py", line 360, in open
    req.timeout = timeout
AttributeError: 'bytes' object has no attribute 'timeout'

Anyone knows how to deal with that?

Hanfei Sun
  • 42,165
  • 37
  • 115
  • 221

3 Answers3

24

You could use urllib.parse.quote() to encode the path section of URL.

#!/usr/bin/env python3
from urllib.parse   import quote
from urllib.request import urlopen

url = 'http://zh.wikipedia.org/wiki/' + quote("毛泽东")
content = urlopen(url).read()
jfs
  • 374,366
  • 172
  • 933
  • 1,594
11

The fantastic requests library does this for you out of the box:

>>> url='http://zh.wikipedia.org/wiki/'+"毛泽东"
>>> import requests
>>> r = requests.get(url)
>>> len(r.content)
818747
jterrace
  • 61,216
  • 21
  • 150
  • 193
  • Probably sometimes you should use r.text, not r.content - to see unicode characters in the content. – sergzach Apr 25 '18 at 15:01
2

You can not use arbitrary unicode strings as part of an URL. The URL must be properly encoded. See here for details:

http://www.w3schools.com/tags/ref_urlencode.asp

In particular you want to use the urllib.quote() or urllib.quote_plus() method of the Python API for quoting your URL properly.

http://docs.python.org/library/urllib.html

Andreas Jung
  • 1
  • 19
  • 73
  • 119