373

If I do

url = "http://example.com?p=" + urllib.quote(query)
  1. It doesn't encode / to %2F (breaks OAuth normalization)
  2. It doesn't handle Unicode (it throws an exception)

Is there a better library?

Peter Mortensen
  • 30,030
  • 21
  • 100
  • 124
Paul Tarjan
  • 46,930
  • 58
  • 166
  • 211

5 Answers5

481

Python 2

From the documentation:

urllib.quote(string[, safe])

Replace special characters in string using the %xx escape. Letters, digits, and the characters '_.-' are never quoted. By default, this function is intended for quoting the path section of the URL.The optional safe parameter specifies additional characters that should not be quoted — its default value is '/'

That means passing '' for safe will solve your first issue:

>>> urllib.quote('/test')
'/test'
>>> urllib.quote('/test', safe='')
'%2Ftest'

About the second issue, there is a bug report about it. Apparently it was fixed in Python 3. You can workaround it by encoding as UTF-8 like this:

>>> query = urllib.quote(u"Müller".encode('utf8'))
>>> print urllib.unquote(query).decode('utf8')
Müller

By the way, have a look at urlencode.

Python 3

In Python 3, the function quote has been moved to urllib.parse:

>>> import urllib.parse
>>> print(urllib.parse.quote("Müller".encode('utf8')))
M%C3%BCller
>>> print(urllib.parse.unquote("M%C3%BCller"))
Müller
Flimm
  • 115,689
  • 38
  • 227
  • 240
Nadia Alramli
  • 105,894
  • 35
  • 170
  • 151
  • 2
    Thanks you, both worked great. urlencode just calls quoteplus many times in a loop, which isn't the correct normalization for my task (oauth). – Paul Tarjan Nov 08 '09 at 09:14
  • 7
    the spec: [rfc 2396](https://www.ietf.org/rfc/rfc2396.txt) defines these as reserved `reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" | "$" | ","` Which is what urllib.quote is dealing with. – Jeff Sheffield Sep 23 '15 at 17:42
  • 8
    `urllib.parse.quote` [docs](https://docs.python.org/3/library/urllib.parse.html#url-quoting) – Andreas Haferburg Dec 16 '16 at 10:50
  • Also, in the case of encoding a search query, you maybe better off using quote_plus: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_plus 1. It encodes slashes by default 2. It also encodes spaces – Pavel Vergeev May 30 '18 at 09:50
  • `six.moves.urllib.parse.quote(u"Müller".encode('utf8'))` for Python 2 and 3. – Bob Stein Dec 10 '18 at 21:00
  • 1
    if you wanna retain the colon from http: , do `urllib.parse.quote('http://example.com/some path/').replace('%3A', ':')` – chrizonline May 09 '19 at 07:27
  • 2
    @chrizonline Just use `urllib.parse.quote(url, safe=':/')`. Even better, encode `some path`, then join strings. This is Python, not PHP. – Pavel Vlasov Dec 23 '21 at 09:26
  • safe="" is missing in Python 3 answer! – PythoNic Jun 04 '22 at 14:48
202

In Python 3, urllib.quote has been moved to urllib.parse.quote, and it does handle Unicode by default.

>>> from urllib.parse import quote
>>> quote('/test')
'/test'
>>> quote('/test', safe='')
'%2Ftest'
>>> quote('/El Niño/')
'/El%20Ni%C3%B1o/'
Peter Mortensen
  • 30,030
  • 21
  • 100
  • 124
Paolo Moretti
  • 51,429
  • 22
  • 99
  • 91
  • 2
    The name `quote` is rather vague as a global. It might be nicer to use something like urlencode: `from urllib.parse import quote as urlencode`. – Luc Mar 05 '19 at 16:35
  • 3
    Note that there is a function named `urlencode` in `urllib.parse` already that does something completely different, so you'd be better off picking another name or risk seriously confusing future readers of your code. – jaymmer - Reinstate Monica Apr 02 '20 at 02:41
63

I think module requests is much better. It's based on urllib3.

You can try this:

>>> from requests.utils import quote
>>> quote('/test')
'/test'
>>> quote('/test', safe='')
'%2Ftest'

My answer is similar to Paolo's answer.

Peter Mortensen
  • 30,030
  • 21
  • 100
  • 124
Aminah Nuraini
  • 16,240
  • 7
  • 82
  • 100
  • 8
    `requests.utils.quote` is link to python `quote`. See [request sources](https://github.com/kennethreitz/requests/blob/master/requests/compat.py#L36). – Cjkjvfnby Aug 05 '15 at 14:11
  • 22
    `requests.utils.quote` is a thin compatibility wrapper to `urllib.quote` for python 2 and `urllib.parse.quote` for python 3 – Jeff Sheffield Sep 23 '15 at 17:30
  • without reading the comments, this is creating confusion... – PythoNic Jun 04 '22 at 14:46
15

If you're using Django, you can use urlquote:

>>> from django.utils.http import urlquote
>>> urlquote(u"Müller")
u'M%C3%BCller'

Note that changes to Python mean that this is now a legacy wrapper. From the Django 2.1 source code for django.utils.http:

A legacy compatibility wrapper to Python's urllib.parse.quote() function.
(was used for unicode handling on Python 2)
Peter Mortensen
  • 30,030
  • 21
  • 100
  • 124
Rick Westera
  • 2,945
  • 1
  • 33
  • 23
5

It is better to use urlencode here. There isn't much difference for a single parameter, but, IMHO, it makes the code clearer. (It looks confusing to see a function quote_plus! - especially those coming from other languages.)

In [21]: query='lskdfj/sdfkjdf/ksdfj skfj'

In [22]: val=34

In [23]: from urllib.parse import urlencode

In [24]: encoded = urlencode(dict(p=query,val=val))

In [25]: print(f"http://example.com?{encoded}")
http://example.com?p=lskdfj%2Fsdfkjdf%2Fksdfj+skfj&val=34

Documentation

Peter Mortensen
  • 30,030
  • 21
  • 100
  • 124
balki
  • 24,438
  • 28
  • 97
  • 142