346

I have spent plenty of time as far as I am newbie in Python.
How could I ever decode such a URL:

example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0

to this one in python 2.7: example.com?title==правовая+защита

url=urllib.unquote(url.encode("utf8")) is returning something very ugly.

Still no solution, any help is appreciated.

Martijn Pieters
  • 963,270
  • 265
  • 3,804
  • 3,187
swordholder
  • 3,989
  • 3
  • 17
  • 14
  • 4
    In the general case, the tail of a URL is just a cookie. You can't know which local character-set encoding the server uses or even whether the URL encodes a string or something completely different. (Granted, many URLs *do* encode a human-readable string; and often, you can guess the encoding very easily. But it's not possible in the generally case or completely automatically.) – tripleee Jan 29 '18 at 12:45

4 Answers4

556

The data is UTF-8 encoded bytes escaped with URL quoting, so you want to decode, with urllib.parse.unquote(), which handles decoding from percent-encoded data to UTF-8 bytes and then to text, transparently:

from urllib.parse import unquote

url = unquote(url)

Demo:

>>> from urllib.parse import unquote
>>> url = 'example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0'
>>> unquote(url)
'example.com?title=правовая+защита'

The Python 2 equivalent is urllib.unquote(), but this returns a bytestring, so you'd have to decode manually:

from urllib import unquote

url = unquote(url).decode('utf8')
Keyur Potdar
  • 6,949
  • 6
  • 25
  • 39
Martijn Pieters
  • 963,270
  • 265
  • 3,804
  • 3,187
  • 2
    So why is the + character left in the string? I thought that %2B was the + character and + literals were removed during decoding? – AlexLordThorsen Oct 02 '14 at 22:29
  • 8
    @Rawrgulmuffins `+` is a space in [`x-www-form-urlencoded` data](http://en.m.wikipedia.org/wiki/Application/x-www-form-urlencoded#The_application.2Fx-www-form-urlencoded_type); you'd use `urllib.parse.parse_qs()` to parse that, or use `urllib.parse.unquote_plus()`. But they should only appear in the query string, not the rest of the URL. – Martijn Pieters Oct 02 '14 at 23:14
159

If you are using Python 3, you can use urllib.parse

url = """example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0"""

import urllib.parse
urllib.parse.unquote(url)

gives:

'example.com?title=правовая+защита'
Martin Thoma
  • 108,021
  • 142
  • 552
  • 849
pavan
  • 3,095
  • 3
  • 20
  • 28
20

You can achieve an expected result with requests library as well:

import requests

url = "http://www.mywebsite.org/Data%20Set.zip"

print(f"Before: {url}")
print(f"After:  {requests.utils.unquote(url)}")

Output:

$ python3 test_url_unquote.py

Before: http://www.mywebsite.org/Data%20Set.zip
After:  http://www.mywebsite.org/Data Set.zip

Might be handy if you are already using requests, without using another library for this job.

ivanleoncz
  • 7,457
  • 4
  • 52
  • 48
0

In HTML the URLs can contain html entities. This replaces them, too.

#from urllib import unquote #earlier python version
from urllib.request import unquote
from html import unescape
unescape(unquote('https://v.w.xy/p1/p22?userId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&confirmationToken=7uAf%2fxJoxRTFAZdxslCn2uwVR9vV7cYrlHs%2fl9sU%2frix9f9CnVx8uUT%2bu8y1%2fWCs99INKDnfA2ayhGP1ZD0z%2bodXjK9xL5I4gjKR2xp7p8Sckvb04mddf%2fiG75QYiRevgqdMnvd9N5VZp2ksBc83lDg7%2fgxqIwktteSI9RA3Ux9VIiNxx%2fZLe9dZSHxRq9AA'))
Roland Puntaier
  • 2,925
  • 27
  • 32