I am using Beautiful Soup to parse webpages and printing the name of the webpages visited on the terminal. However, often the name of the webpage has single left (\u2018) and right(\u2019) character which the python can't print as it gives charmap encoding error. Is there any way to remove these characters?
Asked
Active
Viewed 7.6k times
34
-
please add some of your code and your Output and expected output. it will be good to Draw better solutions – sundar nataraj Jun 23 '14 at 04:46
-
3Note: U+2018 = Left Single Quotation Mark, U+2019 = Right Single Quotation Mark – Marjorie Roswell Jun 19 '17 at 05:27
-
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="UTF-8") is helpful to avoid encoding error in the first place. – Yiğit Apr 23 '20 at 21:34
1 Answers
59
These codes are Unicode for the single left and right quote characters. You can replace them with their ASCII equivalent which Python shouldn't have any problem printing on your system:
>>> print u"\u2018Hi\u2019"
‘Hi’
>>> print u"\u2018Hi\u2019".replace(u"\u2018", "'").replace(u"\u2019", "'")
'Hi'
Alternatively with regex:
import re
s = u"\u2018Hi\u2019"
>>> print re.sub(u"(\u2018|\u2019)", "'", s)
'Hi'
However Python shouldn't have any problem printing the Unicode version of these as well. It's possible that you are using str() somewhere which will try to convert your unicode to ascii and throw your exception.
Martin Konecny
- 54,197
- 19
- 131
- 151
-
that regex is hard to read. Its not wrong, at least it doesnt seem wrong. But it certainly is not easy to read. And can lead to errors. Because people reading it might think it works like `(ab|c)` to match ab or c. But it does match ab and ac. I just had someone break their entire code base, because they didnt use the regex properly. Whole library broken, because they tried doing (.js|.jsx|.ts|.tsx) – Toskan May 11 '20 at 19:59