how to turn characters in wrong codec into space in python?

Asked Jun 08 '15 at 17:38

Active Jun 08 '15 at 18:05

Viewed 61 times

I need to read data from an external source, which is from MS.

As you know, MS likes to embed binary to simple text, so sometimes I run into trouble when I encounter such issue:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 1833: ordinal not in range(128)

So my goal is, instead of this error, I want the text to be output to stdout anyway, but with all these wrong characters turned into white space.

Any idea how to do this? I hope it's not gonna be catch exception and a for loop to manually change it.

edited Jun 08 '15 at 18:05

Paradox

4,473
8
39
84

asked Jun 08 '15 at 17:38

Jason Hu

6,049
1
18
36

@PadraicCunningham you think encode sounds properer? or could you offer a better tag? when i describe this problem, i would say this is a **text encoding** issue. correct me. – Jason Hu Jun 08 '15 at 17:47
I meant are you encoding the strings or is it just printing in a cmd shell that is causing the issue? – Padraic Cunningham Jun 08 '15 at 17:48
@PadraicCunningham the same, output to stdout, or turn it from ascii to unicode. since this code itself doesn't make sense. – Jason Hu Jun 08 '15 at 17:49
you could `u'\u2019'.encode("utf-8",errors="ignore")` but that won't add a space, tu'\u2019' is also a `’` – Padraic Cunningham Jun 08 '15 at 17:51
A similar problem was discussed [here](http://stackoverflow.com/questions/24737594/unicode-encode-error-ascii-codec-cant-encode-character-u-u2019). One more thing: What about reading the file in binary mode and replacing desired characters? – albert Jun 08 '15 at 17:51
@PadraicCunningham that's close. but em... i hope there could be some placeholder to show there was something here. – Jason Hu Jun 08 '15 at 17:54
@KarolyHorvath how to search? – Jason Hu Jun 08 '15 at 17:54
@HuStmpHrrr, where are you running the code from? – Padraic Cunningham Jun 08 '15 at 17:55
@PadraicCunningham unix console. – Jason Hu Jun 08 '15 at 17:58
what does `locale` output? – Padraic Cunningham Jun 08 '15 at 17:59
@HuStmpHrrr: like this: https://www.google.com/search?q=python+replace+all+non-ascii – Karoly Horvath Jun 08 '15 at 17:59
@KarolyHorvath yeah. that's a nice point. – Jason Hu Jun 08 '15 at 18:01

how to turn characters in wrong codec into space in python?

0 Answers0