Delete every non utf-8 symbols from string

Question

I have a big amount of files and parser. What I Have to do is strip all non utf-8 symbols and put data in mongodb. Currently I have code like this.

with open(fname, "r") as fp:
    for line in fp:
        line = line.strip()
        line = line.decode('utf-8', 'ignore')
        line = line.encode('utf-8', 'ignore')

somehow I still get an error

bson.errors.InvalidStringData: strings in documents must be valid UTF-8: 
1/b62010montecassianomcir\xe2\x86\x90ta0\xe2\x86\x90008923304320733/290066010401040101506055soccorin

I don't get it. Is there some simple way to do it?

UPD: seems like Python and Mongo don't agree about definition of Utf-8 Valid string.

In my head algrithm of parsing is not very important end point is in first few lines. Maybe I'm mistaken — Darth Kotik, Oct 24 '14 at 05:27

Irshad Bhat · Accepted Answer · 2016-03-11T09:43:35.180

90

Try below code line instead of last two lines. Hope it helps:

line=line.decode('utf-8','ignore').encode("utf-8")

edited Mar 11 '16 at 09:43

answered Oct 24 '14 at 13:45

Irshad Bhat

7,941
1
21
32

1

I have some invisible chars which are appearing only after insertion. any fix for this? – user168983 Sep 29 '16 at 08:04
1

@user168983 can you give an example – Irshad Bhat Sep 29 '16 at 19:41
3

this `line.decode('utf-8','ignore').encode("utf-8")` produce this error _AttributeError: 'str' object has no attribute 'decode'_, i work with python3 – Chedi Bechikh Jan 20 '17 at 22:22
21

@ChediBechikh Here is how you do it in python3 `bytes(line, 'utf-8').decode('utf-8','ignore')` – Irshad Bhat Jan 20 '17 at 22:46
this does not change anything, the error exist when printing so i write this code `print(ligne.rstrip().encode('utf-8','ignore')) ` but the problem with encode is , it add a b caracter like **b'press'** – Chedi Bechikh Jan 20 '17 at 23:19
2

This doesn't seem to work. I get lots of special chars: `\00\00\00\00\00` – dengar81 Feb 01 '21 at 11:27
It looks like the only difference between the code in the question and the answer is that encode() no longer has ignore. Why would this make a difference? The default is strict which would raise an exception rather than skip the invalid utf8? – Richard EB Dec 07 '21 at 20:42

score 33 · Answer 2 · answered Apr 27 '17 at 06:19

33

For python 3, as mentioned in a comment in this thread, you can do:

line = bytes(line, 'utf-8').decode('utf-8', 'ignore')

The 'ignore' parameter prevents an error from being raised if any characters are unable to be decoded.

If your line is already a bytes object (e.g. b'my string') then you just need to decode it with decode('utf-8', 'ignore').

answered Apr 27 '17 at 06:19

Alex

10,175
6
60
68

2

But if `line` is already ` str` in py3, is it allowed to be non-utf8? – Dan M. Jan 11 '21 at 11:13

score 7 · Answer 3 · answered Jan 05 '17 at 05:23

Example to handle no utf-8 characters

import string

test=u"\n\n\n\n\n\n\n\n\n\n\n\n\n\nHi <<First Name>>\nthis is filler text \xa325 more filler.\nadditilnal filler.\n\nyet more\xa0still more\xa0filler.\n\n\xa0\n\n\n\n\nmore\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nfiller.\x03\n\t\t\t\t\t\t    almost there \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthe end\n\n\n\n\n\n\n\n\n\n\n\n\n"

print ''.join(x for x in test if x in string.printable)

this removes all non-ascii characters, which includes many, many valid UTF-8 characters — szxk, Jun 08 '17 at 18:08

score 1 · Answer 4 · answered Jul 28 '18 at 05:04

1

with open(fname, "r") as fp:
for line in fp:
    line = line.strip()
    line = line.decode('cp1252').encode('utf-8')

answered Jul 28 '18 at 05:04

Willem

1,296
1
8
7

2

This will be horribly wrong if the original input encoding is not in fact code page 1252. You remove the error, but produce garbage. The error is there for a reason, to prevent you from producing garbage. – tripleee Nov 09 '20 at 09:10

Delete every non utf-8 symbols from string

4 Answers4

Linked