21

As stated in title, I would like to check in given file object (opened as binary stream) is valid UTF-8 file.

Anyone?

Thanks

malat
  • 11,258
  • 13
  • 77
  • 141
Jox
  • 7,032
  • 14
  • 48
  • 63

2 Answers2

28
def try_utf8(data):
    "Returns a Unicode object on success, or None on failure"
    try:
       return data.decode('utf-8')
    except UnicodeDecodeError:
       return None

data = f.read()
udata = try_utf8(data)
if udata is None:
    # Not UTF-8.  Do something else
else:
    # Handle unicode data
Daniel Stutzbach
  • 70,140
  • 17
  • 84
  • 76
  • Obviously I didn't do my homework good enough when there is more that one solution simple as this :( Thanks! – Jox Jul 16 '10 at 23:53
14

You could do something like

import codecs
try:
    f = codecs.open(filename, encoding='utf-8', errors='strict')
    for line in f:
        pass
    print "Valid utf-8"
except UnicodeDecodeError:
    print "invalid utf-8"
michael
  • 451
  • 3
  • 7
  • 1
    Could be simpler by using only one line: `codecs.open("path/to/file", encoding="utf-8", errors="strict").readlines()` instead of 3. – colidyre May 07 '19 at 19:06