Looking for an UTF-8 table

Question

I'm looking for a csv file containing the utf-8 encoding table. This file would contain the code point, the escaped html entity and the symbol itself.

I already found websites providing these correspondances, like this one or this other one. However, they do not provide an offline access and pasting all the content would be, while technically possible, a little bit tedious.

Would you know where such a file is available ?

Be aware that some Unicode "characters" are "combining marks", which bind with the preceding character to form a grapheme (e.g., an accent that combines with a latin letter) — Walter Tross, Apr 11 '14 at 22:17
If I were you, I would base my work on the official Unicode documents at www.unicode.org, in particular http://www.unicode.org/reports/ and even more in particular the UnicodeData.txt file — Walter Tross, Apr 14 '14 at 07:18
Numeric HTML entities are easily built as &#x<HEX>;, where <HEX> is the Unicode codepoint in hex (the first column of the UnicodeData.txt file, with or without leading zeros) — Walter Tross, Apr 15 '14 at 16:18

philshem · Accepted Answer · 2014-04-09T10:41:30.380

With python you can lookup each unicode character by its integer code using unichr.

import sys
with open('unicode.csv','wb') as output:
    for i in xrange(sys.maxunicode):
        output.write(unicode(i))
        output.write(u',')
        output.write(unichr(i).encode('utf-8'))
        output.write(u',')
        output.write(unichr(i).encode('ascii', 'xmlcharrefreplace'))
        output.write(u'\n')
print sys.maxunicode

This gives you a file (unicode.csv) which has the unciode integer, unicode representation, unicode character, and HTML escaped character (for non-ascii).

For example, each line looks like this:

64058,u'\ufa3a',墨,&#64058;

I put the code and the unicode.csv file on github for easier access.

Note: Because the unicode character set includes newline characters, CSV is not really the best format. (See lines 10 to 13.) I also added a python code to generate a JSON file, which is more safe than CSV for storing unicode characters.

The best solution would be to generate the lookup on the fly (my code takes about 1 second, but the load times for CSV and JSON are much longer). — philshem, Apr 09 '14 at 14:12

Looking for an UTF-8 table

1 Answers1