3

My daily work often involves handling databases from around the world, as I support customers. Frequently encoded in CSV / TXT.

For legacy reasons, encoding often needs to be done into GB2312 (Windows codepage 936). And at times, certain characters don't convert.

I know that GB2312 does not cover as much as GB18030 or UTF8. But how much less ? And is there a way to figure out, without comparing every character in the full list ?

MyICQ
  • 141
  • 4
  • UTF in an encoding, not a character set. Do you mean Unicode? – fefe Apr 21 '22 at 09:15
  • @fefe UTF-8 is a character encoding: it encodes characters. It makes sense to describe how many characters can be encoded using it. – wizzwizz4 Apr 21 '22 at 17:17
  • @wizzwizz4 not really. UTF-8 is literally a Unicode Transformation Format; it does not deal with characters at all. Rather, it specifies how to turn code points (non-negative integers from [ 0x0 .. 0x10ffff ]) to sequences of one to four bytes. Strictly speaking, 'Unicode' is not a character set either; 'Unicode v14' is one, 'Unicode v15' is another. – John Frazer Apr 22 '22 at 04:48

1 Answers1

5

GB2312 is a strict subset of GBK, which in turn is a strict subset of GB18030.

Windows CP936 originally only covers GB2312, but expanded to most of GBK since Win95.

GB2312 has 6763 Chinese characters.

GBK has 21003 Chinese character. ref

GB18030 (originally) has 27484 Chinese characers. (same ref as above)

Also, GB18030 contains a mapping into Unicode, which means all Unicode characters can also be represented in GB18030.

Unicode 14.0 has 92865 Chinese characters (CJK Unified Ideograph). These characters in Unicode are called CJK Unified Ideographs as they are not only used in Chinese, but also in Japanese, Korean, and other languages that use (or used to use) Chinese characters. Some characters are never used in Chinese.

However, the characters in GB2312 are used most frequently. They already cover most characters in everyday use. The use of other characters are rare.

fefe
  • 8,735
  • 18
  • 35
  • An additional note: GBK also contains traditional characters, while GB2312 does not. Names, among other things, sometimes contain traditional characters even in Mainland China. – Max Xiong Apr 25 '22 at 18:34