Python: Check if a string contains chinese character?

Question

A string maybe this

ipath= "./data/NCDC/上海/虹桥/9705626661750dat.txt"

or this

ipath = './data/NCDC/ciampino/6240476818161dat.txt'

How do I know the first string contains chinese?

I find this answer maybe helpful: Find all Chinese text in a string using Python and Regex

but it didn't work out:

import re
ipath= "./data/NCDC/上海/虹桥/9705626661750dat.txt"
re.findall(ur'[\u4e00-\u9fff]+', ipath) # => []

Are you using Python 2? In Python 3, it [seems working](https://ideone.com/lPHSky) without `r` when declaring the regex. — Wiktor Stribiżew, Jan 04 '16 at 08:44
Have a look at this [Python 2 demo](https://ideone.com/i0unNw) - does it work for you? — Wiktor Stribiżew, Jan 04 '16 at 08:51

xecgr · Accepted Answer · 2019-06-11T06:02:41.257

23

The matched string should be unicode as well

>>> import re
>>> ipath= u"./data/NCDC/上海/虹桥/9705626661750dat.txt"
>>> re.findall(r'[\u4e00-\u9fff]+', ipath)
[u'\u4e0a\u6d77', u'\u8679\u6865']

edited Jun 11 '19 at 06:02

answered Jan 04 '16 at 08:52

xecgr

4,753
3
16
28

1

Some reason, add `u` gives me a syntax error, remove it works for me `re.findall(r"[\u4e00-\u9fff]+", ipath)` – LYu May 13 '18 at 03:50
@xecgr I get the same problem (and solution) mentioned by @LYu above. Please can you explain what the `u` is required for and why this is causing problems that can be resolved by removing it from your code? – Matt Shepherd Nov 29 '18 at 17:07

timgeb · Answer 2 · 2016-01-04T09:23:41.787

10

If you just want to know whether there is a chinese character in your string you don't need re.findall, use re.search and the fact that match objects are truthy.

>>> import re
>>> ipath= u'./data/NCDC/上海/虹桥/9705626661750dat.txt'
>>> ipath2 = u'./data/NCDC/ciampino/6240476818161dat.txt'
>>> for x in (ipath, ipath2):
...     if re.search(u'[\u4e00-\u9fff]', x):
...         print 'found chinese character in ' + x
... 
found chinese character in ./data/NCDC/上海/虹桥/9705626661750dat.txt

edited Jan 04 '16 at 09:23

answered Jan 04 '16 at 09:03

timgeb

73,231
20
109
138

should this be faster? – dcsan Dec 29 '19 at 23:54

score 6 · Answer 3 · edited Jun 02 '18 at 06:03

6

And for those of us who don't care for re:

>>> ipath= u"./data/NCDC/上海/虹桥/6240476818161dat.txt"
>>> for i in range(len(ipath)):
...  if ipath[i] > u'\u4e00' and ipath[i] < u'\u9fff':
...   print ipath[i]
... 
上
海
虹
桥

Edit: for the full list of Chinese characters this SO link is worth looking at as the range U+4E00..U+9FFF is not complete. What's the complete range for Chinese characters in Unicode?

edited Jun 02 '18 at 06:03

brandon

43
5

answered Jan 04 '16 at 09:04

Rolf of Saxony

19,475
3
34
50

This is simply taking what was done at the C level and moving it to Python. For large blocks of text, it will be significantly slower. – Brad Solomon Jul 29 '19 at 16:34
@BradSolomon and your point is? This was a non `re` response, it doesn't claim to be anything else, certainly not faster. – Rolf of Saxony Jul 29 '19 at 16:50

score 3 · Answer 4 · answered Oct 16 '18 at 13:47

Using these codepoint ranges, we can write an is_cjk function:

# list of cjk codepoint ranges
# tuples indicate the bottom and top of the range, inclusive
cjk_ranges = [
        ( 0x4E00,  0x62FF),
        ( 0x6300,  0x77FF),
        ( 0x7800,  0x8CFF),
        ( 0x8D00,  0x9FCC),
        ( 0x3400,  0x4DB5),
        (0x20000, 0x215FF),
        (0x21600, 0x230FF),
        (0x23100, 0x245FF),
        (0x24600, 0x260FF),
        (0x26100, 0x275FF),
        (0x27600, 0x290FF),
        (0x29100, 0x2A6DF),
        (0x2A700, 0x2B734),
        (0x2B740, 0x2B81D),
        (0x2B820, 0x2CEAF),
        (0x2CEB0, 0x2EBEF),
        (0x2F800, 0x2FA1F)
    ]

def is_cjk(char):
    char = ord(char)
    for bottom, top in cjk_ranges:
        if char >= bottom and char <= top:
            return True
    return False

Which we can then use to process text, using functions like filter, any, all, and map to process the text character-by-character, or compose more complex functions:

txt = "./data/NCDC/上海/虹桥/9705626661750dat.txt"
txt_sanitized = "./data/NCDC/9705626661750dat.txt"
any(map(is_cjk, txt)) # True
any(map(is_cjk, txt_sanitized)) # False
''.join(filter(is_cjk, txt)) # '上海虹桥'

Note that the CJK ranges will include not only Chinese characters but also may include Korean and Japanese characters. For more complex usage, try a dedicated library like cjklib.

score 2 · Answer 5 · answered Jan 04 '16 at 08:55

import re
ipath= raw_input()
print re.findall(ur'[\u4e00-\u9fff]+', ipath.decode("utf-8"))

Output:./data/NCDC/上海/虹桥/9705626661750dat.txt [u'\u4e0a\u6d77', u'\u8679\u6865']

You need to decode the input to make it unicode.

or

 import re
 ipath= unicode(raw_input(),encoding="utf-8")
 print re.findall(ur'[\u4e00-\u9fff]+', ipath)

score 2 · Answer 6 · answered Sep 14 '21 at 20:32

Use \p{Han} regex with PyPi regex:

import regex
ipath = "./data/NCDC/上海/虹桥/9705626661750dat.txt"
print(regex.findall(r'\p{Han}+', ipath) )
# => ['上海', '虹桥']

See Python proof.

The regex.search is enough to detect:

if regex.search(r'\p{Han}', ipath):
    print(f'"{ipath}" contains Chinese!')

score 1 · Answer 7 · answered Jan 04 '16 at 08:59

'' is a bytestring on Python 2. Either add from __future__ import unicode_literals at the top of the module or use unicode literals: u'':

>>> import re
>>> ipath= u"./data/NCDC/上海/虹桥/9705626661750dat.txt"
>>> re.findall(ur'[\u4e00-\u9fff]+', ipath)
[u'\u4e0a\u6d77', u'\u8679\u6865']

score 0 · Answer 8 · answered Sep 11 '18 at 18:18

0

According to this question, the range should be [\u2E80-\u2FD5\u3190-\u319f\u3400-\u4DBF\u4E00-\u9FCC]

answered Sep 11 '18 at 18:18

Kevin He

1,154
7
19

Python: Check if a string contains chinese character?

8 Answers8

Linked