119

I'm testing how some of my code handles bad data, and I need a few series of bytes that are invalid UTF-8.

Can you post some, and ideally, an explanation of why they are bad/where you got them?

alex
  • 460,746
  • 196
  • 858
  • 974
twk
  • 16,060
  • 21
  • 70
  • 96
  • 4
    Possible duplicate of [Really Good, Bad UTF-8 example test data](http://stackoverflow.com/questions/1319022/really-good-bad-utf-8-example-test-data) – Claudiu Apr 18 '16 at 15:04

5 Answers5

83

Take a look at Markus Kuhn's UTF-8 decoder capability and stress test file

You'll find examples of many UTF-8 irregularities, including lonely start bytes, continuation bytes missing, overlong sequences, etc.

Nemanja Trifunovic
  • 23,910
  • 3
  • 48
  • 87
61

In PHP:

$examples = array(
    'Valid ASCII' => "a",
    'Valid 2 Octet Sequence' => "\xc3\xb1",
    'Invalid 2 Octet Sequence' => "\xc3\x28",
    'Invalid Sequence Identifier' => "\xa0\xa1",
    'Valid 3 Octet Sequence' => "\xe2\x82\xa1",
    'Invalid 3 Octet Sequence (in 2nd Octet)' => "\xe2\x28\xa1",
    'Invalid 3 Octet Sequence (in 3rd Octet)' => "\xe2\x82\x28",
    'Valid 4 Octet Sequence' => "\xf0\x90\x8c\xbc",
    'Invalid 4 Octet Sequence (in 2nd Octet)' => "\xf0\x28\x8c\xbc",
    'Invalid 4 Octet Sequence (in 3rd Octet)' => "\xf0\x90\x28\xbc",
    'Invalid 4 Octet Sequence (in 4th Octet)' => "\xf0\x28\x8c\x28",
    'Valid 5 Octet Sequence (but not Unicode!)' => "\xf8\xa1\xa1\xa1\xa1",
    'Valid 6 Octet Sequence (but not Unicode!)' => "\xfc\xa1\xa1\xa1\xa1\xa1",
);

From http://www.php.net/manual/en/reference.pcre.pattern.modifiers.php#54805

philfreo
  • 39,285
  • 26
  • 123
  • 140
6

The idea of patterns of ill-formed byte-sequences can be gotten from the table of well-formed byte sequences. See "Table 3-7. Well-Formed UTF-8 Byte Sequences" in the Unicode Standard 6.2.

    Code Points    First Byte Second Byte Third Byte Fourth Byte
  U+0000 -   U+007F   00 - 7F
  U+0080 -   U+07FF   C2 - DF    80 - BF
  U+0800 -   U+0FFF   E0         A0 - BF     80 - BF
  U+1000 -   U+CFFF   E1 - EC    80 - BF     80 - BF
  U+D000 -   U+D7FF   ED         80 - 9F     80 - BF
  U+E000 -   U+FFFF   EE - EF    80 - BF     80 - BF
 U+10000 -  U+3FFFF   F0         90 - BF     80 - BF    80 - BF
 U+40000 -  U+FFFFF   F1 - F3    80 - BF     80 - BF    80 - BF
U+100000 - U+10FFFF   F4         80 - 8F     80 - BF    80 - BF

Here are the examples generated from U+24B62. I used them for a bug report: Bug #65045 mb_convert_encoding breaks well-formed character

// U+24B62: "\xF0\xA4\xAD\xA2"
"\xF0\xA4\xAD"    ."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2"
"\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD"

The oversimplification of range of trailing bytes([0x80, 0xBF]) can be seen in the various libraries.

// U+0800 - U+0FFF
\xE0\x80\x80

// U+D000 - U+D7FF
\xED\xBF\xBF

// U+10000 -  U+3FFFF
\xF0\x80\x80\x80

// U+100000 - U+10FFFF
\xF4\xBF\xBF\xBF
Dave Moten
  • 11,689
  • 2
  • 35
  • 44
masakielastic
  • 4,390
  • 1
  • 37
  • 41
4

,̆ was particularly evil. I see it as combined on ubuntu.

comma-breve

user1015281
  • 59
  • 1
-6

Fuzz Testing - generate a random sequence of octets. Most likely you'll get some illegal sequences sooner than later.

shoosh
  • 73,722
  • 51
  • 204
  • 318
  • 13
    There's nothing worse than having heisenbugs or eisentests. Tests pass 10 times, you release the product, test fails. – Eric Duminil Nov 21 '17 at 15:03
  • 2
    @EricDuminil ever heard of srand() ? – shoosh Nov 21 '17 at 19:22
  • 4
    Fair enough. Could you please mention it in the answer so I could revert my downvote? – Eric Duminil Nov 21 '17 at 19:26
  • 3
    Ahah. Well, there's always something new to learn, that's why I come to SO in the first place. I think your `srand()` advice is a good idea, it might help other people here. – Eric Duminil Nov 22 '17 at 08:20
  • 1
    We can create the invalid strings directly, we don't need randomness to try and eventually find them, though string processing libraries would (probably!) benefit from fuzzing just in case. – galva Mar 02 '20 at 13:42