Invalid characters in File.ReadAllText

Question

I'm calling File.ReadAllText() in a program designed to format some files that I have.

Some of these files contain the ® (174) symbol. However, when the text is being read, the returned string contains � (65533) symbols where the ® (174) should be.

What would cause this and how can I fix it?

score 14 · Accepted Answer · answered Mar 18 '13 at 15:47

14

This is likely due to a mismatch in the Encoding. Use the ReadAllText overload which allows you to specify the proper Encoding to use when reading the file.

The default overload will assume UTF-8 unless it can detect UTF-32. Any other encoding will come through incorrectly.

answered Mar 18 '13 at 15:47

Reed Copsey

539,124
75
1,126
1,354

1

Is there a reason that the method doesn't use Encoding.Default as it's default encoding? – mrK Mar 18 '13 at 15:59
@mrK Not sure why it's that way, but that's what the framework designer's chose to use. It's documented, but I agree, an odd choice. – Reed Copsey Mar 18 '13 at 16:03
2

One explanation could be that `Encoding.Default` uses the so-called current ANSI code page of the system which varies from system to system. Using UTF-8 avoids encoding errors you get from encoding and decoding on systems with different current ANSI code pages. Furthermore UTF-8 can encode all of UNICODE. – Martin Liversage Mar 18 '13 at 16:53

score 14 · Answer 2 · edited May 23 '17 at 12:22

14

Most likely the file contains a different encoding than the default. If you know it, you can specify it using the File.ReadAllText Method (String, Encoding) override.

Code sample:

string readText = File.ReadAllText(path, Encoding.Default);  // <-- change the encoding to whatever the encoding really is

If you DON'T know the encoding, see this previous SO question: How to use ReadAllText when file encoding unknown

edited May 23 '17 at 12:22

Community

1
1

answered Mar 18 '13 at 15:48

David

70,778
16
128
169

1

Encoding.Default no longer works in .NET Core, always returns utf8... – Stefan Steiger Jun 16 '20 at 11:26

score 12 · Answer 3 · answered Mar 18 '13 at 15:47

You need to specify the encoding when you call File.ReadAllText, unless the file is actually in UTF-8, which it sounds like it's not. (Basically the one-parameter overload is equivalent to passing in UTF-8 as the second argument. It will also detect UTF-32 with an appropriate byte-order mark, I believe.)

The first thing is to work out which encoding it is in (e.g. ISO-8859-1 - but you need to check this) and then pass that as a second argument.

For example:

Encoding isoLatin1 = Encoding.GetEncoding(28591);
string text = File.ReadAllText(path, isoLatin1);

It's always important that you know what encoding binary data is using before you try to read it as text. That's true for files, network streams, anything.

score 0 · Answer 4 · answered Mar 18 '13 at 15:48

The character you are reading is the Replacement character

used to replace an incoming character whose value is unknown or unrepresentable in Unicode compare the use of U+001A as a control character to indicate the substitute function

http://www.fileformat.info/info/unicode/char/fffd/index.htm

You are getting this because the actual encoding of the file does not match the encoding your program expects.

By default ReadAllText expects UTF-8. It is encountering a byte sequence that does not represent a valid UTF-8 character, so replacing it with the Replacement character.

Invalid characters in File.ReadAllText

4 Answers4

Linked

Related