Convert UTF-16 to UTF-8 under Windows and Linux, in C

Question

I was wondering if there is a recommended 'cross' Windows and Linux method for the purpose of converting strings from UTF-16LE to UTF-8? or one should use different methods for each environment?

I've managed to google few references to 'iconv' , but for somreason I can't find samples of basic conversions, such as - converting a wchar_t UTF-16 to UTF-8.

Anybody can recommend a method that would be 'cross', and if you know of references or a guide with samples, would very appreciate it.

Thanks, Doori Bar

See this previous question: http://stackoverflow.com/questions/148403/utf8-to-from-wide-char-conversion-in-stl — Mark Ransom, May 20 '10 at 02:13

score 12 · Answer 1 · edited Aug 21 '21 at 00:39

12

Change encoding to UTF-8 with PowerShell:

Get-Content PATH\temp.txt -Encoding Unicode | Set-Content -Encoding UTF8 PATH2\temp.txt

edited Aug 21 '21 at 00:39

Ben Thul

28,821
4
42
65

answered Mar 11 '15 at 08:46

user4657497

129
1
2

Works great on Windows 2008 R2. BOM is also added. – Maris B. Oct 04 '16 at 11:51

score 6 · Answer 2 · answered May 19 '10 at 18:57

6

The open source ICU library is very commonly used.

answered May 19 '10 at 18:57

Hans Passant

897,808
140
1,634
2,455

score 6 · Answer 3 · answered May 20 '10 at 02:08

6

If you don't want to use ICU,

Windows: WideCharToMultiByte
Linux: iconv (Glibc)

answered May 20 '10 at 02:08

Alex B

79,216
40
196
275

Remy Lebeau · Answer 4 · 2020-06-01T19:38:34.943

5

#include <iconv.h>

wchar_t *src = ...; // or char16_t* on non-Windows platforms
int srclen = ...;
char *dst = ...;
int dstlen = ...;
iconv_t conv = iconv_open("UTF-8", "UTF-16");
iconv(conv, (char*)&src, &srclen, &dst, &dstlen);
iconv_close(conv);

edited Jun 01 '20 at 19:38

answered May 20 '10 at 02:03

Remy Lebeau

505,946
29
409
696

2

I suppose "UTF-16" and "UTF-8" should switch places. – DooriBar May 20 '10 at 12:34
Where does iconv... come from? Answer: #include – Also, this seems to be Linux specific. – Thomas Tempelmann Jun 01 '20 at 19:11
[libiconv](https://www.gnu.org/software/libiconv/) is a cross-platform library, it is available on many platforms, including Linux. – Remy Lebeau Jun 01 '20 at 19:39

score 4 · Answer 5 · answered Dec 04 '13 at 04:37

I have run into this problem too, I solve it by using boost locale library

try
{           
    std::string utf8 = boost::locale::conv::utf_to_utf<char, short>(
                        (short*)wcontent.c_str(), 
                        (short*)(wcontent.c_str() + wcontent.length()));
    content = boost::locale::conv::from_utf(utf8, "ISO-8859-1");
}
catch (boost::locale::conv::conversion_error e)
{
    std::cout << "Fail to convert from UTF-8 to " << toEncoding << "!" << std::endl;
    break;
}

The boost::locale::conv::utf_to_utf function try to convert from a buffer that encoded by UTF-16LE to UTF-8, The boost::locale::conv::from_utf function try to convert from a buffer that encoded by UTF-8 to ANSI, make sure the encoding is right(Here I use encoding for Latin-1, ISO-8859-1).

Another reminder is, in Linux std::wstring is 4 bytes long, but in Windows std::wstring is 2 bytes long, so you would better not use std::wstring to contain UTF-16LE buffer.

score 4 · Answer 6 · answered Apr 03 '20 at 10:26

4

If you have MSYS2 installed then the iconv package (which is installed by default) lets you use:

 iconv -f utf-16le -t utf-8 <input.txt >output.txt

answered Apr 03 '20 at 10:26

M.M

134,614
21
188
335

score 2 · Answer 7 · answered Oct 12 '12 at 17:19

2

There's also utfcpp, which is a header-only library.

answered Oct 12 '12 at 17:19

Kevin Smyth

1,832
20
21

score 1 · Answer 8 · answered Mar 10 '21 at 14:16

1

Another portable C possibility to convert string between UTF-8, UTF-16, UTF-32, wchar - is mdz_unicode library.

answered Mar 10 '21 at 14:16

maxdz

11
1
2

score 0 · Accepted Answer · edited Jun 26 '14 at 12:49

0

Thanks guys, this is how I managed to solve the 'cross' windows and linux requirement:

Downloaded and installed: MinGW , and MSYS
Downloaded the libiconv source package
Compiled libiconv via MSYS.

That's about it.

edited Jun 26 '14 at 12:49

n0p

3,377
2
27
47

answered May 20 '10 at 12:36

DooriBar

387
1
3
8

Convert UTF-16 to UTF-8 under Windows and Linux, in C

9 Answers9

Linked

Related