Как использовать юникод в c

Unicode in C

Unicode started with a fixed two-byte character set but later on, it was changed. Unicode consists of more than a hundred thousand characters and over a hundred languages to handle the vast multitude of different languages including complex characters like emojis, modifiers, and other unknown characters.

If we try to print a special character like an emoji in C language, the compiler will not give the result of printing that emoji. Rather, it returns a code for that emoji which will not be helpful for the user. To resolve this matter, we will practice the Unicode process in C.

Syntax:

To print a Unicode in C language, we use a function named_setmode in which we define the bits of character. In the following figure, we are giving U16 as a parameter, so it prints only the characters with 16 Bit limit. By default, C language only prints 8 Bit characters.

We use the wprintf instead of the printf function to print the line. And we will put L at the start of the statement that we want to print. Apart from the following syntax, we also have to add two includes which are:

Note: Unicode is not a function or method in C, so there is no specific syntax to it. The syntax attached here is just for reference.

Example 1:

We will get a better understanding of the topic by following this example. In the figure, you can understand that we imported two extra libraries into our code. One is <wchar.h> and the other is <locale.h>. The <wchar.h> allows us to create the new datatypes to store the special characters in them. In C language, the <locale.h> header is used to define the location-based settings for example symbols like currencies and different date formats.

In the main function code block, we called a setlocale() function. This is the function of the <locale.h> header. In the setlocale() function, we passed a parameter of LC_CTYPE. This function has many parameters like LC_ALL which sets everything. LC_CTYPE affects all the character functions. It defines the character attributes like case conversion and character classifications.

After that, we declare two variables of wchar_t datatype: star1 and star2. We passed the Unicode for that emoji to our variables. After that, we wrote the values of our variables. We discussed earlier that we use the wprintf for Unicode printing. We can also see in the following code that we used the wprintf instead of printf.

wchar_t star1 = 0x2606 ;

wchar_t star2 = 0x2605 ;

After the successful execution of our C code, we get the following output. We can see that instead of printing the values that we passed to our constants, the system printed two stars. This is because we passed the Unicode for these stars to our variables. Then, because of the setlocale() function and its LC_CTYPE parameter, the system checked the character against that specific code and found a black and white star against that value. We also used the wprintf, so the system printed the character against the passed value of black star and white star.

Example 2:

In the previous example, we used a complex method to let you understand how unicoding works. In this example, we will perform a task to print a Unicode with the help of as less lines of code as possible. Depending upon the Operating System of your computer or machine, you can also print the Unicode characters by simply using the printf function. But for that purpose, you will have to pass a value \U to let the compiler know that it has to print a Unicode character.

In this example, we try to print a smiley face which is a non-English character. We pass the Unicode value of that emoji to our code. In the following code, we simply printed a message “Hello there” with a smiley at the end of the statement. The code for happy smiley is “0001F600”. So, we passed it starting with \U just like how printing a string \s is passed so that the system understands that it has to print a string.

One thing that you might have noticed is that we have not used the headers which we used in the previous example. This is because we are not using those functions and techniques to let the system read the Unicode characters.

The output of our code after the compilation is as follows. The system prints the character message as it is but it first reads the \U sign and understands that the next value that is passed to it is a Unicode. So, the system will understand that it has to convert the next code into its respective emoji. After converting the value into the Unicode characters, the system will display the smiley face as an output.

Conclusion

We discussed about unicoding in this article. We explained what a Unicode character is, why it is used and what were the reasons that the unicoding standard was introduced. We discussed how to operate with the Unicode characters in C language as C language provides support for only about 256 characters. In the given examples, we explained how we can encode a set of values into a Unicode emoji or character. By the whole explanation that were previously given, we can say that unicoding is a very helpful approach to tackling the communication bridge. With the help of Unicode developers and programmers from every corner of the world, we can code and write the programs in their languages. On top of that, the whole applications can be developed in different languages which helps them to become more understandable and readable for the users. Unicoding helps to change the UI languages of not only the applications but of the whole Operating system as well. So, unicoding is not a specific function or library in the C language. It is an international coding standard that can be applied in any programming language.

How to use Unicode in C++?

It’s so simple that is the first thing that one learns.

But my problem is that I don’t know how to do the same thing if I enter the name using japanese characters.

So, if you know how to do this in C++, please show me an example (that I can compile and test)

user362981 : Thanks for your help. I compiled the code that you wrote without problem, them the console window appears and I cannot enter any Japanese characters on it (using IME). Also if I change a word in your code («hello») to one that contains Japanese characters, it also will not display these.

Svisstack : Also thanks for your help. But when I compile your code I get the following error:

5 Answers 5

You’re going to get a lot of answers about wide characters. Wide characters, specifically wchar_t do not equal Unicode. You can use them (with some pitfalls) to store Unicode, just as you can an unsigned char . wchar_t is extremely system-dependent. To quote the Unicode Standard, version 5.2, chapter 5:

With the wchar_t wide character type, ANSI/ISO C provides for inclusion of fixed-width, wide characters. ANSI/ISO C leaves the semantics of the wide character set to the specific implementation but requires that the characters from the portable C execution set correspond to their wide character equivalents by zero extension.

The width of wchar_t is compiler-specific and can be as small as 8 bits. Consequently, programs that need to be portable across any C or C++ compiler should not use wchar_t for storing Unicode text. The wchar_t type is intended for storing compiler-defined wide characters, which may be Unicode characters in some compilers.

So, it’s implementation defined. Here’s two implementations: On Linux, wchar_t is 4 bytes wide, and represents text in the UTF-32 encoding (regardless of the current locale). (Either BE or LE depending on your system, whichever is native.) Windows, however, has a 2 byte wide wchar_t , and represents UTF-16 code units with them. Completely different.

A better path: Learn about locales, as you’ll need to know that. For example, because I have my environment setup to use UTF-8 (Unicode), the following program will use Unicode:

But there’s nothing Unicode about it. It merely reads in characters, which come in as UTF-8 because I have my environment set that way. I could just as easily say «heck, I’m part Czech, let’s use ISO-8859-2»: Suddenly, the program is getting input in ISO-8859-2, but since it’s just regurgitating it, it doesn’t matter, the program will still perform correctly.

Now, if that example had read in my name, and then tried to write it out into an XML file, and stupidly wrote <?xml version=»1.0″ encoding=»UTF-8″ ?> at the top, it would be right when my terminal was in UTF-8, but wrong when my terminal was in ISO-8859-2. In the latter case, it would need to convert it before serializing it to the XML file. (Or, just write ISO-8859-2 as the encoding for the XML file.)

On many POSIX systems, the current locale is typically UTF-8, because it provides several advantages to the user, but this isn’t guaranteed. Just outputting UTF-8 to stdout will usually be correct, but not always. Say I am using ISO-8859-2: if you mindlessly output an ISO-8859-1 «è» ( 0xE8 ) to my terminal, I’ll see a «č» ( 0xE8 ). Likewise, if you output a UTF-8 «è» ( 0xC3 0xA8 ), I’ll see (ISO-8859-2) «Ă¨» ( 0xC3 0xA8 ). This barfing of incorrect characters has been called Mojibake.

Often, you’re just shuffling data around, and it doesn’t matter much. This typically comes into play when you need to serialize data. (Many internet protocols use UTF-8 or UTF-16, for example: if you got data from an ISO-8859-2 terminal, or a text file encoded in Windows-1252, then you have to convert it, or you’ll be sending Mojibake.)

Sadly, this is about the state of Unicode support, in both C and C++. You have to remember: these languages are really system-agnostic, and don’t bind to any particular way of doing it. That includes character-sets. There are tons of libraries out there, however, for dealing with Unicode and other character sets.

In the end, it’s not all that complicated really: Know what encoding your data is in, and know what encoding your output should be in. If they’re not the same, you need to do a conversion. This applies whether you’re using std::cout or std::wcout . In my examples, stdin or std::cin and stdout / std::cout were sometimes in UTF-8, sometimes ISO-8859-2.

Try replacing cout with wcout, cin with wcin, and string with wstring. Depending on your platform, this may work:

There are other ways, but this is sort of the «minimal change» answer.

The above article is a must read which explains what unicode is but few lingering questions remains. Yes UNICODE has a unique code point for every character in every language and furthermore they can be encoded and stored in memory potentially differently from what the actual code is. This way we can save memory by for example using UTF-8 encoding which is great if the language supported is just English and so the memory representation is essentially same as ASCII – this of course knowing the encoding itself. In theory if we know the encoding, we can store these longer UNICODE characters however we like and read it back. But real world is a little different.

How do you store a UNICODE character/string in a C++ program? Which encoding do you use? The answer is you don’t use any encoding but you directly store the UNICODE code points in a unicode character string just like you store ASCII characters in ASCII string. The question is what character size should you use since UNICODE characters has no fixed size. The simple answer is you choose character size which is wide enough to hold the highest character code point (language) that you want to support.

The theory that a UNICODE character can take 2 bytes or more still holds true and this can create some confusion. Shouldn’t we be storing code points in 3 or 4 bytes than which is really what represents all unicode characters? Why is Visual C++ storing unicode in wchar_t then which is only 2 bytes, clearly not enough to store every UNICODE code point?

The reason we store UNICODE character code point in 2 bytes in Visual C++ is actually exactly the same reason why we were storing ASCII (=English) character into one byte. At that time, we were thinking of only English so one byte was enough. Now we are thinking of most international languages out there but not all so we are using 2 bytes which is enough. Yes it’s true this representation will not allow us to represent those code points which takes 3 bytes or more but we don’t care about those yet because those folks haven’t even bought a computer yet. Yes we are not using 3 or 4 bytes because we are still stingy with memory, why store the extra 0(zero) byte with every character when we are never going to use it (that language). Again this is exactly the same reasons why ASCII was storing each character in one byte, why store a character in 2 or more bytes when English can be represented in one byte and room to spare for those extra special characters!

In theory 2 bytes are not enough to present every Unicode code point but it is enough to hold anything that we may ever care about for now. A true UNICODE string representation could store each character in 4 bytes but we just don’t care about those languages.

Imagine 1000 years from now when we find friendly aliens and in abundance and want to communicate with them incorporating their countless languages. A single unicode character size will grow further perhaps to 8 bytes to accommodate all their code points. It doesn’t mean we should start using 8 bytes for each unicode character now. Memory is limited resource, we allocate what what we need.

Can I handle UNICODE string as C Style string?

In C++ an ASCII strings could still be handled in C++ and that’s fairly common by grabbing it by its char * pointer where C functions can be applied. However applying current C style string functions on a UNICODE string will not make any sense because it could have a single NULL bytes in it which terminates a C string.

A UNICODE string is no longer a plain buffer of text, well it is but now more complicated than a stream of single byte characters terminating with a NULL byte. This buffer could be handled by its pointer even in C but it will require a UNICODE compatible calls or a C library which could than read and write those strings and perform operations.

This is made easier in C++ with a specialized class that represents a UNICODE string. This class handles complexity of the unicode string buffer and provide an easy interface. This class also decides if each character of the unicode string is 2 bytes or more – these are implementation details. Today it may use wchar_t (2 bytes) but tomorrow it may use 4 bytes for each character to support more (less known) language. This is why it is always better to use TCHAR than a fixed size which maps to the right size when implementation changes.

How do I index a UNICODE string?

It is also worth noting and particularly in C style handling of strings that they use index to traverse or find sub string in a string. This index in ASCII string directly corresponded to the position of item in that string but it has no meaning in a UNICODE string and should be avoided.

What happens to the string terminating NULL byte?

Are UNICODE strings still terminated by NULL byte? Is a single NULL byte enough to terminate the string? This is an implementation question but a NULL byte is still one unicode code point and like every other code point, it must still be of same size as any other(specially when no encoding). So the NULL character must be two bytes as well if unicode string implementation is based on wchar_t. All UNICODE code points will be represented by same size irrespective if its a null byte or any other.

Does Visual C++ Debugger shows UNICODE text?

Yes, if text buffer is type LPWSTR or any other type that supports UNICODE, Visual Studio 2005 and up support displaying the international text in debugger watch window (provided fonts and language packs are installed of course).

Summary:

C++ doesn’t use any encoding to store unicode characters but it directly stores the UNICODE code points for each character in a string. It must pick character size large enough to hold the largest character of desirable languages (loosely speaking) and that character size will be fixed and used for all characters in the string.

Right now, 2 bytes are sufficient to represent most languages that we care about, this is why 2 bytes are used to represent code point. In future if a new friendly space colony was discovered that want to communicate with them, we will have to assign new unicode code pionts to their language and use larger character size to store those strings.

A Modern C++ and Unicode primer

Without doubt, the near-universal acceptance of Unicode has been one of the globalized software success stories of recent years. With its availability, internationalization (i18n) is no longer an esoteric topic, while localization (l10n) of software can, in many cases, be performed without either recompilation of, or even access to, the source-code.

The Unicode Standard (UCS-4) defines slightly over a million code points, which are often written as hexadecimal (eg. U+20AC for the Euro currency symbol). A number of encodings exist in 32, 16 and eight-bit forms, in both big- and little-endian (they are: UTF-32BE, UTF-32LE, UTF-16BE, UTF-16LE and UTF-8). The UTF-8 (Unicode Transformation Format – Eight Bit) encoding is possibly the most common and can encode any Unicode code point (from either UCS-2/16 bit or UCS-4/32 bit) into a code sequence of between one and four bytes in length.

The C++20 Standard introduced a new type char8_t which is intended to hold a single element (code unit) of a UTF-8 code sequence. This removes any ambiguities that may arise from using plain char ; char8_t is always eight bits and unsigned, and should not be used to hold other forms of encoded or raw data. To complement this new type, there are new string ( std::u8string ), string literal ( u8″. » ) and character literal ( u8′.’ ) entities available ( std::u8string is actually just a template specialization as std::basic_string<char8_t> ).

The Unicode literal syntax of Modern C++ can be used to specify UTF-8 code sequences within string literals without the use of a UTF-8 compatible editor (although many coding environments do support UTF-8 for editing source code). The two forms are: \uABCD and \U00ABCDEF and are of fixed length so that other text can follow directly without ambiguity to the compiler, so u8″\u20AC0.50″ is €0.50 as exactly four hexadecimal digits follow \u .

In fact, UTF-8 strings need not contain any top-bit-set characters, and such strings are known as UTF-7, having all code points represented by a single code unit. A UTF-7 code point is (deliberately) identical to the 7-bit ASCII encoding. Also, some eight-bit values, notably 0xFF , never appear as code units in a well-formed UTF-8 string, so could be usefully used as end-of-string markers, if required.

Unicode literals can be used within string literals, and these strings can then be manipulated or output to streams. Note however that the C++ Standard does not specify how Unicode string objects are put to the stream output objects std::cout / std::wcout ; under modern Linuxes your console probably uses a UTF-8 encoding by default, while under Windows it may be necessary to issue a chcp 65001 command to set the UTF-8 code page for a running console session. The following code embeds code point U+20AC at the beginning of two string literals:

As shown above, s1 may not display correctly under Windows even with the correct code page selected, while for s2 a cast is necessary in order to avoid an error similar to “no match for operator<<“. It is possible to define a suitable global operator<< to perform the same reinterpret_cast for all u8string objects:

Another issue is manipulating a std::u8string where the elements have varying lengths (some languages mandate that such strings are immutable, or read-only, while C++ does not). Single code unit sequences encode 7-bits, two-sequences encode 11-bits, three-sequences encode 16-bits and four-sequences encode all possible 21-bits of UCS-4. It is possible to know the length of a well-formed UTF-8 code sequence from the value of the first code unit; a function to calculate the length of a well-formed UTF-8 string in the style of strlen() from the C Standard Library is shown below:

The return value of -1 is for strings which are obviously ill-formed, however checks for all continuation bytes having valid values of 0b10xxxxxx are not carried out by this code, so in production code you should use something more bullet-proof. Similarly for indexing into a std::u8string , counting from the beginning (or end, if you are careful and only need a negative offset) is always necessary and the return type should be char32_t to allow for all possible encoded values. The possibility of encountering a byte order mark (or BOM, code point U+FEFF , UTF-8 sequence EF BB BF ) at the start of the string should also be considered.

Under Windows, the wchar_t type is considered to hold a single UTF-16 code unit (that is either a single UCS-2 code point, or half of a code sequence for a UCS-4 code point above U+10000 ). This means that UCS-2/UCS-4 encoded as UTF-16 can be passed directly to std::wcout and other “wide character” stream objects (so long as a cast from char16_t* to wchar_t* is performed, see above). Under Linux, wchar_t is a platform-defined size, probably 32 bits, however it is unlikely to hold a UCS-4 value. Conversion functions to/from char32_t may be available, and may again be a 1:1 mapping; output to the console using the std::wcout family is possible but should be tested as working correctly.

Often Unicode text will be held in memory for processing or editing using char32_t , but then the issue of outputting to console, GUI or disk file, involves character set conversion to char16_t / wchar_t or char8_t / char . You will almost certainly want to use a library rather than writing your own functions for such operations; the International Components for Unicode website is a good place to start. (As of C++20 the Standard Library conversion functions using std::codecvt are deprecated as they are vulnerable to some malformed multi-byte code sequences.)

Unicode in C and C++: What You Can Do About It Today

Given modern hardware resources, it is unacceptable that we can’t yet routinely communicate text in different scripts or containing technical symbols. Fortunately, we are getting there.

I. Encoding text

The first thing to know is that you do not have to worry about most problems with digital text. The most difficult work is handled below the application layer, in OSes, UI libraries, and the C library. To give you an idea of what goes on though, here is a summary of software problems surrounding text:

Encoding
Mapping characters to numbers. Many such mappings exist; once you know the encoding of a piece of text, you know what character is meant by a particular number. Unicode is one such mapping, and a popular one since it incorporates more characters than any other at this time.
Display
Once you know what character is meant, you have to find a font that has the character and render it. This task is much complicated by the need to display both left-to-right and right-to-left text, the existence of combining characters that modify previous characters and have zero width, the fact that some languages require wider character cells than others, and context-sensitive letterforms.
Input
An input method is a way to map keystrokes (most likely several keystrokes on a typical keyboard) to characters. Input is also complicated by bidirectional text.
Internationalization (i18n)
This refers to the practice of translating a program into multiple languages, effectively by translating all of the program’s strings.
Lexicography
Code that processes text as more than just binary data might have to become a lot smarter. The problems of searching, sorting, and modifying letter case (upper/lower) vary per-language. If your application doesn’t need to perform such tasks, consider yourself lucky. If you do need these operations, you can probably find a UI toolkit or i18n library that already implements them.

The encoding I’ll talk about is called Unicode. Unicode officially encodes 1,114,112 characters, from 0x000000 to 0x10FFFF. (The idea that Unicode is a 16-bit encoding is completely wrong.) For maximum compatibility, individual Unicode values are usually passed around as 32-bit integers (4 bytes per character), even though this is more than necessary. For convenience, the first 128 Unicode characters are the same as those in the familiar ASCII encoding.

The consensus is that storing four bytes per character is wasteful, so a variety of representations have sprung up for Unicode characters. The most interesting one for C programmers is called UTF-8. UTF-8 is a «multi-byte» encoding scheme, meaning that it requires a variable number of bytes to represent a single Unicode value. Given a so-called «UTF-8 sequence», you can convert it to a Unicode value that refers to a character.

UTF-8 has the property that all existing 7-bit ASCII strings are still valid. UTF-8 only affects the meaning of bytes greater than 127, which it uses to represent higher Unicode characters. A character might require 1, 2, 3, or 4 bytes of storage depending on its value; more bytes are needed as values get larger. To store the full range of possible 32-bit characters, UTF-8 would require a whopping 6 bytes. But again, Unicode only defines characters up to 0x10FFFF, so this should never happen in practice.

UTF-8 is a specific scheme for mapping a sequence of 1-4 bytes to a number from 0x000000 to 0x10FFFF:

The x’s are bits to be extracted from the sequence and glued together to form the final number.

It is fair to say that UTF-8 is taking over the world. It is already used for filenames in Linux and is supported by all mainstream web browsers. This is not surprising considering its many nice properties:

It can represent all 1,114,112 Unicode characters.
Most C code that deals with strings on a byte-by-byte basis still works, since UTF-8 is fully compatible with 7-bit ASCII.
Characters usually require fewer than four bytes.
String sort order is preserved. In other words, sorting UTF-8 strings per-byte yields the same order as sorting them per-character by logical Unicode value.
A missing or corrupt byte in transmission can only affect a single character—you can always find the start of the sequence for the next character just by scanning a couple bytes.
There are no byte-order/endianness issues, since UTF-8 data is a byte stream.

See What is UTF-8? for more information about UTF-8.

Side note: Some consider UTF-8 to be discriminatory, since it allows English text to be stored efficiently at one byte per character while other world scripts require two bytes or more. This is a troublesome point, but it should not get in the way of Unicode adoption. First of all, UTF-8 was not really designed to preferentially encode English text. It was designed to preserve compatibility with the large body of existing code that scans for special characters such as line breaks, spaces, NUL terminators, and so on. Furthermore, the encoding used internally by a program has little impact on the user as long as it is able to represent their data without loss. UTF-8 is a great boon, especially for C programming. Think of it this way: if it allows you to internationalize an application that would have been difficult to convert otherwise, it is much less discriminatory than the alternative.

II. The C library

«Multibyte character» or «multibyte string» refers to text in one of the many (possibly language-specific) encodings that exist throughout the world. A multibyte character does not necessarily require more than one byte to store; the term is merely intended to be broad enough to encompass encodings where this is the case. UTF-8 is in fact only one such encoding; the actual encoding of user input is determined by the user’s current locale setting (selected as an option in a system dialog or stored as an environment variable in UNIX). Strings you get from the user will be in this encoding, and strings you pass to printf() are supposed to be as well. Strings within your program can of course be in any encoding you want, but you might have to convert them for proper display.

«Wide character» or «wide character string» refers to text where each character is the same size (usually a 32-bit integer) and simply represents a Unicode character value («code point»). This format is a known common currency that allows you to get at character values if you want to. The wprintf() family is able to work with wide character format strings, and the «%ls» format specifier for normal printf() will print wide character strings (converting them to the correct locale-specific multibyte encoding on the way out).

The C library also provides functions like towupper() that can convert a wide character from any language to uppercase (if applicable). strftime() can format a date and time string appropriately for the current locale, and strcoll() can do international sorting. These and other functions that depend on locale must be initialized at the beginning of your program using

You don’t have to do anything with the locale string returned by setlocale(), but you can use it to query your user’s locale settings (more on this later).

The C library pretty much assumes you will be using multibyte strings throughout your program (since that’s what you get as input). Since multibyte strings are opaque, a lot of functions beginning with «mb» are provided to deal with them. Personally, I don’t like not knowing what encoding my strings use. One concrete problem with the multibyte thing is file I/O— a given file could be in any encoding, independent of locale. When you write a file or send data over a network, keeping the multibyte encoding might be a bad idea. (Even if all software uses only the proper locale-independent C library functions, and all platforms support all encodings internally, there is still no single standard for communicating the encoding of a piece of text; email messages and HTML tags do it in various ways.) You also might be able to do more efficient processing, or avoid rewriting code, if you knew the encoding your strings used.

Your encoding options

You are free to choose a string encoding for internal use in your program. The choice pretty much boils down to either UTF-8, wide (4-byte) characters, or multibyte. Each has its advantages and disadvantages:

Pro: compatible with all existing strings and most existing code
Pro: takes less space
Pro: widely used as an interchange format (e.g. in XML)
Con: more complex processing, O(n) string indexing

Pro: easy to process
Con: wastes space
Pro/Con: although you can use the syntax to easily include wide-character strings in C programs, the size of wide characters is not consistent across platforms (some incorrectly use 2-byte wide characters)
Con: should not be used for output, since spurious zero bytes and other low-ASCII characters with common meanings (such as ‘/’ and ‘\n’) will likely be sprinkled throughout the data.

Pro: no conversions ever needed on input and output
Pro: built-in C library support
Pro: provides the widest possible internationalization, since in rare cases conversion between local encodings and Unicode does not work well
Con: strings are opaque
Con: perpetuates incompatibilities. For example, there are three major encodings for Russian. If one Russian sends data to another through your program, the recipient will not be able to read the message if his or her computer is configured for a different Russian encoding. But if your program always converts to UTF-8, the text is effectively normalized so that it will be widely legible (especially in the future) no matter what encoding it started in.

In this article I will advocate and give explicit instruction on using UTF-8 as an internal string encoding. Many Linux users already set their environment to a UTF-8 locale, in which case you won’t even have to do any conversions. Otherwise you will have to convert multibyte to wide to UTF-8 on input, and back to multibyte on output. Nevertheless, UTF-8 has its advantages.

III. What to do right now

Here’s your to-do list:

«char» no longer means character

Most C string library routines still work with UTF-8, since they only scan for terminating NUL characters. A notable exception is strchr(), which in this context is more aptly named «strbyte()». Since you will be passing character codes around as 32-bit integers, you need to replace this with a routine such as my u8_strchr() that can scan UTF-8 for a given character. The traditional strchr() returns a pointer to the location of the found character, and u8_strchr() follows suit. However, you might want to know the index of the found character, and since u8_strchr() has to scan through the string anyway, it keeps a count and returns a character index as well.

With the old strchr(), you could use pointer arithmetic to determine the character index. Now, any use of pointer arithmetic on strings is likely to be broken since characters are no longer bytes. You’ll have to find and fix any code that assumes «(char*)b — (char*)a» is the number of characters between a and b (though it is still of course the number of bytes between a and b).

The functions mbstowcs() and wcstombs() convert from and to locale-specific encodings, respectively. «mbs» means multibyte string (i.e. the locale-specific string), and «wcs» means wide character string (universal 4-byte characters). Clearly, if you use wide characters internally, you are in luck here. If you use UTF-8, there is a chance that the user’s locale will be set to UTF-8 and you won’t have to do any conversion at all. To take advantage of that situation, you will have to specifically detect it (I’ll provide a function for it). Otherwise, you will have to convert from multibyte to wide to UTF-8.

Version 1.6 (1.5.x while in development) of the FOX toolkit uses UTF-8 internally, giving your program a nice all-UTF-8-all-the-time environment. GTK2 and Qt also support UTF-8.

In your own code, you can use my u8_inc() and u8_dec() to move through strings. If you develop libraries or languages, be sure to expose some kind of inc() and dec() API so nobody has to move through a string by repeatedly requesting the nth character.

IV. Some UTF-8 routines

This library is quite incomplete; you might want to look at related FSF offerings and libutf8. libutf8 provides the multibyte and wide character C library routines mentioned above, in case your C library doesn’t have them.

Since performance is sometimes a concern with UTF-8, I made my routines as fast and lightweight as possible. They perform minimal error checking— in particular, they do not bother to determine whether a sequence is valid UTF-8, which can actually be a security problem. I justify this decision by reiterating that the intention of the library is to manipulate an internal encoding; you can enforce that all strings you store in memory be valid UTF-8, enabling the library to make that assumption. Routines for validating and converting from/to UTF-8 are available free from Unicode, Inc.

Note that my routines do not need to support the many encodings of the world—the C library can handle that. If the current locale is not UTF-8, you call mbstowcs() on user input to convert any encoding (whatever it is) to a wide character string, then use my u8_toutf8() to convert it to the UTF-8 your program is comfortable with. Here’s an example input routine wrapping readline():

The first call to mbstowcs() uses the special parameter value NULL to find the number of characters in the opaque multibyte string.

Anyway, on with the routines. They are divided into four groups:

Group 1: conversions

Note that the library uses «unsigned int» as its wide character type.
You can convert a known number of bytes, or a NUL-terminated string. The length of a UTF-8 string is often communicated as a byte count, since that’s what really matters. Recall that you can usually treat a UTF-8 string like a normal C-string with N characters (where N is the number of bytes in the UTF-8 sequence), with the possibility that some characters are >127.

Group 2: moving through UTF-8 strings

Group 3: Unicode escape sequences
In the absence of Unicode input methods, Unicode characters are often notated using special escape sequences beginning with \u or \U. \u expects up to four hexadecimal digits, and \U expects up to eight. With these routines your program can accept input and give output using such sequences if necessary.