UTF-8, a transformation format of Unicode and ISO 10646
A byte-oriented encoding that represents Unicode characters while preserving ASCII compatibility.
Yergeau, F., "UTF-8, a transformation format of Unicode and ISO 10646", RFC 2044, DOI 10.17487/RFC2044, October 1996, <https://www.rfc-editor.org/info/rfc2044>.
@misc{rfc2044,
series = {Request for Comments},
number = 2044,
howpublished = {RFC 2044},
publisher = {RFC Editor},
doi = {10.17487/RFC2044},
url = {https://www.rfc-editor.org/info/rfc2044},
author = {F. Yergeau},
title = {{UTF-8, a transformation format of Unicode and ISO 10646}},
pagetotal = 6,
year = 1996,
month = oct,
} RFC text
Contents
Network Working Group F. Yergeau
Request for Comments: 2044 Alis Technologies
Category: Informational October 1996
UTF-8, a transformation format of Unicode and ISO 10646
Status of this Memo
This memo provides information for the Internet community. This memo
does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.
Abstract
The Unicode Standard, version 1.1, and ISO/IEC 10646-1:1993 jointly
define a 16 bit character set which encompasses most of the world's
writing systems. 16-bit characters, however, are not compatible with
many current applications and protocols, and this has led to the
development of a few so-called UCS transformation formats (UTF), each
with different characteristics. UTF-8, the object of this memo, has
the characteristic of preserving the full US-ASCII range: US-ASCII
characters are encoded in one octet having the usual US-ASCII value,
and any octet with such a value can only be an US-ASCII character.
This provides compatibility with file systems, parsers and other
software that rely on US-ASCII values but are transparent to other
values.
1. Introduction
The Unicode Standard, version 1.1 [UNICODE], and ISO/IEC 10646-1:1993
[ISO-10646] jointly define a 16 bit character set, UCS-2, which
encompasses most of the world's writing systems. ISO 10646 further
defines a 31-bit character set, UCS-4, with currently no assignments
outside of the region corresponding to UCS-2 (the Basic Multilingual
Plane, BMP). The UCS-2 and UCS-4 encodings, however, are hard to use
in many current applications and protocols that assume 8 or even 7
bit characters. Even newer systems able to deal with 16 bit
characters cannot process UCS-4 data. This situation has led to the
development of so-called UCS transformation formats (UTF), each with
different characteristics.
UTF-1 has only historical interest, having been removed from ISO
10646. UTF-7 has the quality of encoding the full Unicode repertoire
using only octets with the high-order bit clear (7 bit US-ASCII
values, [US-ASCII]), and is thus deemed a mail-safe encoding
([RFC1642]). UTF-8, the object of this memo, uses all bits of an
octet, but has the quality of preserving the full US-ASCII range:
US-ASCII characters are encoded in one octet having the normal US-
ASCII value, and any octet with such a value can only stand for an
US-ASCII character, and nothing else.
UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire
into a pair of UCS-2 values from a reserved range. UTF-16 impacts
UTF-8 in that UCS-2 values from the reserved range must be treated
specially in the UTF-8 transformation.
UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of
octets, where the number of octets, and the value of each, depend on
the integer value assigned to the character in ISO 10646. This
transformation format has the following characteristics (all values
are in hexadecimal):
- Character values from 0000 0000 to 0000 007F (US-ASCII repertoire)
correspond to octets 00 to 7F (7 bit US-ASCII values).
- US-ASCII values do not appear otherwise in a UTF-8 encoded charac-
ter stream. This provides compatibility with file systems or
other software (e.g. the printf() function in C libraries) that
parse based on US-ASCII values but are transparent to other val-
ues.
- Round-trip conversion is easy between UTF-8 and either of UCS-4,
UCS-2 or Unicode.
- The first octet of a multi-octet sequence indicates the number of
octets in the sequence.
- Character boundaries are easily found from anywhere in an octet
stream.
- The lexicographic sorting order of UCS-4 strings is preserved. Of
course this is of limited interest since the sort order is not
culturally valid in either case.
- The octet values FE and FF never appear.
UTF-8 was originally a project of the X/Open Joint
Internationalization Group XOJIG with the objective to specify a File
System Safe UCS Transformation Format [FSS-UTF] that is compatible
with UNIX systems, supporting multilingual text in a single encoding.
The original authors were Gary Miller, Greger Leijonhufvud and John
Entenmann. Later, Ken Thompson and Rob Pike did significant work for
the formal UTF-8.
A description can also be found in Unicode Technical Report #4 [UNI-
CODE]. The definitive reference, including provisions for UTF-16
data within UTF-8, is Annex R of ISO/IEC 10646-1 [ISO-10646].
2. UTF-8 definition
In UTF-8, characters are encoded using sequences of 1 to 6 octets.
The only octet of a "sequence" of one has the higher-order bit set to
0, the remaining 7 bits being used to encode the character value. In
a sequence of n octets, n>1, the initial octet has the n higher-order
bits set to 1, followed by a bit set to 0. The remaining bit(s) of
that octet contain bits from the value of the character to be
encoded. The following octet(s) all have the higher-order bit set to
1 and the following bit set to 0, leaving 6 bits in each to contain
bits from the character to be encoded.
The table below summarizes the format of these different octet types.
The letter x indicates bits available for encoding bits of the UCS-4
character value.
UCS-4 range (hex.) UTF-8 octet sequence (binary)
0000 0000-0000 007F 0xxxxxxx
0000 0080-0000 07FF 110xxxxx 10xxxxxx
0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx
0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0400 0000-7FFF FFFF 1111110x 10xxxxxx ... 10xxxxxx
Encoding from UCS-4 to UTF-8 proceeds as follows:
1) Determine the number of octets required from the character value
and the first column of the table above.
2) Prepare the high-order bits of the octets as per the second column
of the table.
3) Fill in the bits marked x from the bits of the character value,
starting from the lower-order bits of the character value and
putting them first in the last octet of the sequence, then the
next to last, etc. until all x bits are filled in.
The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be
obtained from the above, in principle, by simply extending each
UCS-2 character with two zero-valued octets. However, UCS-2 val-
ues between D800 and DFFF, being actually UCS-4 characters trans-
formed through UTF-16, need special treatment: the UTF-16 trans-
formation must be undone, yielding a UCS-4 character that is then
transformed as above.
Decoding from UTF-8 to UCS-4 proceeds as follows:
1) Initialize the 4 octets of the UCS-4 character with all bits set
to 0.
2) Determine which bits encode the character value from the number of
octets in the sequence and the second column of the table above
(the bits marked x).
3) Distribute the bits from the sequence to the UCS-4 character,
first the lower-order bits from the last octet of the sequence and
proceeding to the left until no x bits are left.
If the UTF-8 sequence is no more than three octets long, decoding
can proceed directly to UCS-2 (or equivalently Unicode).
A more detailed algorithm and formulae can be found in [FSS_UTF],
[UNICODE] or Annex R to [ISO-10646].
3. Examples
The Unicode sequence "A<NOT IDENTICAL TO><ALPHA>." (0041, 2262, 0391,
002E) may be encoded as follows:
41 E2 89 A2 CE 91 2E
The Unicode sequence "Hi Mom <WHITE SMILING FACE>!" (0048, 0069,
0020, 004D, 006F, 006D, 0020, 263A, 0021) may be encoded as follows:
48 69 20 4D 6F 6D 20 E2 98 BA 21
The Unicode sequence representing the Han characters for the Japanese
word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows:
E6 97 A5 E6 9C AC E8 AA 9E
MIME registrations
This memo is meant to serve as the basis for registration of a MIME
character encoding (charset) as per [RFC1521]. The proposed charset
parameter value is "UTF-8". This string would label media types
containing text consisting of characters from the repertoire of ISO
10646-1 encoded to a sequence of octets using the encoding scheme
outlined above.
Security Considerations
Security issues are not discussed in this memo.
Acknowledgments
The following have participated in the drafting and discussion of
this memo:
James E. Agenbroad Andries Brouwer
Martin J. D|rst David Goldsmith
Edwin F. Hart Kent Karlsson
Markus Kuhn Michael Kung
Alain LaBonte Murray Sargent
Keld Simonsen Arnold Winkler
Bibliography
[FSS_UTF] X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm.
22p. pbk. 172g. 4/95, X/Open Company Ltd., "File Sys-
tem Safe UCS Transformation Format (FSS_UTF)", X/Open
Preleminary Specification, Document Number P316. Also
published in Unicode Technical Report #4.
[ISO-10646] ISO/IEC 10646-1:1993. International Standard -- Infor-
mation technology -- Universal Multiple-Octet Coded
Character Set (UCS) -- Part 1: Architecture and Basic
Multilingual Plane. UTF-8 is described in Annex R,
adopted but not yet published. UTF-16 is described in
Annex Q, adopted but not yet published.
[RFC1521] Borenstein, N., and N. Freed, "MIME (Multipurpose
Internet Mail Extensions) Part One: Mechanisms for
Specifying and Describing the Format of Internet Mes-
sage Bodies", RFC 1521, Bellcore, Innosoft, September
1993.
[RFC1641] Goldsmith, D., and M. Davis, "Using Unicode with
MIME", RFC 1641, Taligent inc., July 1994.
[RFC1642] Goldsmith, D., and M. Davis, "UTF-7: A Mail-safe
Transformation Format of Unicode", RFC 1642,
Taligent, Inc., July 1994.
[UNICODE] The Unicode Consortium, "The Unicode Standard --
Worldwide Character Encoding -- Version 1.0", Addison-
Wesley, Volume 1, 1991, Volume 2, 1992. UTF-8 is
described in Unicode Technical Report #4.
[US-ASCII] Coded Character Set--7-bit American Standard Code for
Information Interchange, ANSI X3.4-1986.
Author's Address
Francois Yergeau
Alis Technologies
100, boul. Alexis-Nihon
Suite 600
Montreal QC H4M 2P2
Canada
Tel: +1 (514) 747-2547
Fax: +1 (514) 747-2561
EMail: fyergeau@alis.com
AI ELI5
On this page
§1The Challenge of Unicode
Unicode and ISO 10646 define character sets with 65,536 characters (16-bit UCS-2) and up to 2 billion characters (31-bit UCS-4), encompassing writing systems from around the world. However, these wide character sets create a problem: most applications, protocols, file systems, and network code assume 8-bit bytes or even 7-bit ASCII. Sending 16-bit characters through email, over networks, or into parsers built for 7-bit text breaks compatibility.
Several UCS transformation formats (UTF) were proposed to solve this. UTF-1 became obsolete. UTF-7 encodes the full Unicode repertoire using only 7-bit ASCII-safe bytes but is inefficient. UTF-16 uses 16-bit units for some characters and surrogate pairs for others. UTF-8, the subject of this memo, uses variable-length byte sequences while preserving full ASCII compatibility.
The world has many languages with many more characters than English. Unicode tried to fit them all into 16-bit numbers, but that doesn't work with the networks and programs we have, which work best with simple bytes. UTF-8 found a clever trick: use variable-length byte sequences that keep English the same.
§2How UTF-8 Encoding Works
UTF-8 encodes characters using 1 to 6 bytes. For ASCII characters (0x00 to 0x7F), a single byte is used with the high bit set to 0. For multi-byte sequences, the first byte has leading 1 bits indicating the sequence length, followed by a 0 bit (for example, 110xxxxx for a 2-byte sequence). Continuation bytes in the sequence always start with 10xxxxxx, leaving 6 bits for the character value in each continuation byte.
For example, ASCII 'A' (0x41) encodes as the single byte 0x41. The Euro sign (U+20AC) encodes as three bytes: 0xE2 0x82 0xAC. The pattern allows a decoder to determine the sequence length from the first byte alone, find character boundaries by scanning for start bytes (without 10 prefix), and detect truncation or invalid sequences.
Round-trip conversion between UTF-8 and UCS-2 or UCS-4 is straightforward. The only special case is that UCS-2 values in the range 0xD800-0xDFFF, which are UTF-16 surrogate pairs in ISO 10646, require special handling: the surrogate pair must be converted to a UCS-4 character before encoding to UTF-8.
UTF-8 uses variable-length byte groups. English stays one byte. Other letters from other languages become 2, 3, or more bytes. The first byte of a group tells you how long the group is, so programs can always tell where one character starts and the next begins.
Character range UTF-8 byte sequence
0x00-0x7F (ASCII) 0xxxxxxx
0x80-0x7FF 110xxxxx 10xxxxxx
0x800-0xFFFF 1110xxxx 10xxxxxx 10xxxxxx
0x10000-0x1FFFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
0x200000-0x3FFFFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
0x4000000-0x7FFFFFFF 1111110x 10xxxxxx ... 10xxxxxx§3Key Properties and Advantages
UTF-8 has several critical properties that made it ideal for internet protocols. US-ASCII values (0x00 to 0x7F) never appear in multi-byte sequences, only as single-byte ASCII characters. This means any octet with the high bit clear (bytes 0x00 to 0x7F) is always ASCII, providing transparency to ASCII-based systems like file systems, parsers (printf in C), and network protocols.
Character boundaries are easy to find by scanning for start bytes (those not starting with 10). UTF-8 preserves the lexicographic sort order of UCS-4 strings, meaning strings with multi-byte characters sort the same way as their UCS-4 representation. The octet values 0xFE and 0xFF never appear in UTF-8, which is useful for distinguishing UTF-8 from UTF-16 encodings.
These properties combine to make UTF-8 the encoding of choice for environments that historically relied on 8-bit bytes: Unix systems, file systems, email, and the web. Unlike UTF-16, UTF-8 requires no byte-order mark and poses no problems with endianness. Unlike UTF-7, it is efficient for ASCII-heavy content.
UTF-8 is clever: English text looks exactly the same. Programs that don't know about UTF-8 still work fine with English. And you can always find where one character ends and the next starts just by looking at the bytes.
§4Encoding Examples
The RFC provides concrete examples of UTF-8 encoding. The ASCII letter 'A' (U+0041) encodes as a single byte: 0x41. The space and period also encode as single bytes: 0x20 and 0x2E.
The character 'α' (U+03B1, Greek small letter alpha) encodes as two bytes: 0xCE 0xB1. The Unicode character 'β' (U+03B2) encodes as 0xCE 0xB2.
Multi-byte examples include the Euro sign '€' (U+20AC), which encodes as 0xE2 0x82 0xAC. The Japanese Hiragana 'A' (U+3042 Hiragana Letter A) encodes as 0xE3 0x81 0x82. These examples illustrate how the byte sequence length increases for characters with larger code points, yet all ASCII characters remain single bytes.
An English letter like 'A' is one byte: 0x41. A Greek letter is two bytes. A Japanese character is three bytes. You can tell how many bytes a letter needs just by looking at the first byte.
§5MIME Charset Registration
This memo serves as the basis for registering UTF-8 as a MIME charset parameter. The proposed name is 'UTF-8', allowing email, web pages, and other MIME content to declare that their text is encoded in UTF-8. This registration enables mail clients, web browsers, and other applications to correctly handle internationalized content while preserving backward compatibility with ASCII.
The UTF-8 charset registration allows Internet protocols and applications to standardize on a single encoding for Unicode, eliminating confusion between UTF-7, UTF-16, and other variants. By October 1996, UTF-8 was ready to become the standard way to send international text over the internet.
UTF-8 got a name so programs could recognize it. Email and web pages can now say 'this text is UTF-8' and programs know how to read it correctly.