Unicode

Logo of the Unicode Consortium.

Unicode is a computing industry standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The latest version contains a repertoire of 136,755 characters covering 139 modern and historic scripts, as well as multiple symbol sets. The Unicode Standard is maintained in conjunction with ISO/IEC 10646, and both are code-for-code identical.

The Unicode Standard consists of a set of code charts for visual reference, an encoding method and set of standard character encodings, a set of reference data files, and a number of related items, such as character properties, rules for normalization, decomposition, collation, rendering, and bidirectional display order (for the correct display of text containing both right-to-left scripts, such as Arabic and Hebrew, and left-to-right scripts). [1] As of June 2017, the most recent version is Unicode 10.0. The standard is maintained by the Unicode Consortium.

Unicode's success at unifying character sets has led to its widespread and predominant use in the internationalization and localization of computer software. The standard has been implemented in many recent technologies, including modern operating systems, XML, Java (and other programming languages), and the .NET Framework.

Unicode can be implemented by different character encodings. The Unicode standard defines UTF-8, UTF-16, and UTF-32, and several other encodings are in use. The most commonly used encodings are UTF-8, UTF-16 and UCS-2, a precursor of UTF-16.

UTF-8, the most widely used by websites, uses one byte for the first 128 code points, and up to 4 bytes for other characters. The first 128 Unicode code points are the ASCII characters; so an ASCII text is a UTF-8 text.

UCS-2 simply uses two bytes (16 bits) for each character but can only encode the first 65,536 code points, the so-called Basic Multilingual Plane. With 1,114,112 code points on 17 planes being possible, and with over 120,000 code points defined so far, many Unicode characters are beyond the reach of UCS-2. Therefore, UCS-2 is obsolete, though still widely used in software. UTF-16 extends UCS-2, by using the same 16-bit encoding as UCS-2 for the Basic Multilingual Plane, and a 4-byte encoding for the other planes. Therefore, a UCS-2 text is a UTF-16 text.

UTF-32 (also referred to as UCS-4) uses four bytes for each character. Like UCS-2, the number of bytes per character is fixed, facilitating character indexing; but unlike UCS-2, UTF-32 is able to encode all Unicode code points. However, because each character uses four bytes, UTF-32 takes significantly more space than other encodings, and is not widely used.

Origin and development

Unicode has the explicit aim of transcending the limitations of traditional character encodings, such as those defined by the ISO 8859 standard, which find wide usage in various countries of the world but remain largely incompatible with each other. Many traditional character encodings share a common problem in that they allow bilingual computer processing (usually using Latin characters and the local script), but not multilingual computer processing (computer processing of arbitrary scripts mixed with each other).

Unicode, in intent, encodes the underlying characters— graphemes and grapheme-like units—rather than the variant glyphs (renderings) for such characters. In the case of Chinese characters, this sometimes leads to controversies over distinguishing the underlying character from its variant glyphs (see Han unification).

In text processing, Unicode takes the role of providing a unique code point—a number, not a glyph—for each character. In other words, Unicode represents a character in an abstract way and leaves the visual rendering (size, shape, font, or style) to other software, such as a web browser or word processor. This simple aim becomes complicated, however, because of concessions made by Unicode's designers in the hope of encouraging a more rapid adoption of Unicode.

The first 256 code points were made identical to the content of ISO-8859-1 so as to make it trivial to convert existing western text. Many essentially identical characters were encoded multiple times at different code points to preserve distinctions used by legacy encodings and therefore, allow conversion from those encodings to Unicode (and back) without losing any information. For example, the " fullwidth forms" section of code points encompasses a full Latin alphabet that is separate from the main Latin alphabet section because in Chinese, Japanese, and Korean ( CJK) fonts, these Latin characters are rendered at the same width as CJK ideographs, rather than at half the width. For other examples, see duplicate characters in Unicode.

History

Based on experiences with the Xerox Character Code Standard (XCCS) since 1980, [2] the origins of Unicode date to 1987, when Joe Becker from Xerox and Lee Collins and Mark Davis from Apple started investigating the practicalities of creating a universal character set. [3] With additional input from Peter Fenwick and Dave Opstad, [2] Joe Becker published a draft proposal for an "international/multilingual text character encoding system in August 1988, tentatively called Unicode". He explained that "[t]he name 'Unicode' is intended to suggest a unique, unified, universal encoding". [2]

In this document, entitled Unicode 88, Becker outlined a 16-bit character model: [2]

Unicode is intended to address the need for a workable, reliable world text encoding. Unicode could be roughly described as "wide-body ASCII" that has been stretched to 16 bits to encompass the characters of all the world's living languages. In a properly engineered design, 16 bits per character are more than sufficient for this purpose.

His original 16-bit design was based on the assumption that only those scripts and characters in modern use would need to be encoded: [2]

Unicode gives higher priority to ensuring utility for the future than to preserving past antiquities. Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is undoubtedly far below 214 = 16,384. Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally useful Unicodes.

In early 1989, the Unicode working group expanded to include Ken Whistler and Mike Kernaghan of Metaphor, Karen Smith-Yoshimura and Joan Aliprand of RLG, and Glenn Wright of Sun Microsystems, and in 1990, Michel Suignard and Asmus Freytag from Microsoft and Rick McGowan of NeXT joined the group. By the end of 1990, most of the work on mapping existing character encoding standards had been completed, and a final review draft of Unicode was ready.

The Unicode Consortium was incorporated in California on January 3, 1991, [4] and in October 1991, the first volume of the Unicode standard was published. The second volume, covering Han ideographs, was published in June 1992.

In 1996, a surrogate character mechanism was implemented in Unicode 2.0, so that Unicode was no longer restricted to 16 bits. This increased the Unicode codespace to over a million code points, which allowed for the encoding of many historic scripts (e.g., Egyptian Hieroglyphs) and thousands of rarely used or obsolete characters that had not been anticipated as needing encoding. Among the characters not originally intended for Unicode are rarely used Kanji or Chinese characters, many of which are part of personal and place names, making them rarely used, but much more essential than envisioned in the original architecture of Unicode. [5]

The Microsoft TrueType specification version 1.0 from 1992 used the name Apple Unicode instead of Unicode for the Platform ID in the naming table.

Architecture and terminology

Unicode defines a codespace of 1,114,112 code points in the range 0hex to 10FFFFhex. [6] Normally a Unicode code point is referred to by writing "U+" followed by its hexadecimal number. For code points in the Basic Multilingual Plane (BMP), four digits are used (e.g., U+0058 for the character LATIN CAPITAL LETTER X); for code points outside the BMP, five or six digits are used, as required (e.g., U+E0001 for the character LANGUAGE TAG and U+10FFFD for the character PRIVATE USE CHARACTER-10FFFD). [7]

Code point planes and blocks

The Unicode codespace is divided into seventeen planes, numbered 0 to 16:

All code points in the BMP are accessed as a single code unit in UTF-16 encoding and can be encoded in one, two or three bytes in UTF-8. Code points in Planes 1 through 16 (supplementary planes) are accessed as surrogate pairs in UTF-16 and encoded in four bytes in UTF-8.

Within each plane, characters are allocated within named blocks of related characters. Although blocks are an arbitrary size, they are always a multiple of 16 code points and often a multiple of 128 code points. Characters required for a given script may be spread out over several different blocks.

General Category property

Each code point has a single General Category property. The major categories are denoted: Letter, Mark, Number, Punctuation, Symbol, Separator and Other. Within these categories, there are subdivisions. The General Category is not useful for every use, since legacy encodings have used multiple characteristics per single code point. E.g., U+000A <control-000A> Line feed (LF) in ASCII is both a control and a formatting separator; in Unicode the General Category is "Other, Control". Often, other properties must be used to specify the characteristics and behaviour of a code point. The possible General Categories are:

Code points in the range U+D800–U+DBFF (1,024 code points) are known as high-surrogate code points, and code points in the range U+DC00–U+DFFF (1,024 code points) are known as low-surrogate code points. A high-surrogate code point (also known as a leading surrogate) followed by a low-surrogate code point (also known as a trailing surrogate) together form a surrogate pair used in UTF-16 to represent 1,048,576 code points outside BMP. High and low surrogate code points are not valid by themselves. Thus the range of code points that are available for use as characters is U+0000–U+D7FF and U+E000–U+10FFFF (1,112,064 code points). The value of these code points (i.e., excluding surrogates) is sometimes referred to as the character's scalar value.

Certain non-character code points are guaranteed never to be used for encoding characters, although applications may make use of these code points internally if they wish. There are sixty-six noncharacters: U+FDD0–U+FDEF and any code point ending in the value FFFE or FFFF (i.e., U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, … U+10FFFE, U+10FFFF). The set of noncharacters is stable, and no new noncharacters will ever be defined. [12]

Reserved code points are those code points which are available for use as encoded characters, but are not yet defined as characters by Unicode.

Private-use code points are considered to be assigned characters, but they have no interpretation specified by the Unicode standard [13] so any interchange of such characters requires an agreement between sender and receiver on their interpretation. There are three private-use areas in the Unicode codespace:

  • Private Use Area: U+E000–U+F8FF (6,400 characters)
  • Supplementary Private Use Area-A: U+F0000–U+FFFFD (65,534 characters)
  • Supplementary Private Use Area-B: U+100000–U+10FFFD (65,534 characters).

Graphic characters are characters defined by Unicode to have a particular semantic, and either have a visible glyph shape or represent a visible space. As of Unicode 10.0 there are 136,537 graphic characters.

Format characters are characters that do not have a visible appearance, but may have an effect on the appearance or behavior of neighboring characters. For example, U+200C Zero width non-joiner and U+200D Zero width joiner may be used to change the default shaping behavior of adjacent characters (e.g., to inhibit ligatures or request ligature formation). There are 153 format characters in Unicode 10.0.

Sixty-five code points (U+0000–U+001F and U+007F–U+009F) are reserved as control codes, and correspond to the C0 and C1 control codes defined in ISO/IEC 6429. Of these U+0009 (Tab), U+000A (Line Feed), and U+000D (Carriage Return) are widely used in Unicode-encoded texts.

Graphic characters, format characters, control code characters, and private use characters are known collectively as assigned characters.

Abstract characters

The set of graphic and format characters defined by Unicode does not correspond directly to the repertoire of abstract characters that is representable under Unicode. Unicode encodes characters by associating an abstract character with a particular code point. [14] However, not all abstract characters are encoded as a single Unicode character, and some abstract characters may be represented in Unicode by a sequence of two or more characters. For example, a Latin small letter "i" with an ogonek, a dot above, and an acute accent, which is required in Lithuanian, is represented by the character sequence U+012F, U+0307, U+0301. Unicode maintains a list of uniquely named character sequences for abstract characters that are not directly encoded in Unicode. [15]

All graphic, format, and private use characters have a unique and immutable name by which they may be identified. This immutability has been guaranteed since Unicode version 2.0 by the Name Stability policy. [12] In cases where the name is seriously defective and misleading, or has a serious typographical error, a formal alias may be defined, and applications are encouraged to use the formal alias in place of the official character name. For example, U+A015 YI SYLLABLE WU has the formal alias yi syllable iteration mark, and U+FE18 PRESENTATION FORM FOR VERTICAL RIGHT WHITE LENTICULAR BRAKCET (sic) has the formal alias presentation form for vertical right white lenticular bracket. [16]

Unicode Consortium

The Unicode Consortium is a nonprofit organization that coordinates Unicode's development. Full members include most of the main computer software and hardware companies with any interest in text-processing standards, including Adobe Systems, Apple, Google, IBM, Microsoft, Oracle Corporation, and Yahoo!. [17]

The Consortium has the ambitious goal of eventually replacing existing character encoding schemes with Unicode and its standard Unicode Transformation Format (UTF) schemes, as many of the existing schemes are limited in size and scope and are incompatible with multilingual environments.

Versions

Unicode is developed in conjunction with the International Organization for Standardization and shares the character repertoire with ISO/IEC 10646: the Universal Character Set. Unicode and ISO/IEC 10646 function equivalently as character encodings, but The Unicode Standard contains much more information for implementers, covering—in depth—topics such as bitwise encoding, collation and rendering. The Unicode Standard enumerates a multitude of character properties, including those needed for supporting bidirectional text. The two standards do use slightly different terminology.

The Consortium first published The Unicode Standard ( ISBN  0-321-18578-1) in 1991 and continues to develop standards based on that original work. The latest version of the standard, Unicode 10.0, was released in June 2017 and is available from the consortium's website. The last of the major versions (versions x.0) to be published in book form was Unicode 5.0 ( ISBN  0-321-48091-0), but since Unicode 6.0 the full text of the standard is no longer being published in book form. In 2012, however, it was announced that only the core specification for Unicode version 6.1 would be made available as a 692-page print-on-demand paperback. [18] Unlike the previous major version printings of the Standard, the print-on-demand core specification does not include any code charts or standard annexes, but the entire standard, including the core specification, will still remain freely available on the Unicode website.

Thus far, the following major and minor versions of the Unicode standard have been published. Update versions, which do not include any changes to character repertoire, are signified by the third number (e.g., "version 4.0.1") and are omitted in the table below. [19]

Unicode versions
Version Date Book Corresponding ISO/IEC 10646 Edition Scripts Characters
Total [tablenote 1] Notable additions
1.0.0 October 1991 ISBN  0-201-56788-1 (Vol.1) 24 7,161 Initial repertoire covers these scripts: Arabic, Armenian, Bengali, Bopomofo, Cyrillic, Devanagari, Georgian, Greek and Coptic, Gujarati, Gurmukhi, Hangul, Hebrew, Hiragana, Kannada, Katakana, Lao, Latin, Malayalam, Oriya, Tamil, Telugu, Thai, and Tibetan. [20]
1.0.1 June 1992 ISBN  0-201-60845-6 (Vol.2) 25 28,359 The initial set of 20,902 CJK Unified Ideographs is defined. [21]
1.1 June 1993 ISO/IEC 10646-1:1993 24 34,233 4,306 more Hangul syllables added to original set of 2,350 characters. Tibetan removed. [22]
2.0 July 1996 ISBN  0-201-48345-9 ISO/IEC 10646-1:1993 plus Amendments 5, 6 and 7 25 38,950 Original set of Hangul syllables removed, and a new set of 11,172 Hangul syllables added at a new location. Tibetan added back in a new location and with a different character repertoire. Surrogate character mechanism defined, and Plane 15 and Plane 16 Private Use Areas allocated. [23]
2.1 May 1998 ISO/IEC 10646-1:1993 plus Amendments 5, 6 and 7, as well as two characters from Amendment 18 25 38,952 Euro sign and Object Replacement Character added. [24]
3.0 September 1999 ISBN  0-201-61633-5 ISO/IEC 10646-1:2000 38 49,259 Cherokee, Ethiopic, Khmer, Mongolian, Burmese, Ogham, Runic, Sinhala, Syriac, Thaana, Unified Canadian Aboriginal Syllabics, and Yi Syllables added, as well as a set of Braille patterns. [25]
3.1 March 2001 ISO/IEC 10646-1:2000

ISO/IEC 10646-2:2001

41 94,205 Deseret, Gothic and Old Italic added, as well as sets of symbols for Western music and Byzantine music, and 42,711 additional CJK Unified Ideographs. [26]
3.2 March 2002 ISO/IEC 10646-1:2000 plus Amendment 1

ISO/IEC 10646-2:2001

45 95,221 Philippine scripts Buhid, Hanunó'o, Tagalog, and Tagbanwa added. [27]
4.0 April 2003 ISBN  0-321-18578-1 ISO/IEC 10646:2003 52 96,447 Cypriot syllabary, Limbu, Linear B, Osmanya, Shavian, Tai Le, and Ugaritic added, as well as Hexagram symbols. [28]
4.1 March 2005 ISO/IEC 10646:2003 plus Amendment 1 59 97,720 Buginese, Glagolitic, Kharoshthi, New Tai Lue, Old Persian, Syloti Nagri, and Tifinagh added, and Coptic was disunified from Greek. Ancient Greek numbers and musical symbols were also added. [29]
5.0 July 2006 ISBN  0-321-48091-0 ISO/IEC 10646:2003 plus Amendments 1 and 2, as well as four characters from Amendment 3 64 99,089 Balinese, Cuneiform, N'Ko, Phags-pa, and Phoenician added. [30]
5.1 April 2008 ISO/IEC 10646:2003 plus Amendments 1, 2, 3 and 4 75 100,713 Carian, Cham, Kayah Li, Lepcha, Lycian, Lydian, Ol Chiki, Rejang, Saurashtra, Sundanese, and Vai added, as well as sets of symbols for the Phaistos Disc, Mahjong tiles, and Domino tiles. There were also important additions for Burmese, additions of letters and Scribal abbreviations used in medieval manuscripts, and the addition of Capital ẞ. [31]
5.2 October 2009 ISO/IEC 10646:2003 plus Amendments 1, 2, 3, 4, 5 and 6 90 107,361 Avestan, Bamum, Egyptian hieroglyphs (the Gardiner Set, comprising 1,071 characters), Imperial Aramaic, Inscriptional Pahlavi, Inscriptional Parthian, Javanese, Kaithi, Lisu, Meetei Mayek, Old South Arabian, Old Turkic, Samaritan, Tai Tham and Tai Viet added. 4,149 additional CJK Unified Ideographs (CJK-C), as well as extended Jamo for Old Hangul, and characters for Vedic Sanskrit. [32]
6.0 October 2010 ISO/IEC 10646:2010 plus the Indian rupee sign 93 109,449 Batak, Brahmi, Mandaic, playing card symbols, transport and map symbols, alchemical symbols, emoticons and emoji. 222 additional CJK Unified Ideographs (CJK-D) added. [33]
6.1 January 2012 ISO/IEC 10646:2012 100 110,181 Chakma, Meroitic cursive, Meroitic hieroglyphs, Miao, Sharada, Sora Sompeng, and Takri. [34]
6.2 September 2012 ISO/IEC 10646:2012 plus the Turkish lira sign 100 110,182 Turkish lira sign. [35]
6.3 September 2013 ISO/IEC 10646:2012 plus six characters 100 110,187 5 bidirectional formatting characters. [36]
7.0 June 2014 ISO/IEC 10646:2012 plus Amendments 1 and 2, as well as the Ruble sign 123 113,021 Bassa Vah, Caucasian Albanian, Duployan, Elbasan, Grantha, Khojki, Khudawadi, Linear A, Mahajani, Manichaean, Mende Kikakui, Modi, Mro, Nabataean, Old North Arabian, Old Permic, Pahawh Hmong, Palmyrene, Pau Cin Hau, Psalter Pahlavi, Siddham, Tirhuta, Warang Citi, and Dingbats. [37]
8.0 June 2015 ISO/IEC 10646:2014 plus Amendment 1, as well as the Lari sign, nine CJK unified ideographs, and 41 emoji characters [38] 129 120,737 Ahom, Anatolian hieroglyphs, Hatran, Multani, Old Hungarian, SignWriting, 5,771 CJK unified ideographs, a set of lowercase letters for Cherokee, and five emoji skin tone modifiers [39]
9.0 June 2016 ISO/IEC 10646:2014 plus Amendments 1 and 2, as well as Adlam, Newa, Japanese TV symbols, and 74 emoji and symbols [40] 135 128,237 Adlam, Bhaiksuki, Marchen, Newa, Osage, Tangut, and 72 emoji [41] [42]
10.0 June 2017 ISO/IEC 10646:2017 plus 56 emoji characters, 285 hentaigana characters, and 3 Zanabazar Square characters [43] 139 136,755 Zanabazar Square, Soyombo, Masaram Gondi, Nushu, hentaigana (non-standard hiragana), 7,494 CJK unified ideographs, and 56 emoji [44]
  1. ^ The number of characters listed for each version of Unicode is the total number of graphic, format and control characters (i.e., excluding private-use characters, noncharacters and surrogate code points).

Scripts covered

Many modern applications can render a substantial subset of the many scripts in Unicode, as demonstrated by this screenshot from the OpenOffice.org application.

Unicode covers almost all scripts ( writing systems) in current use today. [45][ not in citation given]

A total of 139 scripts are included in the latest version of Unicode (covering alphabets, abugidas and syllabaries), although there are still scripts that are not yet encoded, particularly those mainly used in historical, liturgical, and academic contexts. Further additions of characters to the already encoded scripts, as well as symbols, in particular for mathematics and music (in the form of notes and rhythmic symbols), also occur.

The Unicode Roadmap Committee ( Unicode Roadmap page of the Unicode Consortium Web site. For some scripts on the Roadmap, such as Jurchen and Khitan small script, encoding proposals have been made and they are working their way through the approval process. For others scripts, such as Mayan and Rongorongo, no proposal has yet been made, and they await agreement on character repertoire and other details from the user communities involved.

Some modern invented scripts which have not yet been included in Unicode (e.g., Tengwar) or which do not qualify for inclusion in Unicode due to lack of real-world use (e.g., Klingon) are listed in the ConScript Unicode Registry, along with unofficial but widely used Private Use Area code assignments.

There is also a Medieval Unicode Font Initiative focused on special Latin medieval characters. Part of these proposals have been already included into Unicode.

The Script Encoding Initiative, a project run by Deborah Anderson at the University of California, Berkeley was founded in 2002 with the goal of funding proposals for scripts not yet encoded in the standard. The project has become a major source of proposed additions to the standard in recent years. [46]

Other Languages
Afrikaans: Unicode
Alemannisch: Unicode
አማርኛ: ዩኒኮድ
العربية: يونيكود
অসমীয়া: ইউনিক’ড
azərbaycanca: Unicode
বাংলা: ইউনিকোড
Bân-lâm-gú: Unicode
беларуская: Унікод
беларуская (тарашкевіца)‎: Юнікод
български: Уникод
Boarisch: Unicode
bosanski: Unicode
brezhoneg: Unicode
català: Unicode
Чӑвашла: Юникод
čeština: Unicode
Cymraeg: Unicode
dansk: Unicode
Deutsch: Unicode
eesti: Unicode
Ελληνικά: Γιούνικοντ
español: Unicode
Esperanto: Unikodo
euskara: Unicode
فارسی: یونی‌کد
français: Unicode
Gaeilge: Unicode
galego: Unicode
ગુજરાતી: યુનિકોડ
한국어: 유니코드
Հայերեն: Յունիկոդ
हिन्दी: यूनिकोड
hrvatski: Unikod
Ilokano: Unicode
Bahasa Indonesia: Unicode
interlingua: Unicode
íslenska: Unicode
italiano: Unicode
עברית: יוניקוד
Basa Jawa: Unicode
ಕನ್ನಡ: ಯುನಿಕೋಡ್
ქართული: უნიკოდი
कॉशुर / کٲشُر: यूनिकोड
қазақша: Юникод
Kurdî: Unicode
Кыргызча: Юникод
latviešu: Unikods
lietuvių: Unikodas
magyar: Unicode
മലയാളം: യൂണികോഡ്
मराठी: युनिकोड
მარგალური: იუნიკოდი
Bahasa Melayu: Unicode
монгол: Юникод
မြန်မာဘာသာ: ယူနီကုဒ်
Nederlands: Unicode
नेपाली: युनिकोड
नेपाल भाषा: युनिकोड
日本語: Unicode
norsk: Unicode
norsk nynorsk: Unicode
occitan: Unicode
олык марий: Unicode
ਪੰਜਾਬੀ: ਯੂਨੀਕੋਡ
Plattdüütsch: Unicode
polski: Unikod
português: Unicode
română: Unicode
русский: Юникод
саха тыла: Юникод
संस्कृतम्: युनिकोड
Scots: Unicode
shqip: Unicode
සිංහල: යුනිකෝඩ්
Simple English: Unicode
slovenčina: Unicode
slovenščina: Unicode
کوردی: یوونیکۆد
српски / srpski: Unikod
srpskohrvatski / српскохрватски: Unikod
Basa Sunda: Unicode
suomi: Unicode
svenska: Unicode
Tagalog: Unikodigo
తెలుగు: యూనికోడ్
тоҷикӣ: Юникод
ᏣᎳᎩ: ᏳᏂᎪᏛ
Türkçe: Unicode
українська: Юнікод
اردو: یونیکوڈ
ئۇيغۇرچە / Uyghurche: Unicode
Tiếng Việt: Unicode
walon: Unicôde
吴语: Unicode
ייִדיש: יוניקאד
Yorùbá: Unicode
粵語: 統一碼
中文: Unicode