ASCII and Unicode

Содержание и основные понятия

Back to the table of contents
Alterna · Chapter 13 · page 13.2
ASCII and Unicode
From 7-bit telegraph to 1.1M+ Unicode characters: history, tables, converters
ASCII
American Standard Code for Information Interchange is the first standard code for characters.
Unicode
- her modern heir, including all the writings of the world.
GEF
PIF
OGE No. 2
EEG No. 4
01
Text coding history
From the telegraph to Unicode: How the character coding developed
GEF
OGE No. 2
Evolution of text coding
Bodo (1863)
5 bits, 32
ASCII (1963)
7 bits, 128
Ext.ASCII
8 bits, 256
Unicode
1.1M+ Symbols
UTF-8 (1993)
1-4 byte
Each of the following codes includes the previous one.
Bodo code (1863)
The first electric telegraph code developed by Emile-Bodot. 5 bits = 32 characters: 26 letters of Latin alphabet + 6 characters. Based on the fixed code length, unlike Morse's variable code.
I'm sorry.
It was Bodo who set the principle: one symbol = a fixed number of bits.
ASCII (1963) — American Standard Code for Information Interchange
1963 7 bits 128 characters
Established by the American National Institute of Standards (ANSI), he replaced Morse's telegraph code and became the first mass computer standard.
7 bits
code
128 characters
: 32 managers (NUL, TAB, LF, CR...) + 96 print (A-Z, a-z, 0-9, puncture marks)
Code managers (0-31, 127): Symbols that do not display on the screen but control the behaviour of the device (translation of the line, sound signal, return of the carriage)
Later extended to
8 bits
(Extended ASCII, 256 characters) - National alphabets added: Cyrillic, Greek, Arabic, etc.
I did.
de facto standard
for all computers in the world: any device "understands" ASCII
Curiously, the numbers 0-9 have codes 48-57, capital A-Z: 65-90, linen a-z: 97-122
KOI-8 (1987) - Information Exchange Code, 8 bits
1987 · 8 bits · 256 characters
==History==In the USSR, it was created for a cyrillica in Electronics 60 and USENET (Usenet-conferences), the name of which is "Information Exchange Code 8 bits".
The Russian letters are based on the principle
"code + 80"
: if you take the letter code A to ASCII and add 128 (0x80), you get her KOI-8 code.
A = 0x41
,
А = 0xC1
This stunt allowed programs to automatically switch between Latin and Cyrillic.
Was the main encoded in
Unix/Linux
(before mass transition to UTF-8)
Still used in some Linux systems and network protocols
Speciality: Russian lowercase letters stand
after
(in turn CP1251)
Windows-1251 / CP1251 (1990)
1990 · 8 bits · 256 characters
== sync, corrected by elderman == @elder_man
It's convenient that
Space = 0x20
,
figures = 0 x 30 - 0x39
as in ASCII — compatibility maintained
Russian letters are coming.
Subtotal
In alphabetical order (A=0xC0, B=0XC1...), as opposed to KOI-8
Has become the standard for
Windows systems
and still meets in old files, emails, databases.
Not : many old Russian-language websites still use CP1251
UTF-8 (1993) - Froben Format, RFC 2279
1993 1 - 4 bytes compatible with ASCII
Created by Ken Thompson and Rob Pike (Unix, Bell Labs).
Unicode Transformation Format — 8-bit
.
Reconciling with ASCII
: any ASCII symbol (0–127) is coded exactly by 1 byte — the same value
Latinica = 1 byte, Cyrillic = 2 bytes, CJK (Chinese) = 3 bytes and Emodi = 4 bytes
Now used in
98% web
- The most economical text for a mixed text
Doesn't require a BOM (Byte Order Mark) (Little-endian)
Prefix codes:
0xxxxxxx
- 1 byte,
110xxxxx 10xxxxxx
- Two bytes,
1110xxxx 10xxxxxx 10xxxxxx
- 3 bytes,
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
- 4 bytes.
UTF-16 (1991)
1991 · 2 or 4 bytes
One form of Unicode with
variable length
: 2 bytes for base range (BMP), 4 bytes to sublementary platforms.
Two options:
BE
(Big Indian) and
LE
Little Indian (Little Endian) is a different order of bytes.
BOM
(Byte Order Mark) =
FE FF
for BE,
FF FE
for LE - label at the beginning of the file
Used in
Windows API
,
Java
,
.NET
as an internal representation of the lines
Not recompensable with ASCII: ASC II symbol "A" (0x41) in UTF-16 LE =
41 00
UTF-32 (2003)
2003 · 4 bytes fixed
The simplest form is Unicode: each symbol = exactly 4 bytes.
Fixed length
: convenient for indexing - access to the N symbol for O(1)
It's the most
hungry.
: "A" = 00 00 00 41 (4 bytes instead of 1 in UTF-8)
Rare used in files but used in internal data structures
Doesn't require a BOM, the order of bytes is defined by the platform.
02
Symbol table
The ASCII Basic Table (0–127) and the extended (128–255) UTF-8 encodes the ASC II characters with the same bytes.
GEF
OGE No. 2
📋
Interactive character table
ASCII (0–127)
Extended (128–255)
All (0-255)
Press the box to see the details.
03
Unicode: UTF-8, UTF-16, UTF-32
Three forms of Unicode representation and their characteristics
GEF
EEG No. 4
Comparison of UTF codes
Codes
Byte/symbol
ASCII compatibility
Application
UTF-8
1–4
Yes (1 byte = ASCII)
Web, files, email
UTF-16
2 or 4
None
Windows API, Java, .NET
UTF-32
4
None
Internal data structures
Example: the word "hello" in different codings
UTF-8
6 symbols x 2 bytes =
12 bytes
UTF-16
6 symbols x 2 bytes =
12 bytes
UTF-32
6 symbols x 4 bytes =
24 bytes
🔄
Text-to-code converter
Enter the text and see it presented in ASCII/Unicode.
DEC:
HEX:
BIN:
Size:
🌍
Cross-code converter
Enter the text and select the original code. See how the same text looks in all other codes.
Text:
Reference text code:
04
Text volume calculator
Compute the volume of the text in different codings
🖩
Volume calculator
Number of symbols:
Code:
05
Mini test
📝
Check yourself out: 5 questions
0 / 500 XP
0/5