Alphabetic and informative approaches

Содержание и основные понятия

Back to the table of contents
Alterna · Chapter 10 · page 10.3
Alphabetic and informative approaches
Two approaches to measuring information: through the strength of the alphabet and through the probability of symbols.
Informatics has two approaches to measuring information:
alphabetical
(through the power of the alphabet) and
informative
(through the probability of symbols) they produce different results for mixed texts.
GEF
PIF
OGE No. 3.4
EEG No. 4
01
Alphabetical approach
Measuring information through the power of the alphabet
GEF
OGE No. 3.4
The essence of the alphabetical approach
Each symbol of the power alphabet N carries
Same
Number of information:
i = log
2
(N)
Interpretation of text:
I = K × i
where K is the number of symbols.
I = K × log
2
(N)
.
Benefits and disadvantages
Benefits
Easy calculation
Computer-based coding
Fixed code length
Deficiencies
Doesn't take into account the frequency of symbols
Could be overrated.
Not optimal for compression
When is the alphabetical approach accurate?
An Alphabetic Approach Gives
Exact
the result when all the alphabet symbols
Equivalent
(talking at the same frequency).
Example: A binary code, where 0 and 1 meet equally frequently.
02
Contentary approach
Measuring information through the probability of symbols
GEF
EEG No. 4
The essence of a meaningful approach
Every symbol carries
Miscellaneous
The number of information according to its probability of being available:
i = −log
2
(p)
where
p
- the possibility of a symbol appearing in the message.
Example: Russian text
In the Russian text, letters are found at different frequencies:
Letter
Frequency p
Weight i = -log2(p)
Ooh (most frequent)
0.10
3.32 bits
A
0.08
3.64 bits
E
0.07
3.84 bits
And
0.06
4.06 bits
I.E. (most rare)
0.002
8.97 bits
Rare symbols carry
greater than
Information than frequent information!
Information weight vs Probability
i (beat)
p (probability)
p=0.002, i=8.97
YOH
8.97
p=0.06, i=4.06
And
4.06
p=0.07, i=3.84
E
3.84
p=0.08, i=3.64
A
3.64
p=0.10, i=3.32
Ooh.
3.32
The less the symbol,
The more information he carries.
03
What's entropy?
Uncertainty, information and average number of bits per symbol
GEF
EEG No. 4
Intuition: uncertainty = information
Imagine that you're being told the result of a coin toss.
Honest.
(50/50) - You didn't know it would come out, and you got it.
1 bit
But if the coin
bilateral
(always eagles) - you knew the result in advance, and information =
0 bits
.
Enthropia
- It's a measure.
Uncertainties
(or otherwise,
average number of information
This is the result of a random experiment that you get.
Than
evener
distribution — themes
greater than
Indeterminateness — Topics
greater than
Entropy.
Unevener
— the easier it is to predict the outcome — the themes
less than
Entropy.
The analogy.
Imagine a bag of balloons. If all the balloons are the same color, you always know what you're gonna get out (entropia = 0.0). If balloons of different colours are equal in number, each pull is a surprise.
Shannon entropy formula
Let the source release the different symbols N, and the i-sign has the probability of p
i
Then.
One character information
:
i
i
= −log
2
(p
i
) бит
The less the symbol, the more information it carries. The letter "Y" (p=0.002) carries ~9 bits, and the letter "O" (P=0.10) is only ~3.3 bits.
Enthropia
- this.
Average
The number of information per symbol weighted by probability:
H = −Σ p
i
· log
2
(p
i
)
Opening the formula:
H = −( p
1
·log
2
p
1
+ p
2
·log
2
p
2
+ ... + p
N
·log
2
p
N
)
Every fold is
Probability
a symbol multiplied by his
Information weight
The amount gives an average "surprise" per message.
Why is that the formula?
Shannon's formula is
Mathematical waiting
Remember:
Probability
event p — how often it happens
Information
events i = -log
2
(p) — how surprising it is
Mathematically waiting
= average = sum (probability x value)
Therefore:
H = E[i] = −Σ p
i
· log
2
(p
i
)
It's like the average score: each rating (the information of the symbol) is weighed by the frequency (the probability).
Entropy properties
H = 0
One symbol with p=1. Total certainty. No information.
H = H
max
Equal distribution. Maximum uncertainty.
H
max
= log
2
(N)
The ceiling, more N, more than the maximum.
🎲
Urna Shannon: Feel entropy in practice
Adjust the color of the balloons in the urn, pull them out and watch how much information you get with each pull.
Entropy
.
Preset:
Theoretical entropy (Shannon formula):
H = 0.00 bits
It's long enough.
0
Experimental entropy
0.000 bits
Last extraction:
How does this work:
Every time you pull out a balloon, you get information.
i = −log
2
(p)
where p is the probability of this colour.
many
Information, frequent balloon (big p) —
Not much.
Ethropy H is
Average
Try to pull out 100 balloons and see how the experimental average approaches the theoretical!
Comparison of approaches
Criterion
Alphabetic
Content
Formula
i = log₂(N)
i = −log₂(p)
Basis
Power of alphabet N
Probability of symbol p
Assumption
All symbols are equal.
Symbols differ in probability
Application
Computer coding
Data compression, entropy
Accuracy
Exact with even distribution
Precise always.
Example
ASCII: 256 characters × 8 bits
Russian text: ~4.5 bits/symbol
Two approaches to measuring information
ALFAVITIC
i = log₂(N)
All symbols are equal.
Computer coding
CONTEXT
i = −log₂(p)
Different probability of symbols
Compressive, entropy Shannon
Relationship with the EEG: Fano Condition
Shannon's entropy is directly related to
Fano condition
If the code does not meet the Fano condition, it is not optimal.
Chapter 23: Fano condition
.
Further: Coding and decoding
Knowing the strength of the alphabet and entropy, you're ready for
coding of information
. How to choose the code, how many bits you need on the symbol, what are even and uneven codes -- in
Chapter 11: Coding and decoding
.
04
Entropy calculator
Interactive calculation H = -\p·log2(p) with stepwise review
📊
Shannon entropy calculator
Ask the probability of symbols (the sum must be 1). Get an entropy H with a step-by-step calculation.
05
Task bank
Challenges to alphabetical and substantive approaches
📝
Task bank
Do the job. Choose the answer.
06
Mini-test: measurement approaches
40 cards · alphabetical and informative approaches, entropy Shannon
0 / 800 XP
0 / 40