Last updated: 2026-06-28

Shannon Entropy

Shannon entropy is a measure of the average information content, or unpredictability, of a sequence of symbols. Given a string of text, it answers the question: how many bits (or nats, or hartleys) of information does each character carry on average? The concept was introduced by Claude Shannon in 1948 as the foundation of information theory and remains central to data compression, cryptography, and machine learning.

The entropy formula

For a source that emits symbols from an alphabet of size $k$, where symbol $i$ occurs with probability $p_i$, the Shannon entropy is:

$H = -\sum_{i=1}^{k} p_i \log_b(p_i)$

The base $b$ of the logarithm determines the unit: base 2 gives bits, base $e$ gives nats, and base 10 gives hartleys (also called bans or dits).

Worked example — "hello"

The string "hello" contains five characters: h, e, l, l, o. Character frequencies:

Applying the formula in bits (base 2):

The total entropy for the full string is $1.9219 \times 5 \approx 9.61$ bits. This is the theoretical minimum number of bits needed to losslessly encode "hello" using an optimal code for this particular frequency distribution.

Why the formula works

Each term $-p_i \log_2 p_i$ is the contribution of symbol $i$ to the total uncertainty. Symbols that appear rarely (small $p_i$) carry a lot of information when they do appear — they are surprising. Symbols that appear constantly (large $p_i$) carry little information — they are expected.

Shannon showed that entropy is the unique function satisfying three intuitive axioms: it is continuous in the probabilities, it is maximised by a uniform distribution, and adding a certainly-occurring symbol (probability 1) does not change it. Together, these axioms pin down $H = -\sum p_i \log p_i$ as the only correct measure of average information.

Maximum and minimum entropy

For a source with $k$ distinct symbols:

• Maximum entropy: $H_{max} = \log_2 k$ bits per symbol, achieved when all symbols are equally probable.
• Minimum entropy: $H = 0$, achieved when one symbol occurs with probability 1 (no uncertainty).

Actual English text sits around 1–1.5 bits per letter — far below the theoretical maximum of $\log_2 26 \approx 4.7$ bits — because of the highly non-uniform frequencies of letters and the strong statistical dependencies between adjacent characters (q is almost always followed by u, for example).

Entropy and data compression

Shannon's source coding theorem proves that no lossless compression algorithm can compress a message to fewer bits per symbol than its entropy. Entropy is therefore a hard lower bound on compressed file size.

This connection explains why:

• Files with repetitive content (like a log file full of similar lines) compress well — their entropy is low.
• Truly random or already-encrypted data cannot be compressed — it is already at its entropy limit.
• Compression algorithms such as Huffman coding and arithmetic coding approach the entropy bound, with Huffman coding guaranteed to use at most one extra bit per symbol beyond the bound.

Relation to password strength

Password entropy, as used in security analysis, is a related but distinct concept. It measures the uncertainty from an attacker's perspective, assuming the password is drawn uniformly at random from a pool of possible strings. The formula is $H = L \log_2 N$, where $L$ is the password length and $N$ is the character pool size.

Shannon entropy of a specific password — as this calculator computes — measures the frequency distribution of the characters in that one string. A password like "aaaa" has zero Shannon entropy but is still four characters long. For password security, use the Password Entropy Calculator which models the attacker's search space.