Entropy Calculator
Calculate Shannon entropy of probability distributions with step-by-step breakdown, perplexity, and redundancy metrics.
What Is Shannon Entropy?
Shannon entropy measures average uncertainty in a probability distribution. Higher entropy means outcomes are harder to predict. Lower entropy means one or few outcomes dominate.
Entropy Formula
$$H(X) = -\sum_{i=1}^{n} p_i \log(p_i)$$Each probability $p_i$ must be between 0 and 1, and all probabilities must sum to 1. The logarithm base sets the unit: base 2 gives bits, base $e$ gives nats, base 10 gives dits.
Related Metrics
Maximum entropy for $n$ equally likely outcomes is $\log(n)$. Redundancy is $H_{\max} - H$. Perplexity (base 2) is $2^{H}$ and represents effective number of choices.
Related tools: use the Chi Square Test Calculator for distribution comparisons and the Average Calculator for basic descriptive statistics.
Frequently Asked Questions
Why must probabilities sum to 1?
Shannon entropy is defined for valid probability distributions. If values do not sum to 1, they do not represent a complete set of mutually exclusive outcomes.
What entropy does a fair coin have?
A fair coin with probabilities 0.5 and 0.5 has entropy of 1 bit in base 2. This is the maximum possible entropy for two outcomes.
What does maximum uncertainty mean?
It means the distribution is close to uniform, so no outcome is strongly favored. A fair six-sided die is a common example of high uncertainty.
How is perplexity used?
In language modeling, perplexity summarizes how well a model predicts the next token. Lower perplexity usually means better predictive performance.
Related Tools
HMAC Generator
Holding Period Return Calculator
Hex String Converter
Screenshot Tool
SQL to YAML Converter
Current Ratio Calculator
Find Minimum and Maximum
Thermal Conductivity Calculator
Gestational Age Calculator
Convert Time to Seconds
Sound Intensity Decibel Calculator
Arccos Calculator
Scoreboard
Adler32 Checksum Calculator
Random Word Generator
Goodwill to Assets Ratio Calculator
Blood Donation Time Calculator
Pdf Size
Webp To Gif Converter
Convert Gif To Bmp