A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
The Fisher information I(theta) = E[(d/d_theta log f(X;theta))2] = -E[d2/d_theta2 log f(X;theta)] measures how much a sample from the distribution f(X;theta) tells you about the parameter theta. It quantifies the curvature of the log-likelihood around the true parameter value — sharp peaks (high Fisher information) mean the data is highly informative. The Cramer-Rao bound states that any unbiased estimator of theta has variance at least 1/I(theta), establishing Fisher information as the fundamental limit of parameter estimation. Fisher information connects to KL divergence (it is the second derivative of D_KL) and forms the metric tensor in information geometry.
Shannon entropy and mutual information measure how much uncertainty exists or how much two variables share. Fisher information asks a different question: given that data comes from a parametric family f(x; theta), how much does a sample tell you about the parameter theta? While Shannon's measures are distribution-to-distribution, Fisher information is a property of a parametric model at a specific parameter value.
The score function s(x; theta) = d/d_theta log f(x; theta) measures how the log-likelihood changes as theta varies. Under regularity conditions, E[s] = 0 (the score has zero mean). The Fisher information is the variance of the score: I(theta) = Var[s(X; theta)] = E[s2]. High Fisher information means the log-likelihood is steep — small changes in theta cause large changes in the likelihood, making different theta values easy to distinguish from data. An equivalent expression is I(theta) = -E[d2/d_theta2 log f(X; theta)], relating Fisher information to the curvature (concavity) of the expected log-likelihood.
The Cramer-Rao bound gives Fisher information its operational meaning: for any unbiased estimator theta-hat of theta, Var(theta-hat) >= 1/I(theta). With n i.i.d. observations, the bound becomes 1/(nI(theta)). This says that estimation precision is fundamentally limited by how informative the data is about the parameter. An estimator achieving this bound is called "efficient." Maximum likelihood estimators are asymptotically efficient — they approach the bound as the sample size grows.
The connection to information theory runs deep. Fisher information is the local version of KL divergence: I(theta) = d2/d_epsilon2 D_KL(f(x;theta) || f(x;theta+epsilon))|_{epsilon=0}. KL divergence measures global distributional difference; Fisher information measures infinitesimal difference. This relationship makes Fisher information the natural metric on the space of probability distributions — the starting point of information geometry. It also connects to the maximum entropy principle: the Fisher-efficient estimator is the one that maximizes entropy subject to the observed sufficient statistics, linking estimation theory back to Shannon's framework.