Causal Information Theory

Core Idea

Causal information theory extends Shannon's information theory to directed, causal systems where we care not just about dependence but about causality — who influences whom, and how. While mutual information I(X;Y) quantifies dependence between X and Y, it does not distinguish whether X causes Y, Y causes X, or both are caused by a common confounder. Graphical causal models (directed acyclic graphs) encode causal assumptions. d-separation in a causal graph determines conditional independence: variables that are d-separated given other variables are informationally independent in causal systems respecting the graph. Transfer entropy measures information flow from X to Y in time-series data, accounting for Y's own history. Causal information flow quantifies how much information about cause X is necessary to explain effect Y, beyond what Y's history provides. Interventions (setting a variable to a specific value) are informationally different from observations: an intervention on X breaks X's dependence on its causal parents. Causal information theory provides tools for discovering causal structures from observational data, quantifying causal effects informationally, and understanding limits of causal identification.

Explainer

Shannon's information theory quantifies dependence: mutual information I(X;Y) measures correlation, regardless of direction. Causality is more subtle: we want to know not just if X and Y are dependent, but whether X causes Y, or vice versa, or if both are consequences of a third variable. Causal information theory extends Shannon's framework to address these questions.

Causal Graphs and Conditional Independence:

A causal directed acyclic graph (DAG) encodes causal assumptions: nodes are variables, edges represent direct causal influences. A path from X to Y represents a causal chain. The causal Markov condition states: each variable is conditionally independent of its non-descendants given its parents. This translates the graph structure into testable conditional independence statements. d-separation is a graph algorithm: two variables are d-separated given a conditioning set if all paths between them are blocked by the conditioning set or by collider structures. d-separation implies conditional independence: if X and Z are d-separated given S, then I(X;Z|S) = 0 in any distribution respecting the graph. This allows data to test causal hypotheses: measure whether the predicted conditional independences hold.

Confounding and Intervention:

A key challenge in causal inference is confounding: an unobserved variable that influences both X and Y, creating spurious correlation. Observationally, X and Y appear dependent, but X does not cause Y — the dependence is "confounded" by the third variable. Information-theoretically, I(X;Y) > 0 but this is not information flow from X to Y. The distinction between observation and intervention resolves this. An intervention (denoted do(X=x) in Pearl's notation) sets X to a specific value, severing its dependence on its parents (including confounders). The post-intervention distribution P(Y | do(X=x)) reflects only X's causal effect on Y, not spurious correlations. In observational data, P(Y|X) may reflect confounding; under intervention, P(Y|do(X)) reveals true causal effects. This distinction is fundamental: causal inference from observational data requires assuming no hidden confounders or using sensitivity analyses.

Transfer Entropy and Temporal Causality:

In time-series data, determining causality from X to Y is challenged by the fact that both X and Y may have temporal structure (autoregressive dependence, trends). Transfer entropy T(X → Y) = I(Y_t ; X_past | Y_past) measures information in X's past about Y's future, conditioned on Y's own past. By conditioning on Y_past, we isolate the contribution of X beyond Y's internal dynamics. If T(X → Y) > 0, there is information flow from X to Y suggesting causality. Conversely, if T(X → Y) = 0, X provides no unique predictive information about Y given Y's history. Transfer entropy is a practical tool for causal discovery in time-series data (e.g., neural data, climate variables), though it assumes no hidden confounders and can be computationally expensive to estimate.

Identifiability and Causal Discovery:

Given observed conditional independences, can we determine the true causal graph? Not always. Multiple causal graphs (called a Markov equivalence class) may entail identical conditional independence statements, yielding the same observational distribution. These graphs are observationally indistinguishable from data alone. To resolve this, we need additional information: domain knowledge (ruling out some causal directions), temporal ordering (X must precede Y to cause it), or interventional data. Causal discovery algorithms (e.g., PC algorithm, FCI) attempt to find causal structures from observational data by testing conditional independences. They return a set of plausible graphs (Markov equivalence class), not a unique answer, acknowledging the limits of inference from observational data.

Information Flow in Causal Systems:

Causal information flow quantifies how much information about a cause X is necessary to determine an effect Y. If X perfectly determines Y (deterministic causality), all information about X is transmitted to Y, but information may be lost due to noise. The Markov property states that X's information about Y's future is fully captured by X's direct effect; X does not need information from X's own past (conditional on X's current state) to predict Y. This dramatically reduces the information needed: to predict Y_t, we need information about X_t and Y_t's parents, not the entire history.

Applications:

• Causal Discovery: Infer causal structure from observational or interventional data.
• Causal Effect Estimation: Quantify the effect of an intervention using information-theoretic bounds.
• Graphical Causal Models: Tools like Bayesian networks encode causal assumptions and guide inference.
• Neuroscience: Determine functional connectivity (which brain regions causally influence others) using transfer entropy and causal inference methods.
• Policy Evaluation: Estimate the causal effect of a policy change using observational data and causal assumptions.

Causal information theory unifies causal inference and information theory, providing tools to move beyond correlation to causation, and to quantify and discover causal relationships from data. The framework remains an active frontier, with open questions about identifiability, latent confounding, and computational efficiency of causal discovery.