A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
A continuous-time Markov chain (CTMC) is a Markov process on a countable state space where the process holds in each state for an Exponential(qᵢ) time, then jumps to state j with probability qᵢⱼ/qᵢ. The generator matrix Q (with off-diagonal entries qᵢⱼ ≥ 0 and rows summing to zero) encodes the dynamics. The forward equation dp/dt = pQ and backward equation dp/dt = Qp govern the evolution of transition probabilities, and the stationary distribution π satisfies πQ = 0.
A continuous-time Markov chain (CTMC) extends discrete-time Markov chains to continuous time. The process lives on a countable state space S and evolves by holding in the current state i for a random Exponential(qᵢ) time, then jumping to a new state j ≠ i with probability qᵢⱼ/qᵢ, where qᵢ = Σ_{j≠i} qᵢⱼ is the total exit rate. The exponential holding time is not a choice but a necessity: the memoryless property of the exponential is the only way to ensure the Markov property holds in continuous time.
The dynamics are encoded in the generator matrix (or Q-matrix) Q, where the off-diagonal entry qᵢⱼ ≥ 0 is the rate of transitioning from i to j, and the diagonal entry qᵢᵢ = -Σ_{j≠i} qᵢⱼ makes each row sum to zero. The transition probability matrix P(t) = eQt satisfies the Kolmogorov equations: forward dP/dt = P(t)Q and backward dP/dt = QP(t). For finite state spaces, these are systems of linear ODEs with the matrix exponential as the solution. The generator Q is the continuous-time analogue of the transition matrix P - I from discrete time: it describes instantaneous rates rather than one-step probabilities.
The stationary distribution π satisfies πQ = 0 with Σπᵢ = 1 — the continuous-time analogue of πP = π. For irreducible CTMCs on finite state spaces, a unique stationary distribution always exists. For birth-death processes (where transitions occur only between adjacent states), the stationary distribution has a product-form solution: πₙ = π₀ · (λ₀λ₁...λₙ₋₁)/(μ₁μ₂...μₙ), where λᵢ are birth rates and μᵢ are death rates. This is the foundation of queuing theory (M/M/1, M/M/c, and related queues) and population dynamics.
CTMCs are connected to both discrete-time chains and diffusion processes. The embedded chain Yₙ (the sequence of states visited, ignoring holding times) is a discrete-time Markov chain with transition matrix pᵢⱼ = qᵢⱼ/qᵢ for j ≠ i. The uniformization technique converts a CTMC into a discrete-time chain by embedding it in a Poisson process with rate q ≥ max qᵢ: the chain jumps at every Poisson event, possibly staying in the same state. In the other direction, when the state space is taken to the continuum and transition rates scale appropriately, CTMCs converge to diffusion processes — the Kolmogorov forward equation for CTMCs becomes the Fokker-Planck equation for diffusions.