Markov Chain
A Markov chain is a memoryless stochastic process: the next state depends only on the current state, described by a transition matrix P.
Purpose
To model random transitions between states and analyze stationary distributions and convergence.
Mechanism
P_{ij}=Pr(X_{t+1}=j|X_t=i). The n-step transitions are given by P^n. An irreducible, aperiodic chain has a unique stationary distribution π with πP=π. MCMC exploits this property to sample from target distributions.
Use cases
- MCMC sampling
- Stochastic process modeling
- PageRank