Markov Chain——Applied Math

Markov Chain

Example

Exmp: Symmetric random walk. \(\{\xi_i\}_{i=1}^{\infty}\) i.i.d.

\[P(\xi_i = 1) = P(\xi_i = -1) = \tfrac{1}{2}, \qquad X_n = \sum_i \xi_i \]

Exmp: Ehrenfest' diffusion model

If you have two boxes and some balls, draw one at random from one box and put it in another one. Foucus on the number of balls in one box

Exmp: Autoregressive model of order \(k\).

\[Y_n = a_1 Y_{n-1} + \cdots + a_k Y_{n-k} + R_n \]

\[X_n = \begin{pmatrix} a_1 & a_2 & \cdots & a_k \\ 1 & & & \\ & \ddots & & \\ & & 1 & \end{pmatrix} X_{n-1} + \begin{pmatrix} 1 \\ \\ \\ \end{pmatrix} R_n, \qquad X_n = \begin{pmatrix} Y_n & \cdots & Y_{n-k+1} \end{pmatrix}\]

The \(R_n\) is independent.

---

Transition Probability and Stationary Distribution

Def: transition probability \(p^{(n)}_{kj}\) is the probability starting from \(k\) at step \(n\) and after one step to \(j\)

Def: Time-homogeneous: \(p^{(n)}_{kj}\) independent of \(n\).

Notation: \(P = (p_{ij})\), the Markov Chain is time-homogeneous.

Prop. Chapman–Kolmogorov equation

\[P^{n}(i,j) = \sum_{k} P^{m}(i,k)\, P^{n-m}(k,j) \]

Def: A distribution is invariant if

\[\mu = \mu P \]

Lemma: The spectral radius of \(P\) is \(1\).

Proof: Let \(\lambda u = Pu\).

\[\quad \lambda u_i = \sum_j p_{ij} u_j \]

Let \(u_i\) be maximum:

\[\lambda |u_i| \le \sum_j p_{ij} |u_i| \quad \Rightarrow \quad \lambda \le 1 \]

---

Irreducible

Def: If there is a permutation matrix \(R\) such that

\[R^{T} P R = \begin{pmatrix} A_1 & B \\ 0 & A_2 \end{pmatrix} \]

then \(P\) is reducible. Otherwise irreducible.

Def: \(j\) is accessible from \(i\) if \(\rho(i,j) > 0\). Which means probability of the first time the chain goes from \(i\) to \(j\) is finite

\(j\) and \(i\) communicate if \(\rho(i,j) > 0\) and \(\rho(j,i) > 0\).

Lemma: Irreducible is equivalent to \(i, j\) communicate for all pairs in States.

Proof: If reducible, take \(i\) in row of \(A_2\), \(j\) in column of \(A_1\); then \(i, j\) not communicate.

Suppose \(i, j\) not communicate, \(p(i,j) = 0\), \(j = 1\). Let \(2, \ldots, m\) with \(p(k,1) > 0\), and \(m+1, \ldots, n\) with \(p(k,1) = 0\). For all \(k \in \{m+1, \ldots, n\}\), let \(j \in \{1, 2, \ldots, m\}\), \(p(k,j) = 0\).

Thm. \(A\) irreducible, \(\rho(A)\) be spectral radius.

(1) There is \(x\) and \(y\):

\[Ax = \rho(A)\, x, \qquad y^{T} A = \rho(A)\, y^{T} \]

(2) \(\lambda = \rho(A)\) is eigenvalue of multiplicity \(1\).

Ergodic Theorem

Def: primitive, if \(P^{s}_{ij} > 0\) for an \(s\).

Thm. Markov chain is primitive. Then for any initial \(\mu\):

\[\mu_n = \mu_0 P^{n} \Rightarrow \pi \quad \text{exponential fast} \]

\(\pi\) invariant distribution.

Pf: total variation

\[d(\mu, \widetilde{\mu}_0) = \tfrac{1}{2} \sum_{i \in S} |\mu_i - \widetilde{\mu}_i| \]

\[d(\mu_s, \widetilde{\mu}_s) = \sum_{i \in S} \Big[ \sum_j \widetilde{\mu}_{0,j }(P^{s})_{ji} - \widetilde{\mu}_{0,j} (P^{s})_{ji} \Big]^{+} \]

\[\le \sum_{j\in S} \sum_{i \in S} (P^{s})_{ji}\, (\mu_{0,j } - \widetilde{\mu}_{0,j })^{+} \quad \text{(by the set where } + \text{ is chosen)} \]

\[\le \sum_{j \in S} (\mu_{0,j } - \widetilde{\mu}_{0,j })^{+}(1-\alpha) \]

\(n \to \infty\): \(\; d(\mu_s, \widetilde{\mu}_0) \to 0\)

\[\Rightarrow \quad \mu_0 P \to \pi \quad \text{exp. fast} \]

Thm. Ergodic thm: \(\{X_n\}\) be an irreducible finite Markov Chain, \(f\) bounded.

\[\tfrac{1}{n} \sum_{m \ge 0} f(X_m) \Rightarrow \langle f \rangle_\pi = \sum_i f(i)\, \pi_i \]

Poisson Process

Def: \(N_t\) increasing, right-continuous, integer valued.

1. \(N_0 \to 0\)
2. stationary independent increments
3. \(P(N_{t+h} = N_t + 1 \mid N_t) = \lambda h + o(h)\)

Thm. Any fixed time \(t\): distribution of \(N_t\) is Poisson with parameter \(\lambda t\).

Pf: Let \(p_m(t) = P(N_t = m)\)

\[p_0(t+h) = p_0(t)(1 - \lambda h) + o(h) \]

\[\frac{p_0(t+h) - p_0(t)}{h} = -\lambda p_0(t) + o(h) \]

\[\Rightarrow \quad \frac{dp_0(t)}{dt} = -\lambda p_0(t), \qquad p_0(0) = 1 \]

\[\Rightarrow \quad p_0(t) = e^{-\lambda t} \]

\[p_m(t+h) = p_{m-1}(t)\, \lambda h + o(h) + p_m(t)(1 - \lambda h) + o(h) \]

\[\frac{dp_m}{dt} = \lambda p_{m-1}(t) - \lambda p_m \]

\[p_m(t) = \frac{(\lambda t)^m}{m!} e^{-\lambda t} \quad \text{is a solution, and by uniqueness, concludes the proof} \]

Waiting time distribution

\[T = \inf \{ t : t > 0,\; N_t \ne N_0 \} \]

waiting time distribution:

\[v(t) = P(T > t) = P(N_{t-h} = 0 \;\; \forall h) = \lim_{h} P(N_{t-h} = 0) \]

\[= e^{-\lambda(t-h)} \to e^{-\lambda t} \]

5. Q process

It is a Finite state, right-continuous process with the transition probability in following

\[p_{ij}(t) = P(X_{t+s} = j \mid X_s = i) \]

\[p_{ii}(h) = 1 - \lambda_i h + o(h) \]

\[p_{ij}(h) = \lambda_{ij} h + o(h), \qquad j \ne i \]

\[p_{ij} = 0 \]

\[\Rightarrow \quad \lambda_{ij} \ge 0, \qquad \sum_j \lambda_{ij} = \lambda_i \]

Chapman–Kolmogorov equation

\[P(t+s) = P(t) P(s) = P(s) P(t) \]

\[Q = \lim_{h \to 0} \tfrac{1}{h} \left( P(h) - I \right) \]

\[\frac{dP(t)}{dt} = P(t)\, Q \quad \text{forward equation} \]

\[\frac{dP(t)}{dt} = Q\, P(t) \quad \text{backward equation} \]

Solution: \(P(t) = e^{Q t}\)

Kolmogorov equation for distribution

\[\pi_j(t + dt) = \sum_{i \ne j} \pi_i(t)\, p_{ij}(dt) + \pi_j(t)\, p_{jj}(dt) \]

\[= \sum_{i \ne j} \pi_i(t)\, q_{ij}\, dt + \pi_j(t)\, (1 + p_{jj})\, dt \quad + o(dt) \]

\[\frac{d\pi_j}{dt} = \sum_i \pi_i\, Q_{ij} \quad \Rightarrow \quad \frac{d\pi}{dt} = \pi Q \]

Kolmogorov equation for distribution:

\[\pi(t) = \pi_0 e^{Q t} \quad \Rightarrow \quad \pi = \pi e^{Q t}, \qquad 0 = \pi \cdot Q, \qquad \pi P(t) = \pi \]

Waiting time distribution

\[v_j(t) - v_j(t+h) \;\Rightarrow\; p_{jj}(h) = -p_{jj} h + o(h) \]

\[\Rightarrow \quad \frac{dv_j}{dt} = \lambda_j v_j \]