# 信息熵 Information Theory

$H(X) =\sum_{i=1}^{m} p(x_i) \cdot \log \frac{1}{p(x_i)} = - \sum_{i=1}^{m} p(x_i) \cdot \log p(x_i)$

\begin{aligned}
H(Y|X)
&= \sum_{i=1}^mp(x_i)H(Y|X=x_i) \\
&= -\sum_{i=1}^mp(x_i)\sum_{j = 1}^np(y_j|x_i)\log p(y_j|x_i) \\
&= -\sum_{i=1}^m \sum_{j=1}^np(y_j,x_i)\log p(y_j|x_i) \\
&= -\sum_{x_i,y_j} p(x_i,y_j)\log p(y_j|x_i)
\end{aligned}

$H(X,Y) = -\sum_{i=1}^m\sum_{j=1}^n p(x_i,y_j)\log p(x_i，y_j)$

\begin{aligned}
H(Y|X) &= H(X,Y) - H(X)      \\
H(X|Y) &= H(X,Y) - H(Y)
\end{aligned}

1）$H(Y|X) \ge \max(H(X),H(Y))$ ;

2）$H(X,Y) \le H(X) + H(Y)$ ;

3）$H(X,Y) \ge 0$.

$D(P||Q) = \sum_{x \in X} P(x) \cdot \log\frac{P(x)}{Q(x)}$

$D(P||Q) \ne D(Q||P)$

KL 散度并不满足距离的概念，因为 KL 散度不是对称的,且不满足三角不等式。

$MI(X,Y) = \sum_{i=1}^{m} \sum_{j=1}^{n} p(x_i,y_j) \cdot log_2 {\frac{p(x_i,y_j)}{p(x_i)p(y_j)}}$

$H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y)$

$MI(X,Y) = H(X) -H(Y|X) = H(Y) - H(X|Y)$

$H(p,q) = E_p[-\log q] = -\sum_{i=1}^m{p(x_i) \log{q(x_i)}}$

\begin{aligned}
H(p,q) &= -\sum_x p(x) \log q(x) \\
&= -\sum_x p(x) \log \frac{q(x)}{p(x)}p(x)\\
&= -\sum_x p(x) \log p(x) -\sum_x p(x)  \log  \frac{q(x)}{p(x)}\\
&= H(p)+ D(p||q)
\end{aligned}

$g(D,A) = H(D) – H(D|A)$

$g_R(D,A) = \frac{g(D,A)}{H(D)}$



posted @ 2016-07-26 16:15  ooon  阅读(9418)  评论(0编辑  收藏  举报