概率知识补充--高斯分布1

概率知识补充--高斯分布1

数据

N个样本独立同分布

\[X = (x_{1},x_{2},\cdots,x_{N})^{T} = \left(\begin{array}{c}x_{1}^{\top} \\ x_{0}^{\top} \\ \vdots \\ x_{N}^{\prime}\end{array}\right)_{N \times \beta} \]

\[x_{i} \in \mathbb{R}^{p} \]

\[x_{i} \sim N(\mu,\sigma) \]

\[\theta = (\mu,\sigma) \]

最大似然估计(MLE)

\[\theta_{MLE} = \arg \max_{\theta} P(X|\theta) \]

为了方便证明，数据纬度为一维

令

\[p = 1 ,\ \theta=(\mu, \sigma^{2}) \]

因为是数据独立同分布的，所以

\[\log P(x|\theta) = \log \prod_{i=1}^{N} p(x_{i}|\theta) \\ = \sum_{i=1}^{N} \log p(x_{i}|\theta) \tag{1-1} \]

因为 样本是正态分布的 一个维度的

\[p(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp (-\frac{(x-\mu)^{2}}{2 \sigma^{2}}) \tag{1-2} \]

另外多个维度的高斯分布是下面这样的

\[p(x) = \frac{1}{(2\pi)^{\frac{p}{2}} ｜\Sigma｜^{\frac{1}{2}}}\exp(-\frac{1}{2}(x-\mu)^{\mathrm{T}}\Sigma^{-1}(x-\mu)) \]

将\((1-2)\)式带入\((1-1)\)中,并展开

\[\log P(x|\theta) = \log \prod_{i=1}^{N} p(x_{i}|\theta) \\ = \sum_{i=1}^{N} \log p(x_{i}|\theta) \\ = \sum_{i=1}^{N} \log \frac{1}{\sqrt{2 \pi} \sigma} \exp (-\frac{(x-\mu)^{2}}{2 \sigma^{2}}) \\ = \sum_{i=1}^{N}(\log \frac{1}{\sqrt{2 \pi}}+\log \frac{1}{\sigma} - \frac{(x-\mu)^{2}}{2 \sigma^{2}}) \]

求 \(\mu_{MLE}\),去掉和 \(\mu\)的项

\[\mu_{MLE} = \arg \max_{\mu} \sum_{i=1}^{N} \log P(x_{i}|\theta) \\ = \arg \max_{\mu} -\sum_{i=1}^{N} \frac{(x_{i}-\mu)^{2}}{2 \sigma^{2}} \\ = \arg \min_{\mu} \sum_{i=1}^{N} \frac{(x_{i}-\mu)^{2}}{2 \sigma^{2}} \\ = \arg \min_{\mu} \sum_{i=1}^{N}(x_{i}-\mu)^{2} \]

对 \(\mu\)求导

\[\frac{\partial}{\partial \mu} \sum_{i=1}^{N}(x_{i}-\mu)^{2} =2 \sum_{i=1}^{N} (x_{i} - \mu) = 0 \\ \sum_{i=1}^{N} (x_{i} - \mu) = 0 \\ \sum_{i=1}^{N} x_{i} - \sum_{i=1}^{N}\mu =0 \\ \mu_MLE = \frac{1}{N}\sum_{i=1}^{N} x_{i} \]

同样的方法求 \(\sigma_{MLE}\)

\[\sigma_{MLE} = \arg \max_{\sigma} \sum_{i=1}^{N} - \log \sigma - \frac{1}{2 \sigma^{2}}(x_{i}-\mu)^2 \]

对 \(\sigma\)求导

\[\frac{\partial}{\partial \sigma} \sum_{i=1}^{N} - \log \sigma - \frac{1}{2 \sigma^{2}}(x_{i}-\mu)^2 = \sum_{i=1}^{N} - \frac{1}{\sigma}-\frac{(x_{i}-\mu)^{2}}{2} (-2) \sigma^{-3} = 0\\ \sum_{i=1}^{N} \sigma^{2}-(x_{i}-\mu)^{2}=0 \\ \sigma^{2}_{MLE} = \frac{1}{N}\sum_{i=1}(x_{i}-\mu)^{2} \\ \]

其中 \(\sigma^{2}_{MLE}\) 为有偏估计

\[\mathbb{E}\left[\sigma^{2}_{MLE}\right] = \frac{N-1}{N} \sigma^{2} \]

无偏估计

\[\hat{\sigma} = \frac{1}{N-1} \sum_{i=1}^{N}(x_{i}-\mu_{MLE})^{2} \]