线性判别分析
线性判别分析中有降维，把数据都投影到同一条线上，然后在直线上取一个阈值，将直线分成两条射线，每一条代表一个分类。会损失一些数据信息，但如果这些信息是一些干扰信息，丢失也未尝不是好事。
线性判别分析之后的结果是一个向量，其他的不行吗？
主要指导思想(目标)：类内小，类间大。
公式推导
我们得到的是向量，为了方便计算损失，不妨设\(||\pmb w||=1\)，每一个数据\(\pmb{X}_i\)看作一个向量。那么\(\pmb X_i\pmb w\)是每个数据在\(\pmb w\)方向上的投影。与\(\pmb w\)的其中一个平面是划分平面。
两个不同类别分别命名为\(C_1\)和\(C_2\)，用\(\pmb\mu\),\(\pmb\mu_{C_1}\), \(\pmb\mu_{C_2}\)分别代表全部数据，\(C_1\)数据,\(C_2\)数据的均值，用\(\pmb\Sigma\),\(\pmb\Sigma_{C_1}\), \(\pmb\Sigma_{C_2}\)分别代表全部数据，\(C_1\)数据,\(C_2\)数据的协方差矩阵。
\(\tilde{\mu}\)和\(\tilde{\sigma}^2\)表示投影的均值和方差。
\[{\LARGE
\begin{array}{ccl}
\pmb\mu &=& \frac{1}{N} \sum_1^{N}\pmb X_i \\
\pmb\mu_{C_1} &=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}\pmb X_{C_1i} \\
\pmb\mu_{C_2} &=& \frac{1}{N_{C_2}} \sum_1^{N_{C_2}}\pmb X_{C_1i}\ \\
\pmb\Sigma &=& \frac{1}{N} \sum_1^{N}(\pmb X_i -\pmb\mu)(\pmb X_i -\pmb\mu)^T \\
\pmb\Sigma_{C_1} &=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}(\pmb X_{C_1i} -\pmb\mu_{C_1} )(\pmb X_{C_1i} -\pmb\mu_{C_1} )^T\\
\pmb\Sigma_{C_2} &=& \frac{1}{N_{C_2}} \sum_1^{N_{C_2}}(\pmb X_{C_2i} -\pmb\mu_{C_2} )(\pmb X_{C_2i} -\pmb\mu_{C_2} )^T\\
\end{array}
}
\]
\[{\LARGE
\begin{array}{ccl}
\tilde\mu &=& \frac{1}{N} \sum_1^{N}\pmb X_i\pmb \theta \\
\tilde\mu_{C_1} &=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}\pmb X_{C_1i}\pmb \theta \\
\tilde\mu_{C_2} &=& \frac{1}{N_{C_2}} \sum_1^{N_{C_2}}\pmb X_{C_2i}\pmb \theta \\
\tilde\sigma^2 &=& \frac{1}{N} \sum_1^{N}(\pmb X_i\pmb \theta -\tilde\mu)^2 \\
\tilde\sigma_{C_1}^2 &=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}(\pmb X_{C_1i}\pmb \theta -\tilde\mu_{C_1} )^2\\
\tilde\sigma_{C_2}^2 &=& \frac{1}{N_{C_2}} \sum_1^{N_{C_2}}(\pmb X_{C_2i}\pmb \theta -\tilde\mu_{C_2} )^2\\
\end{array}
}
\]
类间：\((\tilde\mu_{C_1}-\tilde\mu_{C_2})^2\)
类内：\(\tilde\sigma^2_{C_1}+\tilde\sigma^2_{C_2}\)
目标函数:\(J(\pmb \theta) = \frac{(\tilde\mu_{C_1}-\tilde\mu_{C_2})^2}{\tilde\sigma^2_{C_1}+\tilde\sigma^2_{C_2}}\)
\[{\LARGE
\begin{array}{ccl}
J(\pmb \theta) &=& \frac{(\tilde\mu_{C_1}-\tilde\mu_{C_2})^2}{\tilde\sigma^2_{C_1}+\tilde\sigma^2_{C_2}}\\
分子&=&(\tilde\mu_{C_1}-\tilde\mu_{C_2})^2\\
&=&(\frac{1}{N_{C_1}} \sum_1^{N_{C_1}}\pmb X_{C_1i}\pmb \theta- \frac{1}{N_{C_2}} \sum_1^{N_{C_2}}\pmb X_{C_2i}\pmb \theta)^2\\
&=&((\pmb\mu_{C_1}-\pmb\mu_{C_2})\pmb \theta)^2\\
&=&\pmb \theta^T(\pmb\mu_{C_1}-\pmb\mu_{C_2})^T(\pmb\mu_{C_1}-\pmb\mu_{C_2})\pmb \theta\\
\tilde\sigma^2_{C_1} &=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}(\pmb X_{C_1i}\pmb \theta -\tilde\mu_{C_1} )^2\\
&=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}(\pmb X_{C_1i}\pmb \theta -\frac{1}{N_{C_1}} \sum_1^{N_{C_1}}\pmb X_{C_1i}\pmb \theta )^2\\
&=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}((\pmb X_{C_1i} -\frac{1}{N_{C_1}} \sum_1^{N_{C_1}}\pmb X_{C_1i})\pmb \theta )^2\\
&=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}((\pmb X_{C_1i} -\pmb\mu_{C_1})\pmb \theta )^2\\
&=& \frac{1}{N_{C_1}} \sum_1^{N_{C_1}}\pmb \theta^T(\pmb X_{C_1i} -\pmb\mu_{C_1})^T(\pmb X_{C_1i} -\pmb\mu_{C_1})\pmb \theta \\
&=& \pmb\theta^T(\frac{1}{N_{C_1}} \sum_1^{N_{C_1}} (\pmb X_{C_1i} -\pmb\mu_{C_1})^T(\pmb X_{C_1i} -\pmb\mu_{C_1}))\pmb \theta \\
&=& \pmb\theta^T\pmb\Sigma_{C_1}\pmb \theta \\
\tilde\sigma^2_{C_2} &=& \pmb\theta^T\pmb\Sigma_{C_2}\pmb \theta\\
分母&=&\pmb\theta^T\pmb\Sigma_{C_1}\pmb \theta+\pmb\theta^T\pmb\Sigma_{C_2}\pmb \theta\\
&=&\pmb\theta^T(\pmb\Sigma_{C_1}+\pmb\Sigma_{C_2})\pmb \theta\\
\end{array}
}
\]
\({\LARGE \therefore}\)
\(
{\LARGE
\begin{array}{ccl}
J(\pmb \theta) &=& \frac{\pmb \theta^T(\pmb\mu_{C_1}-\pmb\mu_{C_2})^T(\pmb\mu_{C_1}-\pmb\mu_{C_2})\pmb \theta}{\pmb\theta^T(\pmb\Sigma_{C_1}+\pmb\Sigma_{C_2})\pmb \theta}\\
\end{array}
}
\)
设\(S_b = (\pmb\mu_{C_1}-\pmb\mu_{C_2})^T(\pmb\mu_{C_1}-\pmb\mu_{C_2})\),\(S_w = \pmb\Sigma_{C_1}+\pmb\Sigma_{C_2}\)
\(S_b\)就是类内方差
\(S_w\)就是类间方差
此时\({\LARGE J(\pmb \theta) = \frac{\pmb \theta^T S_b \pmb \theta}{\pmb\theta^TS_w\pmb \theta}}\)
求导
\[{\LARGE
\begin{array}{rcl}
\frac{\partial J(\pmb \theta)}{\partial \pmb\theta } &=& \frac{\partial\frac{\pmb \theta^T \pmb S_b \pmb \theta}{\pmb\theta^T\pmb S_w\pmb \theta}}{\partial\pmb\theta}\\
&=& \frac{\partial(\pmb \theta^T \pmb S_b \pmb \theta(\pmb\theta^T\pmb S_w\pmb \theta)^{-1})}{\partial\pmb\theta}\\
&=& \frac{\partial(\pmb \theta^T \pmb S_b \pmb \theta )}{\partial\pmb\theta}(\pmb\theta^T\pmb S_w\pmb \theta)^{-1}+\pmb \theta^T \pmb S_b \pmb \theta \frac{\partial((\pmb\theta^T\pmb S_w\pmb \theta)^{-1})}{\partial\pmb\theta}\\
&=&2\pmb\theta^T\pmb S_b(\pmb\theta^T\pmb S_w\pmb \theta)^{-1}+ \pmb \theta^T \pmb S_b \pmb \theta (- \frac{1}{(\pmb \theta^T \pmb S_w \pmb \theta )^2}) (2\pmb\theta^T\pmb S_w)
\end{array}
}
\]
令导数等于零
\[{\LARGE
\begin{array}{rcl}
\pmb 0 &=& 2\pmb S_b\pmb\theta(\pmb\theta^T\pmb S_w\pmb \theta)^{-1}+
\pmb \theta^T \pmb S_b \pmb \theta (- \frac{1}{(\pmb \theta^T \pmb S_w \pmb \theta )^2}) (2\pmb S_w\pmb\theta)
\\
2\pmb S_b\pmb\theta(\pmb\theta^T\pmb S_w\pmb \theta)^{-1}&=& \pmb \theta^T \pmb S_b \pmb \theta ( \frac{1}{(\pmb \theta^T \pmb S_w \pmb \theta )^2}) (2\pmb S_w\pmb\theta)\\
\pmb S_b\pmb\theta(\pmb\theta^T\pmb S_w\pmb \theta)
&=& (\pmb \theta^T \pmb S_b \pmb \theta )\pmb S_w\pmb\theta\\
(\pmb \theta^T \pmb S_b \pmb \theta )\pmb S_w\pmb\theta
&=& \pmb S_b\pmb\theta(\pmb\theta^T\pmb S_w\pmb \theta) \\
\pmb\theta&=& \pmb S_w^{-1}\frac{\pmb \theta^T \pmb S_w \pmb \theta }
{\pmb\theta^T\pmb S_b\pmb \theta}\pmb S_b\pmb\theta\\
\pmb\theta
&=& \pmb S_w^{-1}\frac{\pmb \theta^T \pmb S_w \pmb \theta }
{\pmb\theta^T\pmb S_b\pmb \theta}(\pmb\mu_{C_1}-\pmb\mu_{C_2})^T(\pmb\mu_{C_1}-\pmb\mu_{C_2})\pmb\theta\\
\end{array}
}
\]
\(\because\)
\(\frac{\pmb \theta^T \pmb S_w \pmb \theta }{\pmb\theta^T\pmb S_b\pmb \theta}\),\((\pmb\mu_{C_1}-\pmb\mu_{C_2})\pmb\theta\)是一个数，不影响\(\pmb \theta\)的方向
\({\LARGE \therefore}\)
\(
{\LARGE
\pmb\theta \propto \pmb S_w^{-1}(\pmb\mu_{C_1}-\pmb\mu_{C_2})^T
}
\)
\({\LARGE \mathcal{{\color{Blue} {if}} } } \pmb S_w \propto \pmb I\)
\({\LARGE
\pmb \theta \propto (\pmb\mu_{C_1}-\pmb\mu_{C_2})^T
}\)
求任意一个点的投影
\(
{\Large
proj_{\pmb \theta}(x) = x^T\pmb\theta
}
\)
求阈值
\(
{\Large
\begin{array}{rcl}
threshold &=& \frac{N_{C_1}\tilde\mu_{C_1}+N_{C_2}\tilde\mu_{C_1}}{N_{C_1}+N_{C_2}}\\
&=& \frac{N_{C_1}\pmb\mu_{C_1}\pmb\theta+N_{C_2}\pmb\mu_{C_1}\pmb\theta}{N_{C_1}+N_{C_2}}
\end{array}
}
\)
依赖
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
人工数据集
n = 100
X = np.random.multivariate_normal((1, 1), [[0.64, 0], [0, 0.64]], size = int(n/2))
X = np.insert(X, 50, np.random.multivariate_normal((3, 3), [[0.64, 0], [0,0.64]], size = int(n/2)),0)
#X = np.insert(X, 0, 1, 1)
m = X.shape[1]
y = np.array([1]*50+[-1]*50).reshape(-1,1)
plt.scatter(X[:50, -2], X[:50, -1])
plt.scatter(X[50:, -2], X[50:, -1], c = "#ff4400")
<matplotlib.collections.PathCollection at 0x7f2b50e680d0>
![image]()
X1 = X[(y==1).reshape(-1)]
X0 = X[(y==-1).reshape(-1)]
n1 = np.array([[X1.shape[0]]])
n0 = np.array([[X0.shape[0]]])
mu1 = X1.mean(axis = 0).reshape(-1,1)
mu0 = X0.mean(axis = 0).reshape(-1,1)
Sigma1 = np.cov(X1.T)
Sigma0 = np.cov(X0.T)
theta = (Sigma1 + Sigma0) @ (mu1 - mu0)
threshold = (n1*mu1 + n0*mu0).T@theta/(n1 + n0)
def getForecast(x):
return x.T @ theta
threshold
array([[-10.45793931]])
预测
print(f'{ 1 if getForecast(np.array([[1],[1]])) > threshold else 0}')
1
分界展示
plt.scatter(X[:50, -2], X[:50, -1])
plt.scatter(X[50:, -2], X[50:, -1], c = "#ff4400")
for i in np.arange(-1,5,0.02):
for j in np.arange(-1,5,0.02):
if abs(getForecast(np.array([[i],[j]])) - threshold) <0.01:
plt.scatter(i,j,c="#000000")
![image]()
浙公网安备 33010602011771号