前面讨论了 y = ax + b 考虑的只有一个 特征值(因素)的情况下,但在很多情况下 特征值不只有一个 打个比方 要预测房价 要考虑的不只是面积 还要有 地段 建造年代 户型 等等 ,此时就要用到多元线性回归了。
( θ 0 , θ 1 , θ 2 , θ 3 , . . . . . , θ n ) (\theta_{0},\theta_{1},\theta_{2},\theta_{3},.....,\theta_{n}) ( θ 0 , θ 1 , θ 2 , θ 3 , . . . . . , θ n ) θ \theta θ ( 1 , X 1 ( i ) , X 2 ( i ) ， X 3 ( i ) , . . . . . , X n ( i ) (1,X_{1}^{(i)},X_{2}^{(i)}，X_{3}^{(i)},.....,X_{n}^{(i)} ( 1 , X 1 ( i ) , X 2 ( i ) ， X 3 ( i ) , . . . . . , X n ( i ) y ^ ( i ) = θ 0 X 0 ( i ) + θ 1 X 1 ( i ) + θ 2 X 2 ( i ) + θ 3 X 3 ( i ) , . . . . . , θ n X n ( i ) \hat{y}^{(i)} =\theta_{0}X_{0}^{(i)}+\theta_{1}X_{1}^{(i)}+\theta_{2}X_{2}^{(i)}+\theta_{3}X_{3}^{(i)},.....,\theta_{n}X_{n}^{(i)} y ^ ( i ) = θ 0 X 0 ( i ) + θ 1 X 1 ( i ) + θ 2 X 2 ( i ) + θ 3 X 3 ( i ) , . . . . . , θ n X n ( i )
θ 0 X 0 ( i ) \theta_{0}X_{0}^{(i)} θ 0 X 0 ( i ) y ^ ( i ) = b + θ 1 X 1 ( i ) + θ 2 X 2 ( i ) + θ 3 X 3 ( i ) , . . . . . , θ n X n ( i ) \hat{y}^{(i)} =b+\theta_{1}X_{1}^{(i)}+\theta_{2}X_{2}^{(i)}+\theta_{3}X_{3}^{(i)},.....,\theta_{n}X_{n}^{(i)} y ^ ( i ) = b + θ 1 X 1 ( i ) + θ 2 X 2 ( i ) + θ 3 X 3 ( i ) , . . . . . , θ n X n ( i )
使用和一元线性同样的 损失函数:a r g m i n ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) 2 {argmin}\sum_{i=1}^{m}(y^{(i)} - \hat{y}^{(i)})^2 a r g m i n ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) 2
写成矩阵形式θ = ( θ 0 , θ 1 , θ 2 , θ 3 , . . . . . , θ n ) T \theta=(\theta_{0},\theta_{1},\theta_{2},\theta_{3},.....,\theta_{n})^T θ = ( θ 0 , θ 1 , θ 2 , θ 3 , . . . . . , θ n ) T X ( i ) = ( 1 , X 1 ( i ) , X 2 ( i ) , X 3 ( i ) , . . . . . , X n ( i ) ) T X^{(i)}=(1,X^{(i)}_{1},X^{(i)}_{2},X^{(i)}_{3},.....,X^{(i)}_{n})^T X ( i ) = ( 1 , X 1 ( i ) , X 2 ( i ) , X 3 ( i ) , . . . . . , X n ( i ) ) T
y ^ ( i ) = θ ⋅ X ( i ) \hat y^{(i)} =\theta \cdot X^{(i)} y ^ ( i ) = θ ⋅ X ( i ) y = θ X + b y=\theta X + b y = θ X + b
输入的样本:X = [ 1 X ( 1 ) ( 1 ) X ( 2 ) ( 1 ) . . . . X ( n ) ( 1 ) 1 X ( 1 ) ( 2 ) X ( 2 ) ( 2 ) . . . . X ( n ) ( 2 ) . . . . . 1 X ( 1 ) ( m ) X ( 2 ) ( m ) . . . . X ( n ) ( m ) ] X = \left[ \begin{matrix} 1 & X^{(1)}_{(1)} & X^{(1)}_{(2)} & .... & X^{(1)}_{(n)}\\ 1 & X^{(2)}_{(1)} & X^{(2)}_{(2)} & .... & X^{(2)}_{(n)}\\ ..... \\1 & X^{(m)}_{(1)} & X^{(m)}_{(2)} & .... & X^{(m)}_{(n)}\\ \end{matrix} \right] X = ⎣ ⎢ ⎢ ⎢ ⎡ 1 1 . . . . . 1 X ( 1 ) ( 1 ) X ( 1 ) ( 2 ) X ( 1 ) ( m ) X ( 2 ) ( 1 ) X ( 2 ) ( 2 ) X ( 2 ) ( m ) . . . . . . . . . . . . X ( n ) ( 1 ) X ( n ) ( 2 ) X ( n ) ( m ) ⎦ ⎥ ⎥ ⎥ ⎤
待求得权重:θ = [ θ 0 θ 1 θ 2 . . . . θ n ] T \theta = \left[ \begin{matrix} \theta_{0} \\ \theta_{1} \\ \theta_{2} \\....\\\theta_{n} \\\end{matrix} \right]^T θ = ⎣ ⎢ ⎢ ⎢ ⎢ ⎡ θ 0 θ 1 θ 2 . . . . θ n ⎦ ⎥ ⎥ ⎥ ⎥ ⎤ T y = [ y 0 y 1 y 2 . . . . y n ] y = \left[ \begin{matrix} y_{0} \\ y_{1} \\ y_{2} \\....\\y_{n} \\\end{matrix} \right] y = ⎣ ⎢ ⎢ ⎢ ⎢ ⎡ y 0 y 1 y 2 . . . . y n ⎦ ⎥ ⎥ ⎥ ⎥ ⎤
X 为 m ∗ ( n + 1 ) 的 矩 阵 X 为m*(n+1)的矩阵 X 为 m ∗ ( n + 1 ) 的 矩 阵 θ 为 ( n + 1 ) ∗ 1 列 的 向 量 \theta为 (n+1) * 1 列的向量 θ 为 ( n + 1 ) ∗ 1 列 的 向 量 y 为 ( n + 1 ) ∗ 1 列 的 向 量 y 为 (n+1) * 1 列的向量 y 为 ( n + 1 ) ∗ 1 列 的 向 量
下标(n) 代表特征(参数)数量θ \theta θ
a r g m i n ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) 2 {argmin}\sum_{i=1}^{m}(y^{(i)} - \hat{y}^{(i)})^2 a r g m i n ∑ i = 1 m ( y ( i ) − y ^ ( i ) ) 2 a r g m i n ( Y − X ⋅ θ ) 2 {argmin}(Y -X \cdot \theta)^2 a r g m i n ( Y − X ⋅ θ ) 2 a r g m i n ( Y − X ⋅ θ ) T ⋅ ( Y − X ⋅ θ ) {argmin}(Y - X \cdot \theta )^T \cdot(Y - X\cdot \theta ) a r g m i n ( Y − X ⋅ θ ) T ⋅ ( Y − X ⋅ θ )
所以 问题就转化成 求上述式子的最小值,求最小值是一个凸优化问题 遇到这种问题 一般 先需要证明 这个式子是有凸函数连续可导。
(以下推导可以不看 直接看结论)
由最小二乘法导出损失函数E_w
证明损失函数E_w 是关于θ的凸函数
对损失函数E_w关于θ求一阶倒数
令已接到数等于0接触θ*
设集合D ⊂ R n D \subset R^n D ⊂ R n x , y ⊂ D x,y \subset D x , y ⊂ D a ⊂ [ 0 , 1 ] a \subset [0,1] a ⊂ [ 0 , 1 ] a x + ( 1 − a ) y ⊂ D ax + (1 - a)y \subset D a x + ( 1 − a ) y ⊂ D 直线 所形成的集合 还在原空间中则称为凸集。
设n原函数f ( x ) f(x) f ( x ) x = ( x 1 , x 2 , . . . , x n ) T x = (x_1,x_2,...,x_n)^T x = ( x 1 , x 2 , . . . , x n ) T x i x_i x i ∂ f ( x ) ∂ x i i = ( 1 , 2 , 3... , n ) \frac {\partial f(x)}{\partial x_i} i=(1,2,3...,n) ∂ x i ∂ f ( x ) i = ( 1 , 2 , 3 . . . , n ) ∇ f ( x ) = [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 . . . ∂ f ( x ) ∂ x n ] \nabla f(x) = \left[
\begin{matrix}
\frac {\partial f(x)}{\partial x_1} \\
\frac {\partial f(x)}{\partial x_2} \\
... \\
\frac {\partial f(x)}{\partial x_n}
\end{matrix}
\right] ∇ f ( x ) = ⎣ ⎢ ⎢ ⎢ ⎡ ∂ x 1 ∂ f ( x ) ∂ x 2 ∂ f ( x ) . . . ∂ x n ∂ f ( x ) ⎦ ⎥ ⎥ ⎥ ⎤ f ( x ) f(x) f ( x ) ∇ f ( x ) \nabla f(x) ∇ f ( x )
Hession(海塞)矩阵定义:设n元函数f ( x ) f(x) f ( x ) x = ( x 1 , x 2 , . . . . , x n ) T x =(x_1,x_2,....,x_n)^T x = ( x 1 , x 2 , . . . . , x n ) T x i x_i x i ∂ 2 f ( x ) ∂ x i ∂ x j ( i , j = 1 , 2 , 3..... , n ) \frac{\partial ^2f(x)}{\partial x_i \partial x_j}(i,j = 1,2,3.....,n) ∂ x i ∂ x j ∂ 2 f ( x ) ( i , j = 1 , 2 , 3 . . . . . , n ) f ( x ) f(x) f ( x ) x x x ∇ 2 f ( x ) = ∇ f ( x ) = [ ∂ 2 f ( x ) ∂ x 1 2 ∂ 2 f ( x ) ∂ x 1 ∂ x 2 . . . ∂ 2 f ( x ) ∂ x 1 ∂ x n ∂ 2 f ( x ) ∂ x 2 ∂ x 1 ∂ 2 f ( x ) ∂ x 2 2 . . . ∂ 2 f ( x ) ∂ x 2 ∂ x n ⋮ ⋮ ⋱ ⋮ ∂ 2 f ( x ) ∂ x n ∂ x 1 ∂ 2 f ( x ) ∂ x n ∂ x 2 . . . ∂ 2 f ( x ) ∂ x n ∂ x n ] \nabla ^2 f(x) =
\nabla f(x) = \left[
\begin{matrix}
\frac {\partial ^2 f(x)}{\partial x_1^2 } & \frac {\partial ^2 f(x)}{\partial x_1 \partial x_2 } & ... & \frac {\partial ^2 f(x)}{\partial x_1\partial x_n} \\
\frac {\partial ^2 f(x)}{\partial x_2 \partial x_1} & \frac {\partial ^2 f(x)}{\partial x_2^2 } & ... & \frac {\partial ^2 f(x)}{\partial x_2\partial x_n} \\
\vdots & \vdots& \ddots & \vdots \\
\frac {\partial ^2 f(x)}{\partial x_n \partial x_1 } & \frac {\partial ^2 f(x)}{\partial x_n \partial x_2 } & ... & \frac {\partial ^2 f(x)}{\partial x_n\partial x_n}
\end{matrix}
\right] ∇ 2 f ( x ) = ∇ f ( x ) = ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎡ ∂ x 1 2 ∂ 2 f ( x ) ∂ x 2 ∂ x 1 ∂ 2 f ( x ) ⋮ ∂ x n ∂ x 1 ∂ 2 f ( x ) ∂ x 1 ∂ x 2 ∂ 2 f ( x ) ∂ x 2 2 ∂ 2 f ( x ) ⋮ ∂ x n ∂ x 2 ∂ 2 f ( x ) . . . . . . ⋱ . . . ∂ x 1 ∂ x n ∂ 2 f ( x ) ∂ x 2 ∂ x n ∂ 2 f ( x ) ⋮ ∂ x n ∂ x n ∂ 2 f ( x ) ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎤ f ( x ) f(x) f ( x ) x x x ∇ 2 f ( x ) \nabla ^2 f(x) ∇ 2 f ( x ) f ( x ) f(x) f ( x ) x x x ∂ 2 f ( x ) ∂ x i x j \frac{\partial ^2f(x)}{\partial x_ix_j} ∂ x i x j ∂ 2 f ( x ) ∂ 2 f ( x ) ∂ x j x i \frac{\partial ^2f(x)}{\partial x_jx_i} ∂ x j x i ∂ 2 f ( x )
设D ⊂ R n D \subset R^n D ⊂ R n f : D ⊂ R n , f:D \subset R^n , f : D ⊂ R n , f ( x ) f(x) f ( x ) D D D f ( x ) f(x) f ( x ) ∇ 2 f ( x ) \nabla ^2 f(x) ∇ 2 f ( x ) D D D 正定的 ,则f ( x ) f(x) f ( x ) 严格凸函数 。f ( x ) f(x) f ( x ) x ∗ x^* x ∗ ∇ f ( x ∗ ) = 0 \nabla f(x ^*) = 0 ∇ f ( x ∗ ) = 0 f ( x ) f(x) f ( x ) x x x
[标量-向量]的矩阵微分公式为: ∂ y ∂ x = ( ∂ y ∂ x 1 ∂ y ∂ x 2 ⋮ ∂ y ∂ x n ) \frac{\partial y}{\partial x} = \left(\begin{matrix}
\frac{\partial y}{\partial x_1} \\
\frac{\partial y}{\partial x_2} \\
\vdots \\
\frac{\partial y}{\partial x_n}
\end{matrix}
\right) ∂ x ∂ y = ⎝ ⎜ ⎜ ⎜ ⎜ ⎛ ∂ x 1 ∂ y ∂ x 2 ∂ y ⋮ ∂ x n ∂ y ⎠ ⎟ ⎟ ⎟ ⎟ ⎞ ( 分 母 布 局 ) (分母布局) ( 分 母 布 局 ) ∂ y ∂ x = ( ∂ y ∂ x 1 ∂ y ∂ x 2 … ∂ y ∂ x n ) \frac{\partial y}{\partial x} = \left(\begin{matrix} \frac{\partial y}{\partial x_1}
\frac{\partial y}{\partial x_2}
\ldots \frac{\partial y}{\partial x_n}
\end{matrix} \right) ∂ x ∂ y = ( ∂ x 1 ∂ y ∂ x 2 ∂ y … ∂ x n ∂ y ) ( 分 子 布 局 ) (分子布局) ( 分 子 布 局 ) x = ( x 1 , x 2 , . . . . . , x n ) T x =(x_1,x_2,.....,x_n)^T x = ( x 1 , x 2 , . . . . . , x n ) T y y y x x x n n n ∂ x T a ∂ x = ∂ a T x ∂ x \frac {\partial x^Ta}{\partial x} =\frac {\partial a^T x}{\partial x} ∂ x ∂ x T a = ∂ x ∂ a T x ∂ y ∂ x = ( ∂ ( a 1 x 1 + a 2 x 2 + . . . + a n x n ) ∂ x 1 ∂ ( a 1 x 1 + a 2 x 2 + . . . + a n x n ) ∂ x 2 ⋮ ∂ ( a 1 x 1 + a 2 x 2 + . . . + a n x n ) ∂ x n ) \frac{\partial y}{\partial x} = \left(\begin{matrix} \frac{\partial (a_1x_1 + a_2x_2 +...+a_nx_n)}{\partial x_1} \\
\frac{\partial (a_1x_1 + a_2x_2 +...+a_nx_n)}{\partial x_2} \\
\vdots \\
\frac{\partial (a_1x_1 + a_2x_2 +...+a_nx_n)}{\partial x_n}
\end{matrix} \right) ∂ x ∂ y = ⎝ ⎜ ⎜ ⎜ ⎜ ⎛ ∂ x 1 ∂ ( a 1 x 1 + a 2 x 2 + . . . + a n x n ) ∂ x 2 ∂ ( a 1 x 1 + a 2 x 2 + . . . + a n x n ) ⋮ ∂ x n ∂ ( a 1 x 1 + a 2 x 2 + . . . + a n x n ) ⎠ ⎟ ⎟ ⎟ ⎟ ⎞ ( a 1 a 2 ⋮ a n ) \left(\begin{matrix} a_1\\a_2 \\ \vdots \\a_n \end{matrix} \right) ⎝ ⎜ ⎜ ⎜ ⎛ a 1 a 2 ⋮ a n ⎠ ⎟ ⎟ ⎟ ⎞ ∂ x T B x ∂ x = ( B + B T ) x \frac{\partial x^TBx}{\partial x} =(B +B^T)x ∂ x ∂ x T B x = ( B + B T ) x
有了以上定理就可以开始证明了:
证明损失函数E θ E\theta E θ θ \theta θ
∂ E θ ∂ θ = ∂ ( ( Y − X ⋅ θ ) T ( Y − X ⋅ θ ) ) ∂ θ \frac{\partial E \theta}{\partial \theta }=\frac{\partial(( Y- X \cdot \theta)^T (Y - X \cdot \theta ))}{\partial \theta} ∂ θ ∂ E θ = ∂ θ ∂ ( ( Y − X ⋅ θ ) T ( Y − X ⋅ θ ) )
=∂ ∂ θ ( Y − θ T X T ) ( Y − X θ ) \frac{\partial}{\partial \theta}( Y-\theta ^T X^T) (Y - X \theta) ∂ θ ∂ ( Y − θ T X T ) ( Y − X θ )
=∂ ∂ θ [ − Y T X θ − θ T X T Y + θ T X T X θ ] \frac{ \partial}{\partial \theta}[-Y^TX\theta - \theta^TX^TY + \theta^TX^TX\theta] ∂ θ ∂ [ − Y T X θ − θ T X T Y + θ T X T X θ ]
=∂ Y T X θ ∂ θ − ∂ θ T X T Y ∂ θ + ∂ θ T X T X θ ∂ θ \frac{ \partial Y^TX \theta}{\partial \theta} -\frac{ \partial \theta ^T X^TY}{\partial \theta} +\frac{ \partial \theta^TX^TX\theta}{\partial \theta} ∂ θ ∂ Y T X θ − ∂ θ ∂ θ T X T Y + ∂ θ ∂ θ T X T X θ
由∂ x T a ∂ x = ∂ a T x ∂ x \frac {\partial x^Ta}{\partial x} =\frac {\partial a^T x}{\partial x} ∂ x ∂ x T a = ∂ x ∂ a T x ∂ y ∂ x = a \frac{\partial y}{\partial x} =a ∂ x ∂ y = a ∂ x T B x ∂ x = ( B + B T ) x \frac{\partial x^TBx}{\partial x} =(B +B^T)x ∂ x ∂ x T B x = ( B + B T ) x
∂ E θ ∂ θ = − X T y − X T y + ( X T X + X T X θ ) \frac{\partial E_\theta}{\partial\theta} = -X^Ty -X^Ty +(X^TX +X^TX\theta) ∂ θ ∂ E θ = − X T y − X T y + ( X T X + X T X θ )
=2 X T ( X θ − Y ) 2X^T(X\theta - Y) 2 X T ( X θ − Y )
再对一阶偏导数求二阶偏导数:∂ 2 E θ ∂ θ ∂ θ T \frac {\partial ^2E \theta}{\partial \theta\partial \theta^T} ∂ θ ∂ θ T ∂ 2 E θ ∂ ∂ θ ( ∂ E θ ∂ θ ) \frac{\partial }{\partial \theta}(\frac{\partial E \theta}{\partial \theta}) ∂ θ ∂ ( ∂ θ ∂ E θ ) ∂ ∂ E θ ( 2 X T X θ − 2 X T Y ) \frac{\partial}{\partial E \theta}(2X^TX\theta - 2X^TY) ∂ E θ ∂ ( 2 X T X θ − 2 X T Y ) 2 X T X 2X^TX 2 X T X
假设X T X X^TX X T X E θ E\theta E θ θ \theta θ E θ E\theta E θ θ \theta θ ∂ E θ ∂ θ \frac{\partial E \theta}{\partial \theta } ∂ θ ∂ E θ 2 X T ( X θ − Y ) = 0 2X^T(X\theta - Y)=0 2 X T ( X θ − Y ) = 0 2 X T X θ − 2 X T Y = 0 2X^TX\theta - 2X^TY =0 2 X T X θ − 2 X T Y = 0 2 X T X θ = 2 X T Y 2X^TX\theta = 2X^TY 2 X T X θ = 2 X T Y X T X θ = X T Y X^TX\theta = X^TY X T X θ = X T Y ( X T X ) − 1 (X^TX)^{-1} ( X T X ) − 1 ( X T X ) − 1 X T X θ = ( X T X ) − 1 X T Y (X^TX)^{-1}X^TX\theta = (X^TX)^{-1}X^TY ( X T X ) − 1 X T X θ = ( X T X ) − 1 X T Y θ = ( X T X ) − 1 X T Y \theta = (X^TX)^{-1}X^TY θ = ( X T X ) − 1 X T Y
以上证明毕：θ = ( X T X ) − 1 X T Y \theta = (X^TX)^{-1}X^TY θ = ( X T X ) − 1 X T Y
问题:时间复杂度高:O(n 3 n^3 n 3 n 2.4 n^{2.4} n 2 . 4
import numpy as np
class LinearRegression :
def __init__ ( self) :
"""初始化linear Regression模型"""
self. coef_ = None
self. interception_ = None
self. _theta = None
def fit_normal ( self, X_train, y_train) :
assert X_train. shape[ 0 ] == y_train. shape[ 0 ] , \
"the size of x_train must be equal to the size of y_train"
X_b = np. hstack( [ np. ones( ( len ( X_train) , 1 ) ) , X_train] )
self. _theta = np . linalg. inv( X_b. T. dot( X_b) ) . dot( X_b. T) . dot( y_train)
self. interception_= self. _theta[ 0 ]
self. coef_ = self. _theta[ 1 : ]
return self
def predict ( self, X_predict) :
assert self. coef_ is not None and self. interception_ is not None , \
"you must run fit_normal before predict"
assert X_predict. shape[ 1 ] == len ( self. coef_) , \
"the size of X_predict must be equal to the size of X_train"
X_predict_b = np. hstack( [ np. ones( ( len ( X_predict) , 1 ) ) , X_predict] )
return X_predict_b. dot( self. _theta)
def score ( self, X_test, y_test) :
y_predict = self. predict( X_test)
return r_Squared( y_test, y_predict)
def __repr__ ( self) :
return "LinearRegression()"
def r_Squared ( y_true, y_predict) :
assert len ( y_true) == len ( y_predict) , \
"the size of y_true must be equal to the size of y_predict"
return 1 - ( mean_squared_error( y_true, y_predict) / np. var( y_true) )
测试
import numpy as np
import matplotlib
import matplotlib. pyplot as plt
from ml_utils. data_split import train_test_split
from sklearn import datasets
boston = datasets. load_boston( )
x = boston. data
y = boston. target
reg = LinearRegression( )
reg. fit_normal( x_train, y_train)
print ( reg. coef_)
print ( reg. interception_)
print ( reg. score( x_test, y_test) )
Sklearn中的线性回归
from sklearn. linear_model import LinearRegression
import numpy as np
import matplotlib
import matplotlib. pyplot as plt
from sklearn import datasets
from sklearn. model_selection import train_test_split
boston = datasets. load_boston( )
x = boston. data
y = boston. target
x_train, x_test, y_train, y_test = train_test_split( x, y, random_state= 666 ) ;
lin_reg = LinearRegression( )
lin_reg. fit( x_train, y_train)
lin_reg. coef_
lin_reg. intercept_
lin_reg. score( x_test, y_test)
浙公网安备 33010602011771号