Regularization —— linear regression

本节主要是练习regularization项的使用原则。因为在机器学习的一些模型中，如果模型的参数太多，而训练样本又太少的话，这样训练出来的模型很容易产生过拟合现象。因此在模型的损失函数中，需要对模型的参数进行“惩罚”，这样的话这些参数就不会太大，而越小的参数说明模型越简单，越简单的模型则越不容易产生过拟合现象。

Regularized linear regression

ex5Lin0

From looking at this plot, it seems that fitting a straight line might be too simple of an approximation. Instead, we will try fitting a higher-order polynomial to the data to capture more of the variations in the points.

Let's try a fifth-order polynomial. Our hypothesis will be

\begin{displaymath} h_{\theta}(x)=\theta_{0}+\theta_{1}x+\theta_{2}x^{2}+\theta_{3}x^{3}+ \theta_{4}x^{4}+\theta_{5}x^{5}\end{displaymath}

This means that we have a hypothesis of six features, because  $x^0, x^1, \ldots x^5$ are now all features of our regression. Notice that even though we are producing a polynomial fit, we still have a linear regression problem because the hypothesis is linear in each feature.

Since we are fitting a 5th-order polynomial to a data set of only 7 points, over-fitting is likely to occur. To guard against this, we will use regularization in our model.

Recall that in regularization problems, the goal is to minimize the following cost function with respect to $\theta$:

\begin{eqnarray*} J(\theta) & = & \frac{{1}}{2m}\left[\sum_{i=1}^{m}(h_{\theta}(... ... & & \mbox{where }\lambda\mbox{ is the regularization parameter}\end{eqnarray*}

The regularization parameter $\lambda$ is a control on your fitting parameters. As the magnitues of the fitting parameters increase, there will be an increasing penalty on the cost function. This penalty is dependent on the squares of the parameters as well as the magnitude of $\lambda$. Also, notice that the summation after $\lambda$ does not include $\theta_{0}^{2}.$

 

lamda 越大，训练出的模型越简单 —— 后一项的惩罚越大

Normal equations

Now we will find the best parameters of our model using the normal equations. Recall that the normal equations solution to regularized linear regression is

\begin{displaymath} \theta=(X^{T}X+\lambda\left[\begin{array}{cccc} 0\\ & 1\\ & & \ddots\\ & & & 1\end{array}\right])^{-1}X^{T}\vec{y}\end{displaymath}

The matrix following $\lambda$ is an $(n+1)\times(n+1)$ diagonal matrix with a zero in the upper left and ones down the other diagonal entries. (Remember that $n$ is the number of features, not counting the intecept term). The vector $\vec{y}$ and the matrix $X$ have the same definition they had for unregularized regression:

\begin{displaymath} \begin{array}{cc} \par \vec{y} = \left[ \begin{array}{c} y... ...m)})^T\mbox{-} \end{array}\right] \nonumber \par \end{array}\end{displaymath}

Using this equation, find values for $\theta$ using the three regularization parameters below:

a. $\lambda = 0$ (this is the same case as non-regularized linear regression)

b. $\lambda = 1$

c. $\lambda = 10$

Code

clc,clear
%加载数据
x = load('ex5Linx.dat');
y = load('ex5Liny.dat');

%显示原始数据
plot(x,y,'o','MarkerEdgeColor','b','MarkerFaceColor','r')

%将特征值变成训练样本矩阵
x = [ones(length(x),1) x x.^2 x.^3 x.^4 x.^5];  
[m n] = size(x);
n = n -1;

%计算参数sidta，并且绘制出拟合曲线
rm = diag([0;ones(n,1)]);%lamda后面的矩阵
lamda = [0 1 10]';
colortype = {'g','b','r'};
sida = zeros(n+1,3); %初始化参数sida
xrange = linspace(min(x(:,2)),max(x(:,2)))';
hold on;
for i = 1:3
    sida(:,i) = inv(x'*x+lamda(i).*rm)*x'*y;%计算参数sida
    norm_sida = norm(sida) % norm 求sida的2阶范数
    yrange = [ones(size(xrange)) xrange xrange.^2 xrange.^3,...
        xrange.^4 xrange.^5]*sida(:,i);
    plot(xrange',yrange,char(colortype(i)))
    hold on
end
legend('traning data', '\lambda=0', '\lambda=1','\lambda=10')%注意转义字符的使用方法
hold off

ex5Lin0

ex5Lin1

 

 

ex5Lin10

posted @ 2014-09-12 15:38  老姨  阅读(536)  评论(0)  编辑  收藏  举报