Regularized Logistic Regression

前言：

上一篇博文介绍到当模型参数过多时，线性回归模型会出现过抑合现象，同样逻辑回归模型也会出现该情况，因此也需要通过正则项来解决该问题。本节内容参考：http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=DeepLearning&doc=exercises/ex5/ex5.html

理论基础：

目标函数：

 （未对正则，如果对，当lambda特别大的时候，参数全部为0，求得的模型为 y = 0，无意义。 

可以看出，当样本数目m比较大的时候，，过抑合趋势减小。这也就是为什么我们要增大训练样本数目的原因。

牛顿法迭代函数：

实验结果：

分别当时，计算模型参数，并画出相应的分界面。

当时，分界面和作者的有点不同，不过实在找不出bug在哪儿 --！

 

实验代码：

regularized_logistic_regression.m

clc,clear,close all;
x = load('ex5Logx.dat');
y = load('ex5Logy.dat');
pos = find( y==1 ); neg = find( y==0 );

X = map_feature(x(:,1), x(:,2)); % each column is a feature

lambda = [0 1 10];
MAX_ITER = 20;
for i = 1 : length(lambda)
    %Newton's method
    theta = Newton( X , y ,lambda(i) , MAX_ITER );
    figure;
    plot(x(pos,1) , x(pos,2) , '+' );
    hold on;
    plot(x(neg,1) , x(neg,2) , 'o' , 'MarkerFaceColor' , 'y');
    drawDecisionBoundray( theta );
    legend('y=1 ' , 'y=0' , 'Decision boundary');
    
    title( strcat( '\lambda = ' , int2str(lambda(i)) ) );
    hold off;
end

 

Newton.m

function theta = Newton(x ,y , lambda , MAX_ITER)
[m n] = size(x);
theta = zeros( n , 1 );
g = inline('1.0 ./ (1.0 + exp(-z) )' );
for j = 1:MAX_ITER
    h = g( x*theta );

    grad_reg_term = (lambda/m) .* theta; grad_reg_term(1) = 0;
    H_reg_term = (lambda/m) .* eye(n);   H_reg_term(1) = 0;

    grad_reg = 1/m * x' * (h-y) + grad_reg_term; % n*1
    H = (1/m) .* x' * diag(h) *diag(1-h) * x + H_reg_term; % n*n
    theta_new  = theta- H \ grad_reg; % n*!
    if abs( theta_new - theta ) < 1e-10
        theta = theta_new;
        break;
    end
    theta = theta_new;
       
end
fprintf('iteration num = %d\n' , j);
end

drawDecisionBoundray.m

function drawDecisionBoundray( theta )
u = linspace(-1, 1.5, 200);
v = linspace(-1, 1.5, 200);

% Initialize space for the values to be plotted
z = zeros(length(u), length(v));

% Evaluate z = theta*x over the grid
for i = 1:length(u)
    for j = 1:length(v)
        % Notice the order of j, i here!
        z(j,i) = map_feature(u(i), v(j))*theta;
    end
end

% Because of the way that contour plotting works
% in Matlab, we need to transpose z, or
% else the axis orientation will be flipped!
z = z';
% Plot z = 0 by specifying the range [0, 0]
contour(u,v,z, [0, 0], 'LineWidth', 2);
end

 

参考资料：
http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=DeepLearning&doc=exercises/ex5/ex5.html
http://www.cnblogs.com/dupuleng/articles/4171491.html