Deep learning：十四(Softmax Regression练习)

转载自：Deep learning：十四(Softmax Regression练习)

前言：

　　这篇文章主要是用来练习softmax regression在多分类器中的应用，关于该部分的理论知识已经在前面的博文中Deep learning：十三(Softmax Regression)有所介绍。本次的实验内容是参考网页：http://deeplearning.stanford.edu/wiki/index.php/Exercise:Softmax_Regression。主要完成的是手写数字识别，采用的是MNIST手写数字数据库，其中训练样本有6万个，测试样本有1万个，且数字是0~9这10个。每个样本是一张小图片，大小为28*28的。

　　实验环境：matlab2012a

实验基础：

　　这次实验只用了softmax模型，也就是说没有隐含层，而只有输入层和输出层，因为实验中并没有提取出MINST样本的特征，而是直接用的原始像素特征。实验中主要是计算系统的损失函数和其偏导数，其计算公式如下所示：

一些matlab函数：

sparse:

　　生成一个稀疏矩阵，比如说sparse(A, B, k)，，其中A和B是个向量，k是个常量。这里生成的稀疏矩阵的值都为参数k，稀疏矩阵位置值坐标点有A和B相应的位置点值构成。

full:

　　生成一个正常矩阵，一般都是利用稀疏矩阵来还原的。

实验错误：

　　按照作者给的starter code，结果连数据都加载不下来，出现如下错误提示：Error using permute Out of memory. Type HELP MEMORY for your options. 结果跟踪定位到loadMNISTImages.m文件中的images = permute(images,[2 1 3])这句代码，究其原因就是说images矩阵过大，在有限内存下不能够将其进行维度旋转变换。可是这个数据已经很小了，才几十兆而已，参考了很多out of memory的方法都不管用，后面直接把改句的前面一句代码images = reshape(images, numCols, numRows, numImages);改成images = reshape(images, numRows, numCols, numImages);反正实现的效果都是一样的。因为原因是内存问题，所以要么用64bit的matlab，要买自己对该函数去优化下，节省运行过程中的内存。

实验结果：

　　Accuracy: 92.640%

　　和网页教程中给的结果非常相近了。

实验主要部分代码：

softmaxExercise.m:

  1 %% CS294A/CS294W Softmax Exercise 
  2 
  3 %  Instructions
  4 %  ------------
  5 % 
  6 %  This file contains code that helps you get started on the
  7 %  softmax exercise. You will need to write the softmax cost function 
  8 %  in softmaxCost.m and the softmax prediction function in softmaxPred.m. 
  9 %  For this exercise, you will not need to change any code in this file,
 10 %  or any other files other than those mentioned above.
 11 %  (However, you may be required to do so in later exercises)
 12 
 13 %%======================================================================
 14 %% STEP 0: Initialise constants and parameters
 15 %
 16 %  Here we define and initialise some constants which allow your code
 17 %  to be used more generally on any arbitrary input. 
 18 %  We also initialise some parameters used for tuning the model.
 19 
 20 inputSize = 28 * 28; % Size of input vector (MNIST images are 28x28)
 21 numClasses = 10;     % Number of classes (MNIST images fall into 10 classes)
 22 
 23 lambda = 1e-4; % Weight decay parameter
 24 
 25 %%======================================================================
 26 %% STEP 1: Load data
 27 %
 28 %  In this section, we load the input and output data.
 29 %  For softmax regression on MNIST pixels, 
 30 %  the input data is the images, and 
 31 %  the output data is the labels.
 32 %
 33 
 34 % Change the filenames if you've saved the files under different names
 35 % On some platforms, the files might be saved as 
 36 % train-images.idx3-ubyte / train-labels.idx1-ubyte
 37 
 38 images = loadMNISTImages('train-images.idx3-ubyte');
 39 labels = loadMNISTLabels('train-labels.idx1-ubyte');
 40 labels(labels==0) = 10; % Remap 0 to 10
 41 
 42 inputData = images;
 43 
 44 % For debugging purposes, you may wish to reduce the size of the input data
 45 % in order to speed up gradient checking. 
 46 % Here, we create synthetic dataset using random data for testing
 47 
 48 % DEBUG = true; % Set DEBUG to true when debugging.
 49 DEBUG = false;
 50 if DEBUG
 51     inputSize = 8;
 52     inputData = randn(8, 100);
 53     labels = randi(10, 100, 1);
 54 end
 55 
 56 % Randomly initialise theta
 57 theta = 0.005 * randn(numClasses * inputSize, 1);%输入的是一个列向量
 58 
 59 %%======================================================================
 60 %% STEP 2: Implement softmaxCost
 61 %
 62 %  Implement softmaxCost in softmaxCost.m. 
 63 
 64 [cost, grad] = softmaxCost(theta, numClasses, inputSize, lambda, inputData, labels);
 65                                      
 66 %%======================================================================
 67 %% STEP 3: Gradient checking
 68 %
 69 %  As with any learning algorithm, you should always check that your
 70 %  gradients are correct before learning the parameters.
 71 % 
 72 
 73 if DEBUG
 74     numGrad = computeNumericalGradient( @(x) softmaxCost(x, numClasses, ...
 75                                     inputSize, lambda, inputData, labels), theta);
 76 
 77     % Use this to visually compare the gradients side by side
 78     disp([numGrad grad]); 
 79 
 80     % Compare numerically computed gradients with those computed analytically
 81     diff = norm(numGrad-grad)/norm(numGrad+grad);
 82     disp(diff); 
 83     % The difference should be small. 
 84     % In our implementation, these values are usually less than 1e-7.
 85 
 86     % When your gradients are correct, congratulations!
 87 end
 88 
 89 %%======================================================================
 90 %% STEP 4: Learning parameters
 91 %
 92 %  Once you have verified that your gradients are correct, 
 93 %  you can start training your softmax regression code using softmaxTrain
 94 %  (which uses minFunc).
 95 
 96 options.maxIter = 100;
 97 %softmaxModel其实只是一个结构体，里面包含了学习到的最优参数以及输入尺寸大小和类别个数信息
 98 softmaxModel = softmaxTrain(inputSize, numClasses, lambda, ...
 99                             inputData, labels, options);
100                           
101 % Although we only use 100 iterations here to train a classifier for the 
102 % MNIST data set, in practice, training for more iterations is usually
103 % beneficial.
104 
105 %%======================================================================
106 %% STEP 5: Testing
107 %
108 %  You should now test your model against the test images.
109 %  To do this, you will first need to write softmaxPredict
110 %  (in softmaxPredict.m), which should return predictions
111 %  given a softmax model and the input data.
112 
113 images = loadMNISTImages('t10k-images.idx3-ubyte');
114 labels = loadMNISTLabels('t10k-labels.idx1-ubyte');
115 labels(labels==0) = 10; % Remap 0 to 10
116 
117 inputData = images;
118 size(softmaxModel.optTheta)
119 size(inputData)
120 
121 % You will have to implement softmaxPredict in softmaxPredict.m
122 [pred] = softmaxPredict(softmaxModel, inputData);
123 
124 acc = mean(labels(:) == pred(:));
125 fprintf('Accuracy: %0.3f%%\n', acc * 100);
126 
127 % Accuracy is the proportion of correctly classified images
128 % After 100 iterations, the results for our implementation were:
129 %
130 % Accuracy: 92.200%
131 %
132 % If your values are too low (accuracy less than 0.91), you should check 
133 % your code for errors, and make sure you are training on the 
134 % entire data set of 60000 28x28 training images 
135 % (unless you modified the loading code, this should be the case)

View Code

softmaxCost.m

 1 function [cost, grad] = softmaxCost(theta, numClasses, inputSize, lambda, data, labels)
 2 
 3 % numClasses - the number of classes 
 4 % inputSize - the size N of the input vector
 5 % lambda - weight decay parameter
 6 % data - the N x M input matrix, where each column data(:, i) corresponds to
 7 %        a single test set
 8 % labels - an M x 1 matrix containing the labels corresponding for the input data
 9 %
10 
11 % Unroll the parameters from theta
12 theta = reshape(theta, numClasses, inputSize);%将输入的参数列向量变成一个矩阵
13 
14 numCases = size(data, 2);%输入样本的个数
15 groundTruth = full(sparse(labels, 1:numCases, 1));%这里sparse是生成一个稀疏矩阵，该矩阵中的值都是第三个值1
16                                                     %稀疏矩阵的小标由labels和1:numCases对应值构成
17 cost = 0;
18 
19 thetagrad = zeros(numClasses, inputSize);
20 
21 %% ---------- YOUR CODE HERE --------------------------------------
22 %  Instructions: Compute the cost and gradient for softmax regression.
23 %                You need to compute thetagrad and cost.
24 %                The groundTruth matrix might come in handy.
25 
26 M = bsxfun(@minus,theta*data,max(theta*data, [], 1));
27 M = exp(M);
28 p = bsxfun(@rdivide, M, sum(M));
29 cost = -1/numCases * groundTruth(:)' * log(p(:)) + lambda/2 * sum(theta(:) .^ 2);
30 thetagrad = -1/numCases * (groundTruth - p) * data' + lambda * theta;
31 
32 
33 
34 % ------------------------------------------------------------------
35 % Unroll the gradient matrices into a vector for minFunc
36 grad = [thetagrad(:)];
37 end

View Code

softmaxTrain.m:

 1 function [softmaxModel] = softmaxTrain(inputSize, numClasses, lambda, inputData, labels, options)
 2 %softmaxTrain Train a softmax model with the given parameters on the given
 3 % data. Returns softmaxOptTheta, a vector containing the trained parameters
 4 % for the model.
 5 %
 6 % inputSize: the size of an input vector x^(i)
 7 % numClasses: the number of classes 
 8 % lambda: weight decay parameter
 9 % inputData: an N by M matrix containing the input data, such that
10 %            inputData(:, c) is the cth input
11 % labels: M by 1 matrix containing the class labels for the
12 %            corresponding inputs. labels(c) is the class label for
13 %            the cth input
14 % options (optional): options
15 %   options.maxIter: number of iterations to train for
16 
17 if ~exist('options', 'var')
18     options = struct;
19 end
20 
21 if ~isfield(options, 'maxIter')
22     options.maxIter = 400;
23 end
24 
25 % initialize parameters
26 theta = 0.005 * randn(numClasses * inputSize, 1);
27 
28 % Use minFunc to minimize the function
29 addpath minFunc/
30 options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
31                           % function. Generally, for minFunc to work, you
32                           % need a function pointer with two outputs: the
33                           % function value and the gradient. In our problem,
34                           % softmaxCost.m satisfies this.
35 minFuncOptions.display = 'on';
36 
37 [softmaxOptTheta, cost] = minFunc( @(p) softmaxCost(p, ...
38                                    numClasses, inputSize, lambda, ...
39                                    inputData, labels), ...                                   
40                               theta, options);
41 
42 % Fold softmaxOptTheta into a nicer format
43 softmaxModel.optTheta = reshape(softmaxOptTheta, numClasses, inputSize);
44 softmaxModel.inputSize = inputSize;
45 softmaxModel.numClasses = numClasses;
46                           
47 end

View Code

softmaxPredict.m:

 1 function [pred] = softmaxPredict(softmaxModel, data)
 2 
 3 % softmaxModel - model trained using softmaxTrain
 4 % data - the N x M input matrix, where each column data(:, i) corresponds to
 5 %        a single test set
 6 %
 7 % Your code should produce the prediction matrix 
 8 % pred, where pred(i) is argmax_c P(y(c) | x(i)).
 9  
10 % Unroll the parameters from theta
11 theta = softmaxModel.optTheta;  % this provides a numClasses x inputSize matrix
12 pred = zeros(1, size(data, 2));
13 
14 %% ---------- YOUR CODE HERE --------------------------------------
15 %  Instructions: Compute pred using theta assuming that the labels start 
16 %                from 1.
17 
18 
19 [nop, pred] = max(theta * data);
20 %  pred= max(peed_temp);
21 
22 
23 % ---------------------------------------------------------------------
24 
25 end

View Code

参考资料：

posted on 2014-11-26 11:37 fzyzwrj 阅读(86) 评论(0) 收藏举报