Exercise: Implement deep networks for digit classification 代码示例

Exercise: Implement deep networks for digit classification 代码示例

练习参考Implement deep networks for digit classification

 

       这个练习用到了一个四层的深度神经网络。第一层是数据输入层；第二、三层是稀疏自编码器层，分别取两个稀疏自编码器的隐藏层作为第二、三层；第四层为Softmax分类器，用于分类0到9的手写数字。在训练Softmax分类器后，又对整个网络进行了微调。在微调时，需要把L2~L4这三层作为一个模型进行调整，调用一次minFunc函数，得到调整后的三层的参数值。经测试，发现微调后在测试数据上的准确率有大幅度提高。

 

       微调时需要利用反向传播算法计算三层的梯度值：

STEP 2: Train the first sparse autoencoder

 

addpath minFunc/
options.Method = 'lbfgs';
options.maxIter = 400;
options.display = 'on';

[sae1OptTheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
                                   inputSize, hiddenSizeL1, ...
                                   lambda, sparsityParam, ...
                                   beta, trainData), ...
                              sae1Theta, options);

STEP 2: Train the second sparse autoencoder

 

 

[sae2OptTheta, cost] = minFunc( @(p) sparseAutoencoderCost(p, ...
                                   hiddenSizeL1, hiddenSizeL2, ...
                                   lambda, sparsityParam, ...
                                   beta, sae1Features), ...
                              sae2Theta, options);

STEP 3: Train the softmax classifier

 

 

options.maxIter = 100;
softmaxModel = softmaxTrain(hiddenSizeL2, numClasses, lambda, ...
                            sae2Features, trainLabels, options);
saeSoftmaxOptTheta = softmaxModel.optTheta(:);

 

Step 4: Implement fine-tuning

stackedAECost.m

 

stack{1}.i = data;
for d = 1:numel(stack)-1
    stack{d}.o = sigmoid(stack{d}.w * stack{d}.i + repmat(stack{d}.b,1,M));
    stack{d+1}.i = stack{d}.o;
end
stack{end}.o = sigmoid(stack{end}.w * stack{end}.i + repmat(stack{end}.b,1,M));

Mat = softmaxTheta * stack{end}.o;
Mat = exp(bsxfun(@minus, Mat, max(Mat, [], 1)));
P = bsxfun(@rdivide, Mat, sum(Mat));
Mat = log(P);
WD = lambda / 2 * sum(sum(softmaxTheta.^2));
cost = -sum(sum(groundTruth.*Mat)) / M + WD;
softmaxThetaGrad = -(groundTruth - P) * stack{end}.o' ./ M + lambda.*softmaxTheta;

stack{end}.delta = -softmaxTheta' * (groundTruth - P) .* stack{end}.o .* (1 - stack{end}.o);
for d = numel(stack)-1:-1:1
    stack{d}.delta = stack{d+1}.w' * stack{d+1}.delta .* stack{d}.o .* (1 - stack{d}.o);
end

for d = 1:numel(stack)
    stackgrad{d}.w = stack{d}.delta * stack{d}.i' / M;
    stackgrad{d}.b = mean(stack{d}.delta,2);
end

STEP 5: Finetune softmax model

 

 

options.Method = 'lbfgs';
options.maxIter = 400;
options.display = 'on';
[stackedAEOptTheta, cost] = minFunc( @(p) stackedAECost(p, inputSize, hiddenSizeL2, ...
                                              numClasses, netconfig, ...
                                              lambda, trainData, trainLabels), ...
                              stackedAETheta, options);

STEP 6: Test 
stackedAEPredict.m

 

 

M = size(data, 2);
stack{1}.i = data;
for d = 1:numel(stack)-1
    stack{d}.o = sigmoid(stack{d}.w * stack{d}.i + repmat(stack{d}.b,1,M));
    stack{d+1}.i = stack{d}.o;
end
stack{end}.o = sigmoid(stack{end}.w * stack{end}.i + repmat(stack{end}.b,1,M));
Mat = softmaxTheta * stack{end}.o;
[~,pred] = max(Mat);