Intel Caffe 与原生Caffe
1. 首先安装好docker，拉取intel caffe image:
$ docker pull bvlc/caffe:intel 试着运行： $ docker run -it bvlc/caffe:intel /bin/bash
2. 拉取 intel caffe 源码：
git clone https://github.com/intel/caffe git checkout 1.0
或者下载源码包：
wget https://github.com/intel/caffe/archive/1.1.0.zip unzip 1.0.zip
3. 编译Intel caffe
sudo apt-get -y install python-devel boost boost-devel cmake numpy \ numpy-devel gflags gflags-devel glog glog-devel protobuf protobuf-devel hdf5 \ hdf5-devel lmdb lmdb-devel leveldb leveldb-devel snappy-devel opencv opencv-devel
cp Makefile.config.example Makefile.config # Adjust Makefile.config (for example, if using Anaconda Python, or if cuDNN is desired)
vim Makefile.config
# Intel(r) Machine Learning Scaling Library (uncomment to build with MLSL) USE_MLSL := 1
多线程编译：
$ make -j <number_of_physical_cores> -k
编译过程中会下载MKL 和MKL-DNN：
Download mklml_lnx_2018.0.1.20171227.tgz git clone --no-checkout https://github.com/01org/mkl-dnn.git /home/ubuntu/yuntong/caffe-master/external/mkldnn/tmp
测试编译结果：
make test make runtest
4. 下载和创建mnist数据集：
cd $CAFFE_ROOT ./data/mnist/get_mnist.sh ./examples/mnist/create_mnist.sh Creating lmdb...
5. 写入intel caffe docker中的caffe运行路径：
vim /home/ubuntu/yuntong/caffe-1.0/examples/mnist/train_lenet.sh
#!/usr/bin/env sh set -e #./build/tools/caffe train --solver=examples/mnist/lenet_solver.prototxt $@ /opt/caffe/build/tools/caffe train --solver=examples/mnist/lenet_solver.prototxt $@
6. 设置CPU模式 vim examples/mnist/lenet_solver.prototxt
# solver mode: CPU or GPU #solver_mode: GPU solver_mode: CPU
7. 运行docker，并在docker中运行mnist训练：
sudo docker run -v "/home/ubuntu/yuntong/:/opt/caffe/share" -it bvlc/caffe:intel /bin/bash cd /opt/caffe/share/caffe-1.0 ./examples/mnist/train_lenet.sh
运行结果如下：
ubuntu@k8s-1:~$ sudo docker run -v "/home/ubuntu/yuntong/:/opt/caffe/share" -it bvlc/caffe:intel /bin/bash root@19eaccc415e1:/workspace# cd /opt/caffe/share/caffe-1.0 root@19eaccc415e1:/opt/caffe/share/caffe-1.0# ./examples/mnist/train_lenet.sh I0408 01:33:10.509523 12 caffe.cpp:285] Use CPU. I0408 01:33:10.510561 12 solver.cpp:107] Initializing solver from parameters: test_iter: 100 test_interval: 500 base_lr: 0.01 display: 100 max_iter: 10000 lr_policy: "inv" gamma: 0.0001 power: 0.75 momentum: 0.9 weight_decay: 0.0005 snapshot: 5000 snapshot_prefix: "examples/mnist/lenet" solver_mode: CPU net: "examples/mnist/lenet_train_test.prototxt" train_state { level: 0 stage: "" } I0408 01:33:10.511216 12 solver.cpp:153] Creating training net from net file: examples/mnist/lenet_train_test.prototxt I0408 01:33:10.523326 12 cpu_info.cpp:453] Processor speed [MHz]: 0 I0408 01:33:10.523360 12 cpu_info.cpp:456] Total number of sockets: 8 I0408 01:33:10.523373 12 cpu_info.cpp:459] Total number of CPU cores: 8 I0408 01:33:10.523385 12 cpu_info.cpp:462] Total number of processors: 8 I0408 01:33:10.523396 12 cpu_info.cpp:465] GPU is used: no I0408 01:33:10.523406 12 cpu_info.cpp:468] OpenMP environmental variables are specified: no I0408 01:33:10.523427 12 cpu_info.cpp:471] OpenMP thread bind allowed: yes I0408 01:33:10.523437 12 cpu_info.cpp:474] Number of OpenMP threads: 8 I0408 01:33:10.524194 12 net.cpp:1052] The NetState phase (0) differed from the phase (1) specified by a rule in layer mnist I0408 01:33:10.524220 12 net.cpp:1052] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy I0408 01:33:10.524510 12 net.cpp:207] Initializing net from parameters: I0408 01:33:10.524531 12 net.cpp:208] name: "LeNet" state { phase: TRAIN level: 0 stage: "" } engine: "MKLDNN" compile_net_state { bn_scale_remove: false bn_scale_merge: false } layer { name: "mnist" type: "Data" top: "data" top: "label" include { phase: TRAIN } transform_param { scale: 0.00390625 } data_param { source: "examples/mnist/mnist_train_lmdb" batch_size: 64 backend: LMDB } } layer { name: "conv1" type: "Convolution" bottom: "data" top: "conv1" param { lr_mult: 1 } param { lr_mult: 2 } convolution_param { num_output: 20 kernel_size: 5 stride: 1 weight_filler { type: "xavier" } bias_filler { type: "constant" } } } layer { name: "pool1" type: "Pooling" bottom: "conv1" top: "pool1" pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layer { name: "conv2" type: "Convolution" bottom: "pool1" top: "conv2" param { lr_mult: 1 } param { lr_mult: 2 } convolution_param { num_output: 50 kernel_size: 5 stride: 1 weight_filler { type: "xavier" } bias_filler { type: "constant" } } } layer { name: "pool2" type: "Pooling" bottom: "conv2" top: "pool2" pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layer { name: "ip1" type: "InnerProduct" bottom: "pool2" top: "ip1" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 500 weight_filler { type: "xavier" } bias_filler { type: "constant" } } } layer { name: "relu1" type: "ReLU" bottom: "ip1" top: "ip1" } layer { name: "ip2" type: "InnerProduct" bottom: "ip1" top: "ip2" param { lr_mult: 1 } param { lr_mult: 2 } inner_product_param { num_output: 10 weight_filler { type: "xavier" } bias_filler { type: "constant" } } } layer { name: "loss" type: "SoftmaxWithLoss" bottom: "ip2" bottom: "label" top: "loss" } ……………………………………………. ……………………………………………. ……………………………………………. I0408 01:36:28.103435 12 solver.cpp:312] Iteration 7300, loss = 0.0219446 I0408 01:36:28.103497 12 solver.cpp:333] Train net output #0: loss = 0.0219446 (* 1 = 0.0219446 loss) I0408 01:36:28.103519 12 sgd_solver.cpp:215] Iteration 7300, lr = 0.00662927 I0408 01:36:30.492499 12 solver.cpp:312] Iteration 7400, loss = 0.00484636 I0408 01:36:30.492563 12 solver.cpp:333] Train net output #0: loss = 0.00484634 (* 1 = 0.00484634 loss) I0408 01:36:30.492584 12 sgd_solver.cpp:215] Iteration 7400, lr = 0.00660067 I0408 01:36:32.912159 12 solver.cpp:474] Iteration 7500, Testing net (#0) I0408 01:36:33.992708 12 solver.cpp:563] Test net output #0: accuracy = 0.9905 I0408 01:36:33.992983 12 solver.cpp:563] Test net output #1: loss = 0.0301301 (* 1 = 0.0301301 loss) I0408 01:36:34.019621 12 solver.cpp:312] Iteration 7500, loss = 0.00250706 I0408 01:36:34.019706 12 solver.cpp:333] Train net output #0: loss = 0.00250702 (* 1 = 0.00250702 loss) I0408 01:36:34.020164 12 sgd_solver.cpp:215] Iteration 7500, lr = 0.00657236 I0408 01:36:36.432328 12 solver.cpp:312] Iteration 7600, loss = 0.00537509 I0408 01:36:36.432528 12 solver.cpp:333] Train net output #0: loss = 0.00537505 (* 1 = 0.00537505 loss) I0408 01:36:36.432566 12 sgd_solver.cpp:215] Iteration 7600, lr = 0.00654433 I0408 01:36:39.159704 12 solver.cpp:312] Iteration 7700, loss = 0.034624 I0408 01:36:39.159781 12 solver.cpp:333] Train net output #0: loss = 0.0346239 (* 1 = 0.0346239 loss) I0408 01:36:39.159811 12 sgd_solver.cpp:215] Iteration 7700, lr = 0.00651658 I0408 01:36:41.873411 12 solver.cpp:312] Iteration 7800, loss = 0.00424178 I0408 01:36:41.873672 12 solver.cpp:333] Train net output #0: loss = 0.00424175 (* 1 = 0.00424175 loss) I0408 01:36:41.873694 12 sgd_solver.cpp:215] Iteration 7800, lr = 0.00648911 I0408 01:36:44.552800 12 solver.cpp:312] Iteration 7900, loss = 0.00208136 I0408 01:36:44.553073 12 solver.cpp:333] Train net output #0: loss = 0.00208134 (* 1 = 0.00208134 loss) I0408 01:36:44.553095 12 sgd_solver.cpp:215] Iteration 7900, lr = 0.0064619 I0408 01:36:47.132925 12 solver.cpp:474] Iteration 8000, Testing net (#0) I0408 01:36:48.254405 12 solver.cpp:563] Test net output #0: accuracy = 0.9905 I0408 01:36:48.254935 12 solver.cpp:563] Test net output #1: loss = 0.0278543 (* 1 = 0.0278543 loss) I0408 01:36:48.279563 12 solver.cpp:312] Iteration 8000, loss = 0.0065576 I0408 01:36:48.279626 12 solver.cpp:333] Train net output #0: loss = 0.00655758 (* 1 = 0.00655758 loss) I0408 01:36:48.279647 12 sgd_solver.cpp:215] Iteration 8000, lr = 0.00643496 I0408 01:36:50.693308 12 solver.cpp:312] Iteration 8100, loss = 0.0102435 I0408 01:36:50.694417 12 solver.cpp:333] Train net output #0: loss = 0.0102435 (* 1 = 0.0102435 loss) I0408 01:36:50.694447 12 sgd_solver.cpp:215] Iteration 8100, lr = 0.00640827 I0408 01:36:53.059345 12 solver.cpp:312] Iteration 8200, loss = 0.0111062 I0408 01:36:53.059619 12 solver.cpp:333] Train net output #0: loss = 0.0111061 (* 1 = 0.0111061 loss) I0408 01:36:53.059643 12 sgd_solver.cpp:215] Iteration 8200, lr = 0.00638185 I0408 01:36:55.439267 12 solver.cpp:312] Iteration 8300, loss = 0.0255548 I0408 01:36:55.439332 12 solver.cpp:333] Train net output #0: loss = 0.0255548 (* 1 = 0.0255548 loss) I0408 01:36:55.439357 12 sgd_solver.cpp:215] Iteration 8300, lr = 0.00635567 I0408 01:36:57.821687 12 solver.cpp:312] Iteration 8400, loss = 0.00810484 I0408 01:36:57.821768 12 solver.cpp:333] Train net output #0: loss = 0.00810483 (* 1 = 0.00810483 loss) I0408 01:36:57.821794 12 sgd_solver.cpp:215] Iteration 8400, lr = 0.00632975 I0408 01:37:00.229344 12 solver.cpp:474] Iteration 8500, Testing net (#0) I0408 01:37:01.341504 12 solver.cpp:563] Test net output #0: accuracy = 0.991 I0408 01:37:01.341583 12 solver.cpp:563] Test net output #1: loss = 0.028333 (* 1 = 0.028333 loss) I0408 01:37:01.368783 12 solver.cpp:312] Iteration 8500, loss = 0.00672253 I0408 01:37:01.368850 12 solver.cpp:333] Train net output #0: loss = 0.00672251 (* 1 = 0.00672251 loss) I0408 01:37:01.368876 12 sgd_solver.cpp:215] Iteration 8500, lr = 0.00630407 I0408 01:37:03.789499 12 solver.cpp:312] Iteration 8600, loss = 0.000701985 I0408 01:37:03.789630 12 solver.cpp:333] Train net output #0: loss = 0.000701961 (* 1 = 0.000701961 loss) I0408 01:37:03.789660 12 sgd_solver.cpp:215] Iteration 8600, lr = 0.00627864 I0408 01:37:06.311506 12 solver.cpp:312] Iteration 8700, loss = 0.00329251 I0408 01:37:06.311738 12 solver.cpp:333] Train net output #0: loss = 0.00329248 (* 1 = 0.00329248 loss) I0408 01:37:06.311763 12 sgd_solver.cpp:215] Iteration 8700, lr = 0.00625344 I0408 01:37:08.734477 12 solver.cpp:312] Iteration 8800, loss = 0.0011685 I0408 01:37:08.734781 12 solver.cpp:333] Train net output #0: loss = 0.00116848 (* 1 = 0.00116848 loss) I0408 01:37:08.734805 12 sgd_solver.cpp:215] Iteration 8800, lr = 0.00622847 I0408 01:37:11.223204 12 solver.cpp:312] Iteration 8900, loss = 0.000881624 I0408 01:37:11.223266 12 solver.cpp:333] Train net output #0: loss = 0.000881607 (* 1 = 0.000881607 loss) I0408 01:37:11.223289 12 sgd_solver.cpp:215] Iteration 8900, lr = 0.00620374 I0408 01:37:13.565495 12 solver.cpp:474] Iteration 9000, Testing net (#0) I0408 01:37:14.642087 12 solver.cpp:563] Test net output #0: accuracy = 0.99 I0408 01:37:14.642159 12 solver.cpp:563] Test net output #1: loss = 0.0268256 (* 1 = 0.0268256 loss) I0408 01:37:14.666667 12 solver.cpp:312] Iteration 9000, loss = 0.011516 I0408 01:37:14.666734 12 solver.cpp:333] Train net output #0: loss = 0.011516 (* 1 = 0.011516 loss) I0408 01:37:14.666755 12 sgd_solver.cpp:215] Iteration 9000, lr = 0.00617924 I0408 01:37:17.068984 12 solver.cpp:312] Iteration 9100, loss = 0.00914626 I0408 01:37:17.069262 12 solver.cpp:333] Train net output #0: loss = 0.00914625 (* 1 = 0.00914625 loss) I0408 01:37:17.069284 12 sgd_solver.cpp:215] Iteration 9100, lr = 0.00615496 I0408 01:37:19.455351 12 solver.cpp:312] Iteration 9200, loss = 0.00317596 I0408 01:37:19.455596 12 solver.cpp:333] Train net output #0: loss = 0.00317595 (* 1 = 0.00317595 loss) I0408 01:37:19.455623 12 sgd_solver.cpp:215] Iteration 9200, lr = 0.0061309 I0408 01:37:21.834389 12 solver.cpp:312] Iteration 9300, loss = 0.00890829 I0408 01:37:21.835710 12 solver.cpp:333] Train net output #0: loss = 0.00890827 (* 1 = 0.00890827 loss) I0408 01:37:21.835734 12 sgd_solver.cpp:215] Iteration 9300, lr = 0.00610706 I0408 01:37:24.199872 12 solver.cpp:312] Iteration 9400, loss = 0.0232409 I0408 01:37:24.199946 12 solver.cpp:333] Train net output #0: loss = 0.0232409 (* 1 = 0.0232409 loss) I0408 01:37:24.199970 12 sgd_solver.cpp:215] Iteration 9400, lr = 0.00608343 I0408 01:37:26.601363 12 solver.cpp:474] Iteration 9500, Testing net (#0) I0408 01:37:27.673274 12 solver.cpp:563] Test net output #0: accuracy = 0.989 I0408 01:37:27.673359 12 solver.cpp:563] Test net output #1: loss = 0.0323742 (* 1 = 0.0323742 loss) I0408 01:37:27.698536 12 solver.cpp:312] Iteration 9500, loss = 0.00388906 I0408 01:37:27.698603 12 solver.cpp:333] Train net output #0: loss = 0.00388905 (* 1 = 0.00388905 loss) I0408 01:37:27.698628 12 sgd_solver.cpp:215] Iteration 9500, lr = 0.00606002 I0408 01:37:30.146077 12 solver.cpp:312] Iteration 9600, loss = 0.00205984 I0408 01:37:30.146361 12 solver.cpp:333] Train net output #0: loss = 0.00205983 (* 1 = 0.00205983 loss) I0408 01:37:30.146386 12 sgd_solver.cpp:215] Iteration 9600, lr = 0.00603682 I0408 01:37:32.567978 12 solver.cpp:312] Iteration 9700, loss = 0.00330913 I0408 01:37:32.568212 12 solver.cpp:333] Train net output #0: loss = 0.00330913 (* 1 = 0.00330913 loss) I0408 01:37:32.568235 12 sgd_solver.cpp:215] Iteration 9700, lr = 0.00601382 I0408 01:37:34.955097 12 solver.cpp:312] Iteration 9800, loss = 0.0134696 I0408 01:37:34.955363 12 solver.cpp:333] Train net output #0: loss = 0.0134696 (* 1 = 0.0134696 loss) I0408 01:37:34.955386 12 sgd_solver.cpp:215] Iteration 9800, lr = 0.00599102 I0408 01:37:37.377465 12 solver.cpp:312] Iteration 9900, loss = 0.00235391 I0408 01:37:37.377655 12 solver.cpp:333] Train net output #0: loss = 0.0023539 (* 1 = 0.0023539 loss) I0408 01:37:37.377678 12 sgd_solver.cpp:215] Iteration 9900, lr = 0.00596843 I0408 01:37:39.850847 12 solver.cpp:707] Snapshot begin I0408 01:37:39.859346 12 solver.cpp:769] Snapshotting to binary proto file examples/mnist/lenet_iter_10000.caffemodel I0408 01:37:39.869576 12 sgd_solver.cpp:754] Snapshotting solver state to binary proto file examples/mnist/lenet_iter_10000.solverstate I0408 01:37:39.878753 12 solver.cpp:734] Snapshot end I0408 01:37:39.888120 12 solver.cpp:436] Iteration 10000, loss = 0.00251002 I0408 01:37:39.888172 12 solver.cpp:474] Iteration 10000, Testing net (#0) I0408 01:37:41.067348 12 solver.cpp:563] Test net output #0: accuracy = 0.9913 I0408 01:37:41.067407 12 solver.cpp:563] Test net output #1: loss = 0.0267652 (* 1 = 0.0267652 loss) I0408 01:37:41.067422 12 solver.cpp:443] Optimization Done. I0408 01:37:41.067432 12 caffe.cpp:345] Optimization Done.
花费时间 01:37:41.067432 - 01:33:10.509523 = 4.31分钟
CPU及IO利用率：
8个CPU基本达到100%
IO很小，MNIST数据集只有几十M，数据都被cache了
8. 加上MKL2017
./examples/mnist/train_lenet.sh -engine "MKL2017"
第一次： 03:01:35.904659 -02:58:30.774215 = 2：55分钟
第二次： 03:05:15.134409 - 03:02:13.449990 = 2：58分钟
对于原生Caffe
docker run -ti bvlc/caffe:cpu caffe –version
sudo docker run -v "/home/ubuntu/yuntong/:/opt/caffe/share" -it bvlc/caffe:cpu /bin/bash
./examples/mnist/train_lenet.sh
运行时间24分钟。
原生caffe 只能在一个线程上跑
运行cifar10
该数据集共有60000张彩色图像，这些图像是32*32，分为10个类，每类6000张图。这里面有50000张用于训练，构成了5个训练批，每一批10000张图；另外10000用于测试，单独构成一批。测试批的数据里，取自10类中的每一类，每一类随机取1000张。抽剩下的就随机排列组成了训练批。注意一个训练批中的各类图像并不一定数量相同，总的来看训练批，每一类都有5000张图。
下面这幅图就是列举了10各类，每一类展示了随机的10张图片：
1. 下载和创建cifar10数据集：
cd $CAFFE_ROOT ./data/mnist/get_cifar10.sh ./examples/cifar10/create_cifar10.sh Creating lmdb...
2. 写入intel caffe docker中的caffe运行路径： ~/yuntong/caffe-1.0/examples$ vim cifar10/train_quick.sh
TOOLS=/opt/caffe/build/tools
3. 设置CPU模式：
vim cifar10_quick_solver_lr1.prototxt cifar10_quick_solver.prototxt
4. Intel Caffe里面运行:
sudo docker run -v "/home/ubuntu/yuntong/:/opt/caffe/share" -it bvlc/caffe:intel /bin/bash cd /opt/caffe/share/caffe-1.0 ./examples/cifar10/train_quick.sh
训练时间： 07:20:47.795905 - 07:08:08.193487 = 12分40
5. 原生 Caffe里面运行:
sudo docker run -v "/home/ubuntu/yuntong/:/opt/caffe/share" -it bvlc/caffe:cpu /bin/bash cd /opt/caffe/share/caffe-1.0 ./examples/cifar10/train_quick.sh
07:26:23.522944 …………… I0408 08:56:53.116524 18 solver.cpp:310] Iteration 5000, loss = 0.449847 I0408 08:56:53.117141 18 solver.cpp:330] Iteration 5000, Testing net (#0) I0408 08:57:30.968313 21 data_layer.cpp:73] Restarting data prefetching from start. I0408 08:57:32.527096 18 solver.cpp:397] Test net output #0: accuracy = 0.7561 I0408 08:57:32.527354 18 solver.cpp:397] Test net output #1: loss = 0.72683 (* 1 = 0.72683 loss) I0408 08:57:32.527364 18 solver.cpp:315] Optimization Done. I0408 08:57:32.527381 18 caffe.cpp:259] Optimization Done.
用时 1.5小时
机器配置：
CPU: 8 * Intel(R) Core(TM) i5-3427U CPU @ 1.80GHz Memory: 16G Storage: SATA SSD
