Creation of a Deep Convolutional Auto-Encoder in Caffe
Volodymyr Turchenko, Artur Luczak
Canadian Centre for Behavioural Neuroscience Department of Neuroscience, University of Lethbridge 4401 University Drive, Lethbridge, AB, T1K 3M4, Canada {vtu, luczak}@uleth.ca
Abstract - The development of a deep (stacked) text file with the description of layers and (ii) Caffe has a convolutional auto-encoder in the Caffe deep learning Matlab wrapper, which is very convenient and allows framework is presented in this paper. We describe getting Caffe results directly into Matlab workspace for simple principles which we used to create this model in their further processing (visualization, etc) [12]. Caffe. The proposed model of convolutional auto- There are several existing solutions/attempts to encoder does not have pooling/unpooling layers yet. The develop and research a CAE model on different results of our experimental research show comparable platforms, but to the best knowledge of the authors there accuracy of dimensionality reduction in comparison with is no current CAE implementation in Caffe yet. The a classic auto-encoder on the example of MNIST dataset. issue of CAE implementation is permanently active in the Caffe users group [13-15]. There are two Keywords – Deep convolutional auto-encoder, machine implementations of one-layer (not-stacked) CAE [16] learning, neural networks, visualization, dimensionality and convolutional Restricted Boltzmann Machine [17] in reduction. Matlab. Mike Swarbrick Jones presented the implementation of CAE in Theano/Lasagne [18-19]. The 1. Introduction Deep Sea team [20], who won a 100,000 US dollar prize (1st place) in the National Data Science Bowl, a data The convolutional auto-encoder (CAE) is one of the science competition where the goal was to classify most wanted architectures in deep learning research. As images of plankton, has reported the use of CAE to pre- an auto-encoder, it is based on the encoder-decoder train only convolutional layers of their network (also paradigm, where an input is first transformed into a implemented in Theano/Lasagne [9]). But they did not typically lower-dimensional space (encoder part) and use this approach in their final winning architecture since then expanded to reproduce the initial data (decoder they did not receive any substantial improvement with part). It is trained in unsupervised fashion allowing it to CAE pre-training. Also there is an existing CAE extract generally useful features from unlabeled data, to implementation in the examples of Torch7 deep learning detect and remove input redundancies and to present framework [21]. essential aspects of analyzing data in robust and The goal of this paper is to present our first results of discriminative representations [1]. Auto-encoders and the practical implementation of a CAE model in Caffe unsupervised learning methods have been widely used in deep learning framework as well as the experimental many scientific and industrial applications, solving research of the proposed model on the example of mainly dimensionality reduction and unsupervised pre- MNIST dataset and a simple visualization technique training tasks. Compared to the architecture of a classic which helped us to receive these results. stacked auto-encoder [2], CAE may be better suited to image processing tasks because it fully utilizes the 2. Creation CAE model in Caffe properties of convolutional neural networks, which have been proven to provide better results on noisy, shifted In the examples of Caffe there are two models which and corrupted image data [3]. Theoretical issues of CAE solve the task of dimensionality reduction. The first is a developments are well described in many research classic stacked auto-encoder, proposed by Hinton et al papers [1, 4-6]. [2], and the second is a Siamese network, proposed by Modern deep learning frameworks, i.e. ConvNet2 LeCun et al [22]. The classic auto-encoder model is well [7], Theano+Lasagne [8-9], Torch7 [10], Caffe [11] and researched and it trains in a purely unsupervised fashion. others, have become very popular tools in the deep The Siamese network consists of two LeNet [3] learning research community since they provide fast architectures coupled in a Siamese way and ended by a deployment of state-of-the-art deep learning models contrastive loss function. Siamese network trains in a along with appropriate training strategies (Stochastic “semi-supervised” fashion, since for forming the training Gradient Descent, AdaDelta, etc.) allowing rapid set we have to label a couple of input images (chosen research progress and emerging commercial randomly), which we are saving into two channels (left applications. Our interest is to apply deep learning and right), by 1, if the images belong to the same class technologies, namely a CAE, for image processing in the and by 0 otherwise. The visualizations how the example neuroscience field. We have chosen Caffe deep learning Caffe implementations of the classic auto-encoder and framework mainly for two reasons: (i) a description of a the Siamese network solve the dimensionality reduction deep neural network is pretty straightforward, it is just a task encoding the test set (10000 examples) of MNIST in a 2-dimensional (2D) space are depicted in Fig. 1 and layer [4], we have noticed, that the developed model is Fig. 2 respectively. very similar to the classic auto-encoder model [2]. The difference is, the first two fully-connected layers of the encoder part have been replaced by two convolutional layers, and the last two fully-connected layers of the decoder part have been replaced by two deconvolution layers. The architecture of the developed CAE model in Caffe is depicted in Fig. 3. Taking into account some similarity between the classic auto-encoder and the developed model of CAE, we have used the following principles during our research: 1. The model should be symmetric in terms of the total size of feature maps and hidden nodes (i.e. number of connections) in both the encoder and decoder parts; 2. Similarly to the example of the classic auto-encoder in Caffe, for the CAE model we have used two loss functions,
during the training and testing and better visualization 3. Experimental results results. It is necessary to note, that in the case of the classic All experimental researches were fulfilled on a fully-connected auto-encoder researched above, the workstation operated under Ubuntu 14.04.2 operation number of trainable parameters (except biases) is equal system. The workstation is equipped with 4-core (8 to the number of connections. But this is not a case of a threads visible in Linux) Inter(R) Xeon(R) [email protected] Table 1. Results of experimental research of classic auto-encoder from Caffe examples Architecture, Data LOSS values, (SIGMOID_CROSS_ENTROPY_LOSS; Visuali- Number of weights, W /W EUCLIDEAN_LOSS), train / test zation, N, Dims Run 01 Run 02 Run 03 classes 784-300-150-75-N- 84/1 2 150.67; 17.40 150.45; 17.40 148.21; 17.02 14 75-150-300-784, 156.15; 18.12 159.97; 18.85 157.87; 18.15 560 K 10 110.50; 10.14 108.04; 9.75 113.24; 10.64 22 113.60; 10.56 112.22; 10.15 116.13; 11.00 30 107.03; 9.56 106.44; 9.38 104.26; 9.09 26 110.76; 10.03 110.25; 9.92 107.64; 9.51
784-500-250-125-N- 43/1 2 147.55; 16.75 146.48; 16.70 144.91; 16.43 17 125-250-500-784, 152.79; 17.44 155.13; 17.97 153.05; 17.56 1097 K 10 105.97; 9.39 103.69; 8.99 103.59; 9.05 23 109.39 ; 9.85 106.32; 9.35 106.45; 9.36 30 93.55; 7.34 94.38; 7.45 90.60; 6.82 25 95.66; 7.51 97.18; 7.76 93.06; 7.08
784-1000-500-250-N- 17/1 2 142.99; 16.08 143.33; 16.15 147.90; 16.85 21 250-500-1000-784, 152.02; 17.40 152.68; 17.53 151.33; 17.20 2819 K 10 105.68; 9.36 102.25; 8.85 107.80; 9.74 25 109.05; 9.80 105.16; 9.15 111.04; 10.16 30 89.51; 6.64 92.91; 7.23 97.33; 7.94 27 91.75; 6.87 95.75; 7.53 99.82; 8.20
784-2000-1000-500-N- 6/1 2 152.25; 17.54 202.58; 26.01 202.58; 26.01 5 500-1000-2000-784, 161.82; 18.98 205.81; 26.44 205.80; 26.44 8138 K 10 130.65; 13.62 139.03; 15.33 201.60; 25.83 12 136.55; 14.44 148.95; 16.85 204.91; 26.28 30 129.13; 13.39 157.92; 18.44 129.75; 13.54 16 134.96; 14.17 166.55; 19.80 135.23; 14.25
784-3000-1500-750-N- 3/1 2 202.64; 26.03 202.64; 26.03 202.64; 26.03 3 750-1500-3000-784, 205.80; 26.44 205.80; 26.44 205.80; 26.44 15957 K 10 202.64; 26.03 202.58; 26.03 202.66; 26.03 7 205.81; 26.44 205.75; 26.43 205.82; 26.44 30 200.96; 25.70 202.64; 26.03 202.64; 26.03 15 204.28; 26.16 205.80; 26.44 205.80; 26.44
CAE since convolutional/deconvolution layers have models is the number of feature maps in conv/deconv fewer trainable parameters, because the sizes of layers and the number of neurons in fully-connected convolutional and deconvolution kernels are the layers in the encoder and decoder parts. In the third trainable parameters in case of CAE. Therefore, in order column of Table 2 we can see (bold font) that both to build a CAE, we have operated with the number of proposed models are practically the same in terms of the connections, i.e. with the sizes of feature maps and total number of elements in feature maps and the number hidden nodes. In this paper we present two architectures: of neurons in the hidden layers of the encoder and (i) Model 1, where we adjusted the number of decoder parts. connections to the best architecture of classic auto- The number of trainable parameters for both models encoder (third row in Table 1) and (ii) Model 2, where and the details of their calculation are specified in the we increased the number of connections after the series last column of Table 2 and in Table 3 respectively. Since of experimental tests of Model 1. the deconvolution operation has the same nature as Table 2 contains the architecture parameters for both convolution [27], we have used the same approach to developed CAE models. In both models we did calculate the number of trainable parameters both in the convolution with kernels conv1 9x9 and conv2 9x9 in encoder and decoder parts. In decoder part, the purpose encoder part and we came back with deconvolution of the deconvolution layer deconv1neur, which kernels deconv2 12x12 and deconv1 17x17 in decoder corresponds to the term (1x1+1)*1 in the 3rd column of part (Fig. 1). We have chosen this size of deconvolution Table 3, is to transform all feature maps of the last kernels in order to restore the same size of MNIST deconvolution layer deconv1 into one restored image image 28x28. The only difference between the two with the same size as the original: 28x28 pixels in case of MNIST. There was an explanation in the Caffe users results in Table 4 are specified for 20000 training group how to do that [28]. Surprisingly, the number of iterations. The comparison of both tables shows that trainable parameters in the encoder part is significantly Model 1 provides the same minimum values of loss bigger than in the decoder part. The major reason for this functions as well as the same number of visualization is the number of trainable parameters between conv2 classes as the best architecture of the classic auto- layer and the first fully-connected layer ip1encode, for encoder (3rd row) from Table 1. The experimental tests example 20500 parameters for the Model 1 (see the 2nd of Model 2 showed better (lower) values of loss column in Table 3). functions reached during the training, and slightly better The results of experimental tests of the two visualization results in comparison with Model 1. The developed CAE models are presented in Table 4, which visualizations showing how Model 2 solves the is organized similarly to Table 1. The ratio Data/C dimensionality reduction task encoding the test set of (number of connections) is specified in the second MNIST in a 2D space are depicted in Figs. 4-6. Similarly column of Table 4. We evaluate 2-, 10- and 30- we have used t-SNE technique [26] to visualize 10- and dimensional CAEs, the number of dimensions 30- dimensional data, produced by the 10- and 30- corresponds to the number of neurons in the last hidden dimensional CAEs. layer ip2encode (see Fig. 3) of the encoder part. The
Table 2. Architecture parameters of two CAE models CAE Size of kernels and number of Size of feature maps and number of Number of Number of hidden nodes hidden nodes connections trainable (C) parameters (W+B) Model 1 784-(9x9x4)-(9x9x2)-125-N- 784-(20x20x4)-(12x12x2)-125-N- 3468 K 22364 125-(12x12x2)-(17x17x2)-784 125-(12x12x2)-(28x28x2)-784 784-1600-288-125-N-125-288-1568-784 Model 2 784-(9x9x8)-(9x9x4)-250-N- 784-(20x20x8)-(12x12x4)-250-N- 8905 K 85726 250-(12x12x4)-(17x17x4)-784 250-(12x12x4)-(28x28x4)-784 784-3200-576-250-N-250-576-3136-784
Table 3. Calculation of the number of trainable parameters in both encoder and decoder parts CAE Number of trainable parameters (W+B) Encoder part Decoder part Model 1 (9x9+1)x4+(9x9+1)x2+164x125+250 = 250+(12x12+1)x2+(17x17+1)x2+(1x1+1)x1 = 22364 328+164+20500+250 = 21242 250+290+580+2 = 1122 Model 2 (9x9+1)x8+(9x9+1)x4+328x250+500 = 500+(12x12+1)x4+(17x17+1)x4+(1x1+1)x1 = 85726 656+328+82000+550 = 83484 500+580+1160+2 = 2242
Table 4. Results of experimental research of developed CAE Architecture, Data LOSS values, (SIGMOID_CROSS_ENTROPY_LOSS; Visuali Number of connections, C /C EUCLIDEAN_LOSS), train / test -zation, N, Dims Run 01 Run 02 Run 03 classes Model 1, 14/1 2 152.68; 17.95 151.83; 17.61 156.65; 18.46 20 784-(20x20x4)-(12x12x2)-125-N- 159.47; 18.71 160.10; 18.76 159.76; 18.66 125-(12x12x2)-(28x28x2)-784, 10 105.14; 9.33 107.86; 9.72 109.37; 9.91 26 3468 K 109.58; 9.88 111.80; 10.24 113.43; 10.39 30 94.53; 7.37 96.02; 7.57 93.58; 7.14 26 98.39; 7.78 98.18; 7.76 96.80; 7.53
Model 2, 5/1 2 142.50; 15.93 143.50; 16.20 143.16; 16.17 22 784-(20x20x8)-(12x12x4)-250-N- 148.74; 16.85 148.33; 16.72 149.68; 16.99 250-(12x12x4)-(28x28x4)-784, 10 90.72; 6.93 91.34; 7.04 93.27; 7.35 26 8905 K 93.12; 7.15 93.20; 7.19 94.91; 7.42 30 71.38; 3.82 70.54; 3.66 71.82; 3.88 26 72.52; 3.86 71.65; 3.69 73.65; 4.00
The learning parameters of the solver of the CAE, but in many cases they were not stable developed CAE with stable architecture were: architectures. Of course, the accepted models and solver_type: SGD, base_lr: 0.006, lr_policy: "fixed", and learning parameters are not unique, we are pretty sure weight_decay: 0.0005. We run several experiments, there are a lot of other configurations, which could changing the architecture and learning parameters of provide stable CAE architectures. The training times for the Model 1 and Model 2 running for 20000 training As mentioned above, during experimental research iterations were 68 and 100 minutes respectively. For we have visualized the feature maps and the outputs of quick reference we have collected all learning the hidden layer neurons (we did not visualize trainable parameters of the classic auto-encoder and the developed filters yet) in order to understand how data are CAE in Table 5. processing inside the CAE. The implementation of such visualization is simple and straightforward thanks to the Matlab wrapper in Caffe. We just created 15 .prototxt files corresponding to the number of CAE layers. After training, and having an appropriate .caffemodel file, we call Caffe for each of those .prototxt files. Received values, produced by each layer were visualized. An example showing how CAE Model 2 encodes and decodes the digit “2” is depicted in Fig. 7. We also calculated the minimum and maximum values for each conv/deconv layer (specified in square brackets in the title of the pictures). This allowed us to understand that in the failed experiments, the outputs of deconv2 and deconv1 layers were saturated, and therefore the pixels of the restored image had the value 0 and the loss values during training were NaN (Not A Number).
Fig. 4. Visualization of MNIST test set in a 2D space Table 5. Learning parameters of classic auto-encoder and by 2-dimensional CAE Model 2 developed CAE in Caffe Learning Training time parameters in file solver.prototxt
Classic base_lr: 0.01, Architecture 784-1000- auto- lr_policy: "step", 500-250-N-250-500- encoder gamma: 0.1 1000-784, 5000 stepsize: 1000, training iterations momentum: 0.9, 2 minutes weight_decay: 0.0005 Deve- base_lr: 0.006, Model 1, Model 2, loped lr_policy: "fixed", 20000 20000 CAE weight_decay: training training 0.0005 iterations iterations 68 100 Fig. 5. Visualization of MNIST test set in a 2D space minutes minutes by 10-dimensional CAE Model 2 + t-SNE All appropriate .prototxt files of the developed CAE along with all Matlab scripts which we have used for all visualizations will be published in the Caffe user group [13]. It is necessary to note, that the developed CAE model is working on the version of Caffe used/distributed by Noh et al [24] (the date of the files in this version is 16.06.2015). We ran the CAE model on the latest version we have (the date of files in this version is 12.07.2015), but the error “Check failed: 4 == bottom[0]->num_axes() (4 vs. 2) Input must have 4 axes, corresponding to (num, channels, height, width)” occurred. It seems, in the newest version, the developers [12] have changed the internal representation of inner product layers: it is a 2-dimensional array now, not 4- dimensional, as it was in the previous version(s). We will be researching how to run the proposed CAE model Fig. 6. Visualization of MNIST test set in a 2D space on the latest version of Caffe in the future. by 30-dimensional CAE Model 2 + t-SNE
Fig. 7. Visualization of encoding and decoding digit “2” by 10-dimensional CAE Model 2
4. Conclusions Autoencoders, Proceedings of the 25th International Conference on Machine Learning (ICML 08), 2008, pp. The development of a deep (stacked) convolutional 1096-1103. 6. G. E. Hinton, A. Krizhevsky, S. D. Wang, Transforming auto-encoder in Caffe deep learning framework and its Auto-Encoders, Lecture Notes in Computer Science, 6791, experimental evaluation are presented in this paper. The 2011, pp. 44-51. paper contains the first research results of our deep 7. Cuda-convnet2, Fast convolutional NNs in C++/CUDA, convolutional auto-encoder. The proposed model is https://github.com/akrizhevsky/cuda-convnet2 simple since it does not contain pooling/unpooling layers 8. Theano, a Python library to define, optimize, and evaluate yet. In contrast to the classic stacked auto-encoder mathematical expressions involving multi-dimensional proposed by Hinton et al [2], convolutional auto- arrays efficiently, http://deeplearning.net/software/theano/ encoders allow using the desirable properties of 9. Lasagne, Lightweight library to build and train neural convolutional neural networks for image data processing networks in Theano, http://lasagne.readthedocs.org/ 10. Torch7, a scientific computing framework, http://torch.ch/ tasks while working within an unsupervised learning 11. Caffe, a deep learning framework, paradigm. The results of our experimental research show http://caffe.berkeleyvision.org/ comparable accuracy in a dimensionality reduction task 12. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. compared with the classic auto-encoder on the example Girshick, S. Guadarrama, T. Darrell, Convolutional MNIST dataset. Architecture for Fast Feature Embedding, During the creation of this convolutional auto- arXiv:1408.5093, 2014. encoder we have used well-known principles, mentioned 13. Caffe User Group, in Section 2 above, which are used by many machine https://groups.google.com/forum/#!forum/caffe-users learning researchers every day. Nevertheless, we believe 14. Way to Convolutional Autoencoder, https://groups.google.com/forum/#!topic/caffe- that our approach and research results, presented in this users/PtbX9H70fsg paper, will help other researchers in general - and the 15. Configurate Convolutional Auto-Encoders in Caffe #1647, Caffe user group in particular - to create efficient deep https://github.com/BVLC/caffe/issues/1647 neural network architectures in future. 16. Matlab/Octave toolbox for deep learning, Application of the developed deep convolutional https://github.com/rasmusbergpalm/DeepLearnToolbox auto-encoder for our tasks in the neuroscience field and 17. Matlab Environment for Deep Architecture Learning, creation of more complex architectures with https://github.com/dustinstansbury/medal pooling/unpooling layers are the directions of our future 18. Convolutional autoencoders in python/theano/lasagne, research. https://swarbrickjones.wordpress.com/2015/04/29/convolut ional-autoencoders-in-pythontheanolasagne/
19. Code for a convolutional autoencoder written on python, Acknowledgements theano, lasagne, nolearn, https://github.com/mikesj- public/convolutional_autoencoder/tree/master We would like to thank the Caffe developers for 20. Classifying plankton with deep neural networks, creating such a powerful framework for deep machine http://benanne.github.io/2015/03/17/plankton.html learning research. We thank Hyeonwoo Noh and Dr. 21. Torch7 auto-encoder demo, David Silver for discussions on some results presented in https://github.com/torch/demos/blob/master/train- this paper and Dr. Eric Chalmers for the editorial help. autoencoder/train-autoencoder.lua 22. R. Hadsell, S. Chopra, Y. LeCun, Dimensionality We gratefully acknowledge the financial support Reduction by Learning an Invariant Mapping, 2006 IEEE provided by the NSERC CREATE BIP program and the Computer Society Conference on Computer Vision and Department of Neuroscience, University of Lethbridge, Pattern Recognition, 2006, pp. 1735-1742. Canada. 23. H. Noh, S. Hong, B. Han, Learning Deconvolution Network for Semantic Segmentation, arXiv:1505.04366v1, References 2015. 24. Modified version of Caffe which support DeconvNet and 1. J. Masci, U. Meier, D. Ciresan, J. Schmidhuber, Stacked DecoupledNet, https://github.com/HyeonwooNoh/caffe Convolutional Auto-Encoders for Hierarchical Feature 25. M. D. Zeiler, R. Fergus, Visualizing and Understanding Extraction, Lecture Notes in Computer Science, 6791, Convolutional Networks, Lecture Notes in Computer 2011, pp. 52-59. Science, 8689, 2014, pp 818-833. 2. G. E. Hinton, R. R. Salakhutdinov, Reducing the 26. L.J.P. van der Maaten, Accelerating t-SNE using Tree- Dimensionality of Data with Neural Networks, Science, Based Algorithms, Journal of Machine Learning Research, vol. 313, no. 5786, 2006, pp. 504-507. 15(Oct), 2014, pp. 3221-3245, 3. Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient- https://lvdmaaten.github.io/tsne/ based Learning Applied to Document Recognition, 27. M. D. Zeiler, G. W. Taylor, R. Fergus, Adaptive Proceedings of the IEEE, Vol. 86, Issue 11, 1998, pp. Deconvolutional Networks for Mid and High Level Feature 2278-2324. Learning, 2011 IEEE International Conference on 4. M. Ranzato, F. J. Huang, Y.-L. Boureau, Y. LeCun, Computer Vision (ICCV’11), 2011, pp. 2018-2025. Unsupervised Learning of Invariant Feature Hierarchies 28. Refactor convolution layer and add deconvolution layer with Applications to Object Recognition, IEEE Conference #1615, https://github.com/BVLC/caffe/pull/1615 on Computer Vision and Pattern Recognition (CVPR 07), 2007, pp. 1-8. 5. P. Vincent, H. Larochelle, Y. Bengio, P.A. Manzagol, Extracting and Composing Robust Features with Denoising