Loading...
|
Please use this identifier to cite or link to this item:
https://nccur.lib.nccu.edu.tw/handle/140.119/122287
|
| Title: | 基於卷積核冗餘的神經網路壓縮機制
Compression of Convolutional Neural Networks Based on Kernel Redundancy |
| Authors: | 陳乃瑋
Chen, Nai-Wei |
| Contributors: | 廖文宏
Liao, Wen-Hung
陳乃瑋
Chen,Nai-Wei |
| Keywords: | 卷積神經網路
模型壓縮
多樣性演算法
卷積核相似度
Convolutional neural network
Model compression
Diversifying algorithm
Kernel similarity |
| Date: | 2018 |
| Issue Date: | 2019-02-12 15:47:36 (UTC+8) |
| Abstract: | 針對卷積神經網路模型與所需計算量過於龐大，導致難以將之部署於行動裝置或嵌入式平台的問題，本論文提出了多樣性演算法用以對CNN模型做壓縮。多樣性演算法的核心概念為保留各個卷積層中最具代表性的濾波器，僅維持卷積核之間的多樣性。我們將卷積層內的濾波器的相似關係以無向圖來表示，每一個節點皆代表一個濾波器，節點與節點之間邊的值則是兩個濾波器之間的餘弦距離，而哪些節點會被移除則是透過邊的值、該節點邊的值加總以及該節點所代表濾波器的權值絕對值加總來決定。經過剪枝後的模型，每一層卷積層的輸出通道數都會有所減少，參數量與浮點數運算量也會相應的降低，且不會產生稀疏網路的問題，而壓縮後的模型最後會再重新訓練使辨識準確率回復至原先的水準。
為了測試多樣性演算法的泛用性，我們使用VGG、ResNet、DenseNet系列模型與CIFAR-10、CIFAR-100資料集進行了十組的壓縮實驗。為了找到最佳的壓縮參數設置，我們在每組實驗上都做了非常充分的壓縮測試，並詳細記錄所有的實驗結果做為調整參數的重要依據，從各組實驗的壓縮結果顯示，多樣性演算法確實是有效的。在CIFAR-10資料集上，我們的方法可以減少VGG16 78.6%的參數量與約46%的浮點數運算量，而在容許約1%準確率差異的情況下，則可以減少90.7%的參數量與接近70%的浮點數運算量。在CIFAR-100資料集上，可以減少VGG16 46%的參數量與18%的浮點數運算量，而在容許約1%準確率差異的情況下，則可以減少約60.7%的參數量與接近37.5%的浮點數運算量。
The model size and floating-point operations (FLOP) required by convolutional neural networks make it difficult to deploy these models to mobile devices or embedded systems. In this thesis, we propose a method known as the diversifying algorithm to compress CNN models. The key concept is to maintain the diversity of convolutional kernels by preserving the most representative filters in each network layer. This is achieved by expressing the network architecture as an undirected graph. The nodes in the graph denote the filters, and the weights are computed using cosine distance. Nodes are removed by considering the combined effects of several factors, including edge weights, the sum of similarity and the sum of weights. After pruning, the number of output channels in each convolutional layer will be reduced. The compressed network is then retrained to retain accuracy with fewer model parameters, FLOPs and also avoid the problem of sparse representation.
We test the efficacy of the proposed diversifying algorithm on three types of CNN models, including VGG, ResNet and DenseNet using both CIFAR-10 and CIFAR-100 datasets through extensive experiments. On CIFAR-10 dataset, our method is able to reduce 78.6% of total parameters and nearly 46% FLOPs in VGG16. If 1% performance loss is allowed, we can achieve 90.7% parameter and 70% FLOP reduction. On CIFAR-100 dataset, we can reduce 46% parameter and 18% FLOP. Furthermore, we can achieve 60.7% parameter and nearly 37.5% FLOP reduction if 1% accuracy loss is allowed. |
| Reference: | [1] Song Han, Huizi Mao, William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv:1510.00149v5, Feb 2016.
[2] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf. Pruning Filters for Efficient ConvNets. arXiv:1608.08710v3, Mar 2017.
[3] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. arXiv:1602.07360v4, Nov 2016.
[4] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:1704.04861v1, Apr 2017.
[5] Alex Krizhevsky, Vinod Nair, Geoffrey Hinton. CIFAR-10 and CIFAR-100 datasets. https://www.cs.toronto.edu/~kriz/cifar.html, last visited on Jan 2018.
[6] 李宏毅, [DSC 2016]系列活動: 李宏毅/一天搞懂深度學習, https://www.slideshare.net/tw_dsconf/ss-62245351, last visited on Jan 2018.
[7] Yann LeCun, Corinna Cortes, Christopher J.C. Burges. THE MNIST DATABASE of handwritten digits. http://yann.lecun.com/exdb/mnist/, last visited on Oct 2018
[8] ImageNet. http://www.image-net.org/, last visited on Jan 2018.
[9] ImageNet Large Scale Visual Recognition Competition (ILSVRC). http://www.image-net.org/challenges/LSVRC/, last visited on Jan 2018.
[10] Yuanqing Lin, Fengjun Lv, Shenghuo Zhu, Ming Yang, Timothee Cour, Kai Yu, Liangliang Cao, Thomas Huang. Large-scale image classification: Fast feature extraction and SVM training. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference, pages 1689-1696. IEEE, 2011.
[11] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[12] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich. Going Deeper with Convolutions. arXiv:1409.4842v1, Sep 2014.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep Residual Learning for Image Recognition. IEEE, pages 770-778, 2016.
[14] Kaiming He. Learning Deep Features for Visual Recognition. http://deeplearning.csail.mit.edu/cvpr2017_tutorial_kaiminghe.pdf, last visited on Oct 2018.
[15] Embedded Systems Developer Kits, Modules, & SDKs | NVIDIA Jetson. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems-dev-kits-modules/, last visited on Oct 2018.
[16] Raspberry Pi. https://www.raspberrypi.org/, last visited on Oct 2018.
[17] Song Han, Jeff Pool, John Tran, William J. Dally. Learning both Weights and Connections for Efficient Neural Networks. arXiv:1506.02626v3, Oct 2015.
[18] Babajide O. Ayinde, Jacek M. Zurada. Building Efficient ConvNets using Redundant Feature Pruning. arXiv:1802.07653v1, Feb 2018.
[19] Karen Simonyan, Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556v6, Apr 2015.
[20] Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger. Densely Connected Convolutional Networks. arXiv:1608.06993v5, Jan 2018.
[21] Sergey Ioffe, Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167v3, Mar 2015.
[22] Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber. Highway Networks. arXiv:1505.00387v2, Nov 2015.
[23] TensorFlow. https://www.tensorflow.org/, last visited on Oct 2018.
[24] Keras: Deep Learning for humans. https://github.com/keras-team/keras, last visited on Oct 2018. |
| Description: | 碩士
國立政治大學
資訊科學系
105753018 |
| Source URI: | http://thesis.lib.nccu.edu.tw/record/#G0105753018 |
| Data Type: | thesis |
| DOI: | 10.6814/THE.NCCU.CS.003.2019.B02 |
| Appears in Collections: | [資訊科學系] 學位論文
|
Files in This Item:
| File |
Size | Format | |
|---|
| 301801.pdf | 4655Kb | Adobe PDF2 | 671 | View/Open |
|
All items in 政大典藏 are protected by copyright, with all rights reserved.
|
