政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/122287

政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/122287

English | 正體中文 | 简体中文 | Post-Print筆數 : 27 | Items with full text/Total items : 113318/144297 (79%)
Visitors : 50968020 Online Users : 936

RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.

Scope

please add "double quotation mark" for query phrases to get precise results

please goto advance search for comprehansive author search

Adv. Search

Home ‧ Login ‧ Upload ‧ Help ‧ About ‧ Administer

Goto mobile version

政大典藏 > College of Informatics > Department of Computer Science > Theses > Item 140.119/122287

Please use this identifier to cite or link to this item: https://nccur.lib.nccu.edu.tw/handle/140.119/122287

Title:	基於卷積核冗餘的神經網路壓縮機制 Compression of Convolutional Neural Networks Based on Kernel Redundancy
Authors:	陳乃瑋 Chen, Nai-Wei
Contributors:	廖文宏 Liao, Wen-Hung 陳乃瑋 Chen,Nai-Wei
Keywords:	卷積神經網路模型壓縮多樣性演算法卷積核相似度 Convolutional neural network Model compression Diversifying algorithm Kernel similarity
Date:	2018
Issue Date:	2019-02-12 15:47:36 (UTC+8)
Abstract:	針對卷積神經網路模型與所需計算量過於龐大，導致難以將之部署於行動裝置或嵌入式平台的問題，本論文提出了多樣性演算法用以對CNN模型做壓縮。多樣性演算法的核心概念為保留各個卷積層中最具代表性的濾波器，僅維持卷積核之間的多樣性。我們將卷積層內的濾波器的相似關係以無向圖來表示，每一個節點皆代表一個濾波器，節點與節點之間邊的值則是兩個濾波器之間的餘弦距離，而哪些節點會被移除則是透過邊的值、該節點邊的值加總以及該節點所代表濾波器的權值絕對值加總來決定。經過剪枝後的模型，每一層卷積層的輸出通道數都會有所減少，參數量與浮點數運算量也會相應的降低，且不會產生稀疏網路的問題，而壓縮後的模型最後會再重新訓練使辨識準確率回復至原先的水準。為了測試多樣性演算法的泛用性，我們使用VGG、ResNet、DenseNet系列模型與CIFAR-10、CIFAR-100資料集進行了十組的壓縮實驗。為了找到最佳的壓縮參數設置，我們在每組實驗上都做了非常充分的壓縮測試，並詳細記錄所有的實驗結果做為調整參數的重要依據，從各組實驗的壓縮結果顯示，多樣性演算法確實是有效的。在CIFAR-10資料集上，我們的方法可以減少VGG16 78.6%的參數量與約46%的浮點數運算量，而在容許約1%準確率差異的情況下，則可以減少90.7%的參數量與接近70%的浮點數運算量。在CIFAR-100資料集上，可以減少VGG16 46%的參數量與18%的浮點數運算量，而在容許約1%準確率差異的情況下，則可以減少約60.7%的參數量與接近37.5%的浮點數運算量。 The model size and floating-point operations (FLOP) required by convolutional neural networks make it difficult to deploy these models to mobile devices or embedded systems. In this thesis, we propose a method known as the diversifying algorithm to compress CNN models. The key concept is to maintain the diversity of convolutional kernels by preserving the most representative filters in each network layer. This is achieved by expressing the network architecture as an undirected graph. The nodes in the graph denote the filters, and the weights are computed using cosine distance. Nodes are removed by considering the combined effects of several factors, including edge weights, the sum of similarity and the sum of weights. After pruning, the number of output channels in each convolutional layer will be reduced. The compressed network is then retrained to retain accuracy with fewer model parameters, FLOPs and also avoid the problem of sparse representation. We test the efficacy of the proposed diversifying algorithm on three types of CNN models, including VGG, ResNet and DenseNet using both CIFAR-10 and CIFAR-100 datasets through extensive experiments. On CIFAR-10 dataset, our method is able to reduce 78.6% of total parameters and nearly 46% FLOPs in VGG16. If 1% performance loss is allowed, we can achieve 90.7% parameter and 70% FLOP reduction. On CIFAR-100 dataset, we can reduce 46% parameter and 18% FLOP. Furthermore, we can achieve 60.7% parameter and nearly 37.5% FLOP reduction if 1% accuracy loss is allowed.
Reference:	[1] Song Han, Huizi Mao, William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv:1510.00149v5, Feb 2016. [2] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf. Pruning Filters for Efficient ConvNets. arXiv:1608.08710v3, Mar 2017. [3] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. arXiv:1602.07360v4, Nov 2016. [4] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:1704.04861v1, Apr 2017. [5] Alex Krizhevsky, Vinod Nair, Geoffrey Hinton. CIFAR-10 and CIFAR-100 datasets. https://www.cs.toronto.edu/~kriz/cifar.html, last visited on Jan 2018. [6] 李宏毅, [DSC 2016]系列活動: 李宏毅/一天搞懂深度學習, https://www.slideshare.net/tw_dsconf/ss-62245351, last visited on Jan 2018. [7] Yann LeCun, Corinna Cortes, Christopher J.C. Burges. THE MNIST DATABASE of handwritten digits. http://yann.lecun.com/exdb/mnist/, last visited on Oct 2018 [8] ImageNet. http://www.image-net.org/, last visited on Jan 2018. [9] ImageNet Large Scale Visual Recognition Competition (ILSVRC). http://www.image-net.org/challenges/LSVRC/, last visited on Jan 2018. [10] Yuanqing Lin, Fengjun Lv, Shenghuo Zhu, Ming Yang, Timothee Cour, Kai Yu, Liangliang Cao, Thomas Huang. Large-scale image classification: Fast feature extraction and SVM training. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference, pages 1689-1696. IEEE, 2011. [11] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in neural information processing systems, pages 1097-1105, 2012. [12] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich. Going Deeper with Convolutions. arXiv:1409.4842v1, Sep 2014. [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep Residual Learning for Image Recognition. IEEE, pages 770-778, 2016. [14] Kaiming He. Learning Deep Features for Visual Recognition. http://deeplearning.csail.mit.edu/cvpr2017_tutorial_kaiminghe.pdf, last visited on Oct 2018. [15] Embedded Systems Developer Kits, Modules, & SDKs \| NVIDIA Jetson. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems-dev-kits-modules/, last visited on Oct 2018. [16] Raspberry Pi. https://www.raspberrypi.org/, last visited on Oct 2018. [17] Song Han, Jeff Pool, John Tran, William J. Dally. Learning both Weights and Connections for Efficient Neural Networks. arXiv:1506.02626v3, Oct 2015. [18] Babajide O. Ayinde, Jacek M. Zurada. Building Efficient ConvNets using Redundant Feature Pruning. arXiv:1802.07653v1, Feb 2018. [19] Karen Simonyan, Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556v6, Apr 2015. [20] Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger. Densely Connected Convolutional Networks. arXiv:1608.06993v5, Jan 2018. [21] Sergey Ioffe, Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167v3, Mar 2015. [22] Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber. Highway Networks. arXiv:1505.00387v2, Nov 2015. [23] TensorFlow. https://www.tensorflow.org/, last visited on Oct 2018. [24] Keras: Deep Learning for humans. https://github.com/keras-team/keras, last visited on Oct 2018.
Description:	碩士國立政治大學資訊科學系 105753018
Source URI:	http://thesis.lib.nccu.edu.tw/record/#G0105753018
Data Type:	thesis
DOI:	10.6814/THE.NCCU.CS.003.2019.B02
Appears in Collections:	[Department of Computer Science ] Theses

Files in This Item:

File	Size	Format
301801.pdf	4655Kb	Adobe PDF2	671	View/Open

All items in 政大典藏 are protected by copyright, with all rights reserved.

社群 sharing

著作權政策宣告 Copyright Announcement

1.本網站之數位內容為國立政治大學所收錄之機構典藏，無償提供學術研究與公眾教育等公益性使用，惟仍請適度，合理使用本網站之內容，以尊重著作權人之權益。商業上之利用，則請先取得著作權人之授權。
The digital content of this website is part of National Chengchi University Institutional Repository. It provides free access to academic research and public education for non-commercial use. Please utilize it in a proper and reasonable manner and respect the rights of copyright owners. For commercial use, please obtain authorization from the copyright owner in advance.

2.本網站之製作，已盡力防止侵害著作權人之權益，如仍發現本網站之數位內容有侵害著作權人權益情事者，請權利人通知本網站維護人員(nccur@nccu.edu.tw)，維護人員將立即採取移除該數位著作等補救措施。
NCCU Institutional Repository is made to protect the interests of copyright owners. If you believe that any material on the website infringes copyright, please contact our staff(nccur@nccu.edu.tw). We will remove the work from the repository and investigate your claim.

DSpace Software Copyright © 2002-2004 MIT & Hewlett-Packard / Enhanced by NTU Library IR team Copyright © - Feedback