Loading...
|
Please use this identifier to cite or link to this item:
https://nccur.lib.nccu.edu.tw/handle/140.119/107015
|
| Title: | 詞彙向量的理論與評估基於矩陣分解與神經網絡
Theory and evaluation of word embedding based on matrix factorization and neural network |
| Authors: | 張文嘉
Jhang, Wun Jia |
| Contributors: | 翁久幸
馬偉雲
Weng, Chiu Hsing
Ma, Wei Yun
張文嘉
Jhang, Wun Jia |
| Keywords: | 矩陣分解
初始值
自然語言處理
神經網絡
Matrix factorization
Initalization
Natural language processing
Neural network |
| Date: | 2017 |
| Issue Date: | 2017-03-02 11:10:02 (UTC+8) |
| Abstract: | 隨著機器學習在越來越多任務中有突破性的發展，特別是在自然語言處理問題上，得到越來越多的關注，近年來，詞向量是自然語言處理研究中最令人興奮的部分之一。在這篇論文中，我們討論了兩種主要的詞向量學習方法。一種是傳統的矩陣分解，如奇異值分解，另一種是基於神經網絡模型（具有負採樣的Skip-gram模型（Mikolov等人提出，2013），它是一種迭代演算法。我們提出一種方法來挑選初始值，透過使用奇異值分解得到的詞向量當作是Skip-gram模型的初始直，結果發現替換較佳的初始值，在某些自然語言處理的任務中得到明顯的提升。
Recently, word embedding is one of the most exciting part of research in natural language processing. In this thesis, we discuss the two major learning approaches for word embedding. One is traditional matrix factorization like singular value decomposition, the other is based on neural network model (e.g. the Skip-gram model with negative sampling (Mikolov et al., 2013b)) which is an iterative algorithm. It is known that an iterative process is sensitive to initial starting values. We present an approach for implementing the Skip-gram model with negative sampling from a given initial value that is using singular value decomposition. Furthermore, we show that refined initial starting points improve the analogy task and succeed in capturing fine-gained semantic and syntactic regularities using vector arithmetic. |
| Reference: | Marco Baroni, Georgiana Dinu, and German Kruszewski. Don`t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1), pages 238-247, 2014.
Yoshua Bengio. Learning deep architectures for ai. Foundations and trends R in Machine Learning, 2(1):1-127, 2009.
Christophe Biernacki, Gilles Celeux, and Gerard Govaert. Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models. Computational Statistics & Data Analysis, 41(3):561-575, 2003.
Paul S Bradley and Usama M Fayyad. Refining initial points for k-means clustering. In ICML, volume 98, pages 91-99. Citeseer, 1998.
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. Distributional semantics in technicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 136-145. Association for Computational Linguistics, 2012.
John Caron. Experiments with lsa scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157-169, 2001.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391, 1990.
Richard O Duda, Peter E Hart, et al. Pattern classification and scene analysis, volume 3. Wiley New York, 1973.
Yoav Goldberg and Omer Levy. word2vec explained: deriving mikolov et al.`s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722, 2014.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 2016.
Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL (2), pages 302-308, 2014a.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization.In Advances in Neural Information Processing Systems, pages 2177-2185, 2014b.
Omer Levy, Yoav Goldberg, and Israel Ramat-Gan. Linguistic regularities in sparse and explicit word representations. In CoNLL, pages 171-180, 2014.
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211-225, 2015.
Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In CoNLL, pages 104-113, 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schutze. Evaluation in information retrieval. Introduction to information retrieval, pages 151-175, 2008.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ecient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013b.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, volume 13, pages 746-751, 2013c.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.
Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pages 337-346. ACM, 2011.
Xin Rong. word2vec parameter learning explained. arXiv preprint arXiv:1411.2738, 2014.
Gerard Salton and Michael J McGill. Introduction to modern information retrieval. 1986.
Fabrizio Sebastiani. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1-47, 2002.
Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. Parsing with compositional vector grammars. In ACL (1), pages 455-465, 2013a.
Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1631, page 1642. Citeseer, 2013b.
Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 41-47. ACM, 2003.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384-394. Association for Computational Linguistics, 2010.
Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. Bilingual word embeddings for phrase-based machine translation. In EMNLP, pages 1393-1398, 2013. |
| Description: | 碩士
國立政治大學
統計學系
103354027 |
| Source URI: | http://thesis.lib.nccu.edu.tw/record/#G0103354027 |
| Data Type: | thesis |
| Appears in Collections: | [統計學系] 學位論文
|
Files in This Item:
| File |
Size | Format | |
|---|
| 402701.pdf | 856Kb | Adobe PDF2 | 40 | View/Open |
|
All items in 政大典藏 are protected by copyright, with all rights reserved.
|
