政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/153388

政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/153388

English | 正體中文 | 简体中文 | Post-Print筆數 : 27 | Items with full text/Total items : 113392/144379 (79%)
Visitors : 51219693 Online Users : 907

RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.

Scope

please add "double quotation mark" for query phrases to get precise results

please goto advance search for comprehansive author search

Adv. Search

Home ‧ Login ‧ Upload ‧ Help ‧ About ‧ Administer

Goto mobile version

政大典藏 > College of Informatics > Department of Computer Science > Theses > Item 140.119/153388

Please use this identifier to cite or link to this item: https://nccur.lib.nccu.edu.tw/handle/140.119/153388

Title:	基於輕量化微調方法之進階檢索模型於改進文件檢索效能 Lightweight Fine-Tuning Dense Retrieval Models for Enhancing Document Retrieval Performance
Authors:	王奕凱 Wang, I-Kai
Contributors:	蔡銘峰王奕凱 Wang, I-Kai
Keywords:	資訊檢索大語言模型參數高效微調低佚適應 Information Retrieval LoRA LLM PEFT
Date:	2024
Issue Date:	2024-09-04 15:01:34 (UTC+8)
Abstract:	資訊檢索（IR）是一項從大規模文本集合中找到與用戶查詢相關資訊的任務。隨著大型語言模型（PLM）的發展達到了新的高度。密集檢索技術便是透過將查詢句和文本輸入大型語言模型，編碼成密集向量進行關聯度計算此項技術能處理語言多樣性和複雜性。大型預訓練語言模型的訓練資源需求高，因此參數高效率微調（PEFT）如適配器、 LoRA（低秩適應）等技術相繼提出，旨在減少微調參數量並保持性能。然而研究指出，此類方法在資訊檢索任務中效果有限，訓練參數過少會影響梯度下降方向，導致模型性能下降。本研究想利用LoRA的靈活性，在不增加額外訓練參數的情況下，以LoRA矩陣再加權文句的向量，增進訓練效果，設計一個更加通用的模型架構，並與其他較先進的LoRA技術結合，以應對PEFT方法在資訊檢索任務中的挑戰。 Information retrieval (IR) is the task of finding information related to user queries from large text collections. With the development of large pre- trained language models (PLMs) reaching new heights, dense retrieval tech- niques have emerged. These techniques involve encoding query sentences and texts into dense vectors using large language models to calculate rel- evance scores. This approach effectively handles linguistic diversity and complexity. However, training large pre-trained language models requires substantial resources. Consequently, parameter-efficient fine-tuning (PEFT) techniques, such as adapters and LoRA (Low-Rank Adaptation), have been proposed to reduce the number of fine-tuning parameters while maintaining performance. Nonetheless, studies indicate that these methods have limited effectiveness in IR tasks, as too few training parameters can affect the direc- tion of gradient descent, leading to degraded model performance. This study aims to leverage the flexibility of LoRA to enhance training effectiveness without increasing additional training parameters. By re-weighting sentence vectors with LoRA matrices, we design a more versatile model architecture. This architecture will be combined with other advanced LoRA techniques to address the challenges of PEFT methods in IR tasks.
Reference:	A. Aghajanyan, L. Zettlemoyer, and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020. [2] V. Boteva, D. Gholipour, A. Sokolov, and S. Riezler. A full-text learning to rank dataset for medical information retrieval. In N. Ferro, F. Crestani, M.-F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff, and G. Silvello, editors, Advances in Information Retrieval, pages 716–722, Cham, 2016. Springer International Pub- lishing. [3] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation. arXiv preprint arXiv:2402.03216, 2024. [4] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient fine- tuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024. [5] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019. [6] S. Hayou, N. Ghosh, and B. Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024. [7] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Ges- mundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. [8] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. [9] N. Hyeon-Woo, M. Ye-Bin, and T.-H. Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning, 2023. 35 [10] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [11] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano. Vera: Vector-based random matrix adaptation, 2024. [12] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Ep- stein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019. [13] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K¨uttler, M. Lewis, W.-t. Yih, T. Rockt¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Sys- tems, 33:9459–9474, 2020. [14] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021. [15] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023. [16] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024. [17] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too. AI Open, 2023. [18] K. Lu, A. Grover, P. Abbeel, and I. Mordatch. Pretrained transformers as universal computation engines, 2021. [19] X. Ma, J. Guo, R. Zhang, Y. Fan, and X. Cheng. Scattered or connected? an opti- mized parameter-efficient tuning approach for information retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Manage- ment, pages 1471–1480, 2022. [20] M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur. Www’18 open challenge: Financial opinion mining and question an- swering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee. 36 [21] N. Muennighoff, H. Su, L. Wang, N. Yang, F. Wei, T. Yu, A. Singh, and D. Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906, 2024. [22] J. Pfeiffer, A. Kamath, A. R¨uckl´e, K. Cho, and I. Gurevych. Adapterfusion: Non- destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020. [23] S.-A. Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017. [24] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. [25] S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gatford, et al. Okapi at trec-3. Nist Special Publication Sp, 109:109, 1995. [26] T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks, 2016. [27] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. [28] N. Thakur, N. Reimers, J. Daxenberger, and I. Gurevych. Augmened sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. arXiv preprint arXiv:2010.08240, 2020. [29] N. Thakur, N. Reimers, A. R¨uckl´e, A. Srivastava, and I. Gurevych. Beir: A heteroge- nous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021. [30] D. Wadden, S. Lin, K. Lo, L. L. Wang, M. van Zuylen, A. Cohan, and H. Hajishirzi. Fact or fiction: Verifying scientific claims. arXiv preprint arXiv:2004.14974, 2020. [31] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022. [32] A. Waswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, 2017. [33] L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148, 2023. [34] S.-Y. Yeh, Y.-G. Hsieh, Z. Gao, B. B. W. Yang, G. Oh, and Y. Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation, 2024. [35] Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
Description:	碩士國立政治大學資訊科學系 111753169
Source URI:	http://thesis.lib.nccu.edu.tw/record/#G0111753169
Data Type:	thesis
Appears in Collections:	[Department of Computer Science ] Theses

Files in This Item:

File	Description	Size	Format
316901.pdf		1890Kb	Adobe PDF	0	View/Open

All items in 政大典藏 are protected by copyright, with all rights reserved.

社群 sharing

著作權政策宣告 Copyright Announcement

1.本網站之數位內容為國立政治大學所收錄之機構典藏，無償提供學術研究與公眾教育等公益性使用，惟仍請適度，合理使用本網站之內容，以尊重著作權人之權益。商業上之利用，則請先取得著作權人之授權。
The digital content of this website is part of National Chengchi University Institutional Repository. It provides free access to academic research and public education for non-commercial use. Please utilize it in a proper and reasonable manner and respect the rights of copyright owners. For commercial use, please obtain authorization from the copyright owner in advance.

2.本網站之製作，已盡力防止侵害著作權人之權益，如仍發現本網站之數位內容有侵害著作權人權益情事者，請權利人通知本網站維護人員(nccur@nccu.edu.tw)，維護人員將立即採取移除該數位著作等補救措施。
NCCU Institutional Repository is made to protect the interests of copyright owners. If you believe that any material on the website infringes copyright, please contact our staff(nccur@nccu.edu.tw). We will remove the work from the repository and investigate your claim.

DSpace Software Copyright © 2002-2004 MIT & Hewlett-Packard / Enhanced by NTU Library IR team Copyright © - Feedback