政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/146894
English  |  正體中文  |  简体中文  |  Post-Print筆數 : 27 |  全文筆數/總筆數 : 113451/144438 (79%)
造訪人次 : 51339949      線上人數 : 828
RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.
搜尋範圍 查詢小技巧：
  • 您可在西文檢索詞彙前後加上"雙引號"，以獲取較精準的檢索結果
  • 若欲以作者姓名搜尋，建議至進階搜尋限定作者欄位，可獲得較完整資料
    •  進階搜尋
    政大機構典藏 > 商學院 > 資訊管理學系 > 學位論文 >  Item 140.119/146894


    請使用永久網址來引用或連結此文件: https://nccur.lib.nccu.edu.tw/handle/140.119/146894


    題名: 多模態融合用於欺騙偵測
    Multimodal Fusion for Deception Detection
    作者: 孫振原
    Sun, Cheng-Yuan
    貢獻者: 蕭舜文
    Hsiao, Shun-Wen
    孫振原
    Sun, Cheng-Yuan
    關鍵詞: 多模態
    欺騙偵測
    注意力機制
    集成學習
    模型校正
    Multimodal
    Deception detection
    Attention mechanism
    Ensemble
    Calibration
    日期: 2023
    上傳時間: 2023-09-01 14:54:51 (UTC+8)
    摘要: 現今，各種影片類型的資料如雨後春筍般冒出。在以人為中心的影片分析領域中，欺騙偵測已成為我們面臨的重要議題。儘管基於人工智慧的欺騙偵測模型已取得了驚人的準確度，但它們往往缺乏可解釋性並作為一個「黑箱」運作。在本研究中，我們提出了一種基於注意力機制的神經網絡來進行欺騙偵測，解決可變長度的影片資料和欺騙過程所帶來的的挑戰。我們的模型不斷評估視覺、音頻和文本訊息，精確地找出顯露欺騙跡象的時刻。此外，我們採用了多模態方法，在具有不同特徵的多個模型之後加入了融合機制，使協同推理在欺騙偵測上能夠實現更為精確及全面。在真實審判資料集上，我們的模型達到了92％的準確度，多模態融合的表現優於單模態方法。該模型可同時輸出每一時刻的注意力權重，從而為包含潛在欺騙線索的特定時間間隔提供有價值的見解。因此，我們的方法不僅在檢測欺騙方面表現出色，還能讓人了解欺騙過程中的動態變化，使其成為多種情境中的有力工具。此外，我們還進行了一項實驗，請學校學生對有關個人事務的問題做出誠實或欺騙性的回答。資料集由309個影片片段組成，誠實和欺騙性回答各佔一半，並使用自動語音識別建立詳細文字記錄。我們還設計了一個類似於LoRA的模型，針對個體校準模型，從而提高對未見個體的準確性。
    Nowadays, various video data are springing up. In the field of human-centric video analysis, deception detection becomes a crucial issue to us. Although AI-based deception detection models have achieved remarkable accuracy, they often lack interpretability and work as a "black box." In this study, we propose an attention-aware neural network for deception detection, addressing the challenges of variable-length video data and the process of deception. Our model continuously assesses visual, audio, and textual information, pinpointing moments revealing signs of deception. Additionally, we embrace a multimodal approach, incorporating a fusion mechanism following multiple models with distinct features, enabling collaborative inference for more accurate and comprehensive deception detection. Our model achieves 92\\% accuracy on a real-life trials dataset, with the multimodal ensemble outperforming unimodal approaches. The model simultaneously outputs attention weights for each moment, providing valuable insights into specific time intervals containing potential deception cues. Thus, our approach not only excels at detecting deception but also offers an understanding of the dynamics underlying deception, making it a powerful tool in a variety of contexts. Moreover, we conducted an experiment involving university students who were asked to respond truthfully or deceptively to questions about personal matters. The dataset consists of 309 video clips, equally divided between truthful and deceptive responses, supplemented with detailed transcripts created through automatic speech recognition. We also design a LoRA-like model to calibrate the model by individuals, improving accuracy for unseen individuals.
    參考文獻: Avola, D., Cinque, L., De Marsico, M., Fagioli, A., and Foresti, G. L. (2020). Lietome: Preliminary study on hand gestures for deception detection via fisher-lstm. Pattern Recognition Letters, 138:455–461.
    Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
    Baltrušaitis, T., Ahuja, C., and Morency, L.-P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443.
    Baltrusaitis, T., Zadeh, A., Lim, Y. C., and Morency, L.-P. (2018). Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 59–66. IEEE.
    Bhamare, A. R., Katharguppe, S., and Nancy, J. S. (2020). Deep neural networks for lie detection with attention on bio-signals. In 2020 7th International Conference on Soft Computing & Machine Intelligence (ISCMI), pages 143–147. IEEE.
    Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.
    Deng, J., Guo, J., Ververas, E., Kotsia, I., and Zafeiriou, S. (2020). Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5203–5212.
    Ding, M., Zhao, A., Lu, Z., Xiang, T., and Wen, J.-R. (2019). Face-focused cross-stream network for deception detection in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7802–7811.
    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
    Gogate, M., Adeel, A., and Hussain, A. (2017). Deep learning driven multimodal fusion for automated deception detection. In 2017 IEEE symposium series on computational intelligence (SSCI), pages 1–6. IEEE.
    Gong, Y., Chung, Y.-A., and Glass, J. (2021). Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778.
    He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
    Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J., et al. (2001). Gradient flow in recurrent nets: the difficulty of learning long-term dependencies.
    Hsiao, S.-W. and Sun, C.-Y. (2022). Attention-aware multi-modal rnn for deception detection. In 2022 IEEE International Conference on Big Data (Big Data), pages 3593–3596. IEEE.
    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
    Karimi, H., Tang, J., and Li, Y. (2018). Toward end-to-end deception detection in videos. In 2018 IEEE International Conference on Big Data (Big Data), pages 1278–1283. IEEE.
    Kollias, D., Tzirakis, P., Nicolaou, M. A., Papaioannou, A., Zhao, G., Schuller, B., Kotsia, I., and Zafeiriou, S. (2019). Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. International Journal of Computer Vision, 127(6-7):907–929.
    Krishnamurthy, G., Majumder, N., Poria, S., and Cambria, E. (2023). A deep learn- ing approach for multimodal deception detection. In 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing), pages 87–96. Springer.
    Mathur, L. and Matarić, M. J. (2020). Introducing representations of facial affect in automated multimodal deception detection. In Proceedings of the 2020 International Conference on Multimodal Interaction, pages 305–314.
    Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., and Sun, C. (2021). Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34:14200–14213.
    Pérez-Rosas, V., Abouelenien, M., Mihalcea, R., and Burzo, M. (2015). Deception detection using real-life trial data. In Proceedings of the 2015 ACM on international conference on multimodal interaction, pages 59–66.
    Schroff, F., Kalenichenko, D., and Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823.
    Serengil, S. I. and Ozpinar, A. (2020). Lightface: A hybrid deep face recognition framework. In 2020 Innovations in Intelligent Systems and Applications Conference (ASYU), pages 23–27. IEEE.
    Serengil, S. I. and Ozpinar, A. (2021). Hyperextended lightface: A facial attribute analysis framework. In 2021 International Conference on Engineering and Emerging Technologies (ICEET), pages 1–4. IEEE.
    Serengil, S. I. and Ozpinar, A. (2023). An evaluation of sql and nosql databases for facial recognition pipelines.
    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
    Venkatesh, S., Ramachandra, R., and Bours, P. (2019). Robust algorithm for multimodal deception detection. In 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pages 534–537. IEEE.
    Weinberger, K. Q. and Saul, L. K. (2009). Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research, 10(2).
    Wu, Y., Lin, Z., Zhao, Y., Qin, B., and Zhu, L.-N. (2021). A text-centered shared-private framework via cross-modal prediction for multimodal sentiment analysis. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4730–4738.
    Wu, Z., Singh, B., Davis, L., and Subrahmanian, V. (2018). Deception detection in videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
    Xu, N., Mao, W., and Chen, G. (2019). Multi-interactive memory network for aspect based multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 371–378.
    Zhang, K., Zhang, Z., Li, Z., and Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499–1503.
    描述: 碩士
    國立政治大學
    資訊管理學系
    110356041
    資料來源: http://thesis.lib.nccu.edu.tw/record/#G0110356041
    資料類型: thesis
    顯示於類別:[資訊管理學系] 學位論文

    文件中的檔案:

    檔案 描述 大小格式瀏覽次數
    604101.pdf2069KbAdobe PDF2126檢視/開啟


    在政大典藏中所有的資料項目都受到原著作權保護.


    社群 sharing

    著作權政策宣告 Copyright Announcement
    1.本網站之數位內容為國立政治大學所收錄之機構典藏，無償提供學術研究與公眾教育等公益性使用，惟仍請適度，合理使用本網站之內容，以尊重著作權人之權益。商業上之利用，則請先取得著作權人之授權。
    The digital content of this website is part of National Chengchi University Institutional Repository. It provides free access to academic research and public education for non-commercial use. Please utilize it in a proper and reasonable manner and respect the rights of copyright owners. For commercial use, please obtain authorization from the copyright owner in advance.

    2.本網站之製作，已盡力防止侵害著作權人之權益，如仍發現本網站之數位內容有侵害著作權人權益情事者，請權利人通知本網站維護人員(nccur@nccu.edu.tw)，維護人員將立即採取移除該數位著作等補救措施。
    NCCU Institutional Repository is made to protect the interests of copyright owners. If you believe that any material on the website infringes copyright, please contact our staff(nccur@nccu.edu.tw). We will remove the work from the repository and investigate your claim.
    DSpace Software Copyright © 2002-2004  MIT &  Hewlett-Packard  /   Enhanced by   NTU Library IR team Copyright ©   - 回饋