政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/143834

政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/143834

English | 正體中文 | 简体中文 | Post-Print筆數 : 27 | 全文筆數/總筆數 : 114409/145439 (79%)
造訪人次 : 53234225 線上人數 : 564

RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.

搜尋範圍

查詢小技巧：

您可在西文檢索詞彙前後加上"雙引號"，以獲取較精準的檢索結果

若欲以作者姓名搜尋，建議至進階搜尋限定作者欄位，可獲得較完整資料

進階搜尋

主頁 ‧ 登入 ‧ 上傳 ‧ 說明 ‧ 關於政大典藏 ‧ 管理

到手機版

政大機構典藏 > 資訊學院 > 資訊科學系 > 學位論文 > Item 140.119/143834

請使用永久網址來引用或連結此文件: https://nccur.lib.nccu.edu.tw/handle/140.119/143834

題名:	深度強化學習歷程的視覺化分析以Procgen Benchmark環境為例 The Learning Process of Deep Reinforcement Learning with Visualization Analysis Based on Procgen Benchmark Environment
作者:	黃亦晨 HUANG, Yi-Chen
貢獻者:	紀明德 Chi, Ming-Te 黃亦晨 HUANG, Yi-Chen
關鍵詞:	深度強化學習視覺化 Procgen Benchmark Deep reinforcement learning Visualization Procgen Benchmark
日期:	2023
上傳時間:	2023-03-09 18:37:03 (UTC+8)
摘要:	強化學習結合了深度學習，也就是深度強化學習—透過傳送遊戲中的畫面給電腦，讓電腦選擇動作，再根據電腦的選擇給予獎勵或是懲罰。在 Procgen Benchmark 環境中，使用者可以自訂環境數量，能夠提供幾乎完全隨機的遊戲環境，降低發生擬合過度的情形，使模型的訓練更為完善。由於神經網路黑盒子的特性，我們只能透過模型的輸出判斷模型是否完善，但無法知道模型是透過什麼方式來做決策，因此需要視覺化的工具輔助觀察模型行為。首先我們呈現了模型相關資訊的圖表，再來使用了基於擾動生成的顯著圖來觀察模型關注區域，此外我們將神經網路中隱藏層的數據提取後使用 T-SNE 降維，並用 K-means 演算法做分群，最後將上述功能整合成一個視覺分析介面，幫助使用者理解模型決策過程，同時也能觀察模型的決策是否與我們預期中相符。透過使用者報告，受試者認為我們設計的獎勵機制優於對照組，驗證了模型良好的性能，此外，受試者也能透過視覺化分析介面觀察出電腦使用的通關策略以及人類玩家通關策略的差異，驗證了視覺化分析介面的有效性。 Deep reinforcement learning combines with deep learning and reinforcement learning. By sending game screens to the computer and allowing the computer to choose actions, the computer is then rewarded or punished based on its choices. Procgen Benchmark environment, in which users can customize the environment numbers, provides almost random game environments and reduces overfitting, making the model training more comprehensive. Due to the black box nature of neural networks, we can judge whether the model is well-performing by only its output, but we cannot know how the model makes decisions. Therefore, visual tools are needed to assist in observing the model behavior. First, we present charts with model-related information. Then, we use saliency maps generated based on perturbation to observe the areas of focus in the model. In addition, we extract data from the hidden layer of the neural network and use T-SNE for dimensionality reduction and K-means algorithm for clustering. Finally, we integrate the aforementioned functions into a visual analysis interface to help users understand the model`s decision-making process and to observe whether the model`s decisions meet up with our expectations. We verify the good performance of the model through user studies, which showed that the participants believe that our designed reward mechanism is better than the control group. Additionally, that participants can observe the differences between computer and human player strategies through the visual analysis interface validate the effectiveness of the visualization analysis interface.
參考文獻:	[1] Cheng, S., Li, X., Shan, G., Niu, B., Wang, Y., & Luo, M. (2022). ACMViz: a visual analytics approach to understand DRL-based autonomous control model. Journal of Visualization, 1-16. [2] Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2020, November). Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning (pp. 2048-2056). PMLR. [3] Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J. (2019, May). Quantifying generalization in reinforcement learning. In International Conference on Machine Learning (pp. 1282-1289). PMLR. [4] Deshpande, S., Eysenbach, B., & Schneider, J. (2020). Interactive visualization for debugging rl. arXiv preprint arXiv:2008.07331. [5] Greydanus, S., Koul, A., Dodge, J., & Fern, A. (2018, July). Visualizing and understanding atari agents. In International conference on machine learning (pp. 1792-1801). PMLR. [6] Hilton, J., Cammarata, N., Carter, S., Goh, G., & Olah, C. (2020). Understanding rl vision. Distill, 5(11), e29. [7] Joo, H. T., & Kim, K. J. (2019, August). Visualization of deep reinforcement learning using grad-CAM: how AI plays atari games?. In 2019 IEEE Conference on Games (CoG) (pp. 1-2). IEEE. [8] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Hassabis, D. (2015). Human-level control through deep reinforcement learning. nature, 518(7540), 529-533. [9] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. [10] Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., & Jimenez Rezende, D. (2019). Towards interpretable reinforcement learning using attention augmented agents. Advances in neural information processing systems, 32. [11] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [12] Van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of machine learning research, 9(11). [13] Wang, J., Zhang, W., Yang, H., Yeh, C. C. M., & Wang, L. (2021). Visual analytics for rnn-based deep reinforcement learning. IEEE Transactions on Visualization and Computer Graphics, 28(12), 4141-4155. [14] Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 (pp. 818-833). Springer International Publishing.
描述:	碩士國立政治大學資訊科學系 109753107
資料來源:	http://thesis.lib.nccu.edu.tw/record/#G0109753107
資料類型:	thesis
顯示於類別:	[資訊科學系] 學位論文

文件中的檔案:

檔案	描述	大小	格式	瀏覽次數
310701.pdf		14075Kb	Adobe PDF2	0	檢視/開啟

在政大典藏中所有的資料項目都受到原著作權保護.

社群 sharing

著作權政策宣告 Copyright Announcement

1.本網站之數位內容為國立政治大學所收錄之機構典藏，無償提供學術研究與公眾教育等公益性使用，惟仍請適度，合理使用本網站之內容，以尊重著作權人之權益。商業上之利用，則請先取得著作權人之授權。
The digital content of this website is part of National Chengchi University Institutional Repository. It provides free access to academic research and public education for non-commercial use. Please utilize it in a proper and reasonable manner and respect the rights of copyright owners. For commercial use, please obtain authorization from the copyright owner in advance.

2.本網站之製作，已盡力防止侵害著作權人之權益，如仍發現本網站之數位內容有侵害著作權人權益情事者，請權利人通知本網站維護人員(nccur@nccu.edu.tw)，維護人員將立即採取移除該數位著作等補救措施。
NCCU Institutional Repository is made to protect the interests of copyright owners. If you believe that any material on the website infringes copyright, please contact our staff(nccur@nccu.edu.tw). We will remove the work from the repository and investigate your claim.

DSpace Software Copyright © 2002-2004 MIT & Hewlett-Packard / Enhanced by NTU Library IR team Copyright © - 回饋