政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/83535

政大機構典藏-National Chengchi University Institutional Repository(NCCUR):Item 140.119/83535

English | 正體中文 | 简体中文 | Post-Print筆數 : 27 | 全文笔数/总笔数 : 113160/144130 (79%)
造访人次 : 50761387 在线人数 : 708

RC Version 6.0 © Powered By DSPACE, MIT. Enhanced by NTU Library IR team.

搜寻范围

查询小技巧：

您可在西文检索词汇前后加上"双引号"，以获取较精准的检索结果

若欲以作者姓名搜寻，建议至进阶搜寻限定作者字段，可获得较完整数据

进阶搜寻

主页 ‧ 登入 ‧ 上传 ‧ 说明 ‧ 关于政大典藏 ‧ 管理

到手机版

政大機構典藏 > 商學院 > 資訊管理學系 > 學位論文 > Item 140.119/83535

请使用永久网址来引用或连结此文件: https://nccur.lib.nccu.edu.tw/handle/140.119/83535

题名:	應用記憶體內運算於多維度多顆粒度資料探勘之研究―以醫療服務創新為例 A Research Into In-memory Computing In Multidimensional, Multi-granularity Data Mining ― With Healthcare Services Innovation
作者:	朱家棋 Chu, Chia Chi
贡献者:	姜國輝季延平 Chiang, Kuo Huie Chi, Yan Ping 朱家棋 Chu, Chia Chi
关键词:	資料探勘多維度關聯分析記憶體內運算創新醫療服務應用 Data Mining Multidimensional and Multi-granularity Data Mining Association Analysis In-Memory Computing Apriori Algorithm
日期:	2015
上传时间:	2016-04-01 10:40:25 (UTC+8)
摘要:	全球面臨人口老化與人口不斷成長的壓力下，對於醫療服務的需求不斷提升。醫療服務領域中常以資料探勘「關聯規則」分析，挖掘隱藏在龐大的醫學資料庫中的知識(knowledge)，以支援臨床決策或創新醫療服務。隨著醫療服務與應用推陳出新(如，電子健康紀錄或行動醫療等)，與醫療機構因應政府政策需長期保存大量病患資料，讓醫療領域面臨如何有效的處理巨量資料。然而傳統的關聯規則演算法，其效能上受到相當大的限制。因此，許多研究提出將關聯規則演算法，在分散式環境中，以Hadoop MapReduce框架實現平行化處理巨量資料運算。其相較於單節點 (single-node) 的運算速度確實有大幅提升。但實際上，MapReduce並不適用於需要密集迭帶運算的關聯規則演算法。本研究藉由Spark記憶體內運算框架，在分散式叢集上實現平行化挖掘多維度多顆粒度挖掘關聯規則，實驗結果可以歸納出下列三點。第一點，當資料規模小時，由於平行化將資料流程分為Map與Reduce處理，因此在小規模資料處理上沒有太大的效益。第二點，當資料規模大時，平行化策略模式與單機版有明顯大幅度差異，整體運行時間相差100倍之多；然而當項目個數大於1萬個時，單機版因記憶體不足而無法運行，但平行化策略依舊可以運行。第三點，整體而言Spark雖然在小規模處理上略慢於單機版的速度，但其運行時間仍小於Hadoop的4倍。大規模處理速度上Spark依舊優於Hadoop版本。因此，在處理大規模資料時，就運算效能與擴充彈性而言，Spark都為最佳化解決方案。 Under the population aging and population growth and rising demand for Healthcare. Healthcare is facing a big issue how to effectively deal with huge amounts of data. Cased by new healthcare services or applications (such as electronic health records or health care, etc), and also medical institutions in accordance with government policy for long-term preservation of a large number of patient data. But the traditional algorithms for mining association rules, subject to considerable restrictions on their effectiveness. Therefore, many studies suggest that the association rules algorithm in a distributed computing, such as Hadoop MapReduce framework implements parallel to process huge amounts of data operations. But in fact, MapReduce does not apply to require intensive iterative computation algorithm of association rules. Studied in this Spark in-memory computing framework, implemented on a distributed cluster parallel mining association rules mining multidimensional granularity, the experimental results can be summed up in the following three points. 1th, when data is small, due to the parallel data flow consists of Map and Reduce, so not much in the small-scale processing of benefits. 2nd, when the data size is large, parallel strategy models and stand-alone obviously significant differences overall running time is 100 times as much when the item number is greater than 10,000, however, stand-alone version cannot run due to insufficient memory, but parallel strategies can still run. 3rd, overall Spark though somewhat slower than the single version in small scale processing speed, but the running time is less than 4 times times the Hadoop. Massive processing speed Spark is still superior to the Hadoop version. Therefore, when working with large data, operational efficiency and expansion elasticity, Spark for optimum solutions.
參考文獻:	[1] Agrawal, Rakesh, and Ramakrishnan Srikant. "Fast algorithms for mining association rules." Proc. 20th int. conf. very large data bases, VLDB. Vol. 1215. 1994. [2] Agrawal, Rakesh, Tomasz Imieliński, and Arun Swami. "Mining association rules between sets of items in large databases." ACM SIGMOD Record. Vol. 22. No. 2. ACM, 1993. [3] Apache Hadoop, https://hadoop.apache.org/,2015 [4] Chae, Young Moon, et al. "Data mining approach to policy analysis in a health insurance domain." International journal of medical informatics 62.2 (2001): 103-111. [5] Cheung, David W., et al. "A fast distributed algorithm for mining association rules." Parallel and Distributed Information Systems, 1996., Fourth International Conference on. IEEE, 1996. [6] Chiang, Johannes, and Chia-Chi Wu. "Mining multi-dimensional association rules in multiple database segmentation." Proceedings of International Conference on Information Management. 2005. [7] Chuang, Kun-Ta, Ming-Syan Chen, and Wen-Chieh Yang. "Progressive sampling for association rules based on sampling error estimation." Advances in Knowledge Discovery and Data Mining. Springer Berlin Heidelberg, 2005. 505-515. [8] Das, Amitabha, Wee-Keong Ng, and Yew-Kwong Woon. "Rapid association rule mining." Proceedings of the tenth international conference on Information and knowledge management. ACM, 2001. [9] Dean, Jeffrey, and Sanjay Ghemawat. "MapReduce: simplified data processing on large clusters." Communications of the ACM 51.1 (2008): 107-113. [10] Do, Tien Dung, Siu Cheung Hui, and Alvis Fong. "Mining frequent itemsets with category-based constraints." Discovery Science. Springer Berlin Heidelberg, 2003. [11] Ghemawat, Sanjay, Howard Gobioff, and Shun-Tak Leung. "The Google file system." ACM SIGOPS operating systems review. Vol. 37. No. 5. ACM, 2003. [12] Han, Jiawei, Jian Pei, and Yiwen Yin. "Mining frequent patterns without candidate generation." ACM SIGMOD Record. Vol. 29. No. 2. ACM, 2000. [13] Khare, Neelu, Neeru Adlakha, and K. R. Pardasani. "An Algorithm for Mining Multidimensional Association Rules using Boolean Matrix." Recent Trends in Information, Telecommunication and Computing (ITC), 2010 International Conference on. IEEE, 2010. [14] Lent, Brian, Arun Swami, and Jennifer Widom. "Clustering association rules." Data Engineering, 1997. Proceedings. 13th International Conference on. IEEE, 1997. [15] Li, Lingjuan, and Min Zhang. "The strategy of mining association rule based on cloud computing." Business Computing and Global Informatization (BCGIN), 2011 International Conference on. IEEE, 2011. [16] Li, Ning, et al. "Parallel implementation of apriori algorithm based on MapReduce." Software Engineering, Artificial Intelligence, Networking and Parallel & Distributed Computing (SNPD), 2012 13th ACIS International Conference on. IEEE, 2012. [17] Lin, Ming-Yen, Pei-Yu Lee, and Sue-Chen Hsueh. "Apriori-based frequent itemset mining algorithms on MapReduce." Proceedings of the 6th international conference on ubiquitous information management and communication. ACM, 2012. [18] Qiu, Hongjian, et al. "YAFIM: A Parallel Frequent Itemset Mining Algorithm with Spark." Parallel & Distributed Processing Symposium Workshops (IPDPSW), 2014 IEEE International. IEEE, 2014. [19] Schuster, Assaf, and Ran Wolff. "Communication-efficient distributed mining of association rules." Data Mining and Knowledge Discovery 8.2 (2004): 171-196. [20] Srikant, Ramakrishnan, and Rakesh Agrawal. "Mining quantitative association rules in large relational tables." ACM SIGMOD Record. Vol. 25. No. 2. ACM, 1996. [21] Wojciechowski, Marek, and Maciej Zakrzewicz. "Dataset filtering techniques in constraint-based frequent pattern mining." Pattern detection and discovery. Springer Berlin Heidelberg, 2002. 77-91. [22] Yang, Xin Yue, Zhen Liu, and Yan Fu. "MapReduce as a programming model for association rules algorithm on Hadoop." Information Sciences and Interaction Sciences (ICIS), 2010 3rd International Conference on. IEEE, 2010. [23] Zaharia, Matei, et al. "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing." Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association, 2012. [24] “Frequent Itemset Mining Dataset Repository”, http://fimi.ua.ac.be/data/
描述:	碩士國立政治大學資訊管理學系 102356020
資料來源:	http://thesis.lib.nccu.edu.tw/record/#G1023560202
数据类型:	thesis
显示于类别:	[資訊管理學系] 學位論文

文件中的档案:

档案	描述	大小	格式	浏览次数
020201.pdf		6432Kb	Adobe PDF2	127	检视/开启

在政大典藏中所有的数据项都受到原著作权保护.

社群 sharing

著作權政策宣告 Copyright Announcement

1.本網站之數位內容為國立政治大學所收錄之機構典藏，無償提供學術研究與公眾教育等公益性使用，惟仍請適度，合理使用本網站之內容，以尊重著作權人之權益。商業上之利用，則請先取得著作權人之授權。
The digital content of this website is part of National Chengchi University Institutional Repository. It provides free access to academic research and public education for non-commercial use. Please utilize it in a proper and reasonable manner and respect the rights of copyright owners. For commercial use, please obtain authorization from the copyright owner in advance.

2.本網站之製作，已盡力防止侵害著作權人之權益，如仍發現本網站之數位內容有侵害著作權人權益情事者，請權利人通知本網站維護人員(nccur@nccu.edu.tw)，維護人員將立即採取移除該數位著作等補救措施。
NCCU Institutional Repository is made to protect the interests of copyright owners. If you believe that any material on the website infringes copyright, please contact our staff(nccur@nccu.edu.tw). We will remove the work from the repository and investigate your claim.

DSpace Software Copyright © 2002-2004 MIT & Hewlett-Packard / Enhanced by NTU Library IR team Copyright © - 回馈