A fast method for frequent pattern discovery with secondary memory

Article type: Research Article

Authors: Lin, Kawuu W.^{a; *} | Chung, Sheng-Hao^b | Chen, Ju-Chin^a | Huang, Sheng-Shiung^a | Lin, Chun-Cheng^b

Affiliations: [a] Department of Computer Science and Information Engineering, National Kaohsiung University of Applied Sciences, Kaohsiung, Taiwan | [b] Department of Industrial Engineering and Management, National Chiao Tung University, Hsinchu, Taiwan

Correspondence: [*] Corresponding author: Kawuu W. Lin, No. 415, Jiangong Rd., Sanmin Dist., Kaohsiung City 807, Taiwan. Tel.: +886 7 3814526 ext: 5806; Fax: +886 7 3837424; E-mail:[email protected]

Abstract: Data mining technology has been widely studied and applied in recent years. Frequent pattern mining is one important technical field of such research. The frequent pattern mining technique is popular not only in academia but also in the business community. With advances in technology, databases have become so large that data mining is impossible because of memory restrictions. In this study, we propose a novel algorithm for Fast mining with Secondary Memory, abbreviated as FSM-Mining, to help improve this situation. FSM-Mining saves a part of the information that is not stored in the memory, and through the use of mixed hard disk and memory mining we are able to complete data mining with limited memory. The results of empirical evaluation under various simulation conditions show that FSM-Mining delivers excellent performance in terms of execution efficiency and scalability.

Keywords: Data mining, frequent patterns, big data

DOI: 10.3233/IDA-170876

Journal: Intelligent Data Analysis, vol. 21, no. S1, pp. S159-S176, 2017