A framework for distributed nearest neighbor classification using Hadoop
Issue title: Selected papers from the ISCA International Conference on Software Engineering and Data Engineering, and the ISCA International Conference on Computer Applications in Industry and Engineering, 2015, and Invited Papers
Guest editors: Takaaki Goto and Narayan C. Debnath
Affiliations: [a] Department of Computer Science, East Carolina University, Greenville, NC 27858, USA | [b] Department of Computer Science, University of South Carolina, Columbia, SC 29208, USA
Correspondence:
[*]
Corresponding author: Qin Ding, Department of Computer Science, East Carolina University, Greenville, NC 27858, USA. Tel.: +1 252 328 9686; Fax: +1 252 328 0715; E-mail:[email protected]
Abstract: Within the field of data mining and machine learning, the K-Nearest Neighbor
algorithm is a classic algorithm which simply yet elegantly classifies data
based upon its similarity to other data. While it follows that the accuracy
increases as more data are provided, handling large sets of data is
difficult to process serially. It is therefore ideal to perform these tasks
in parallel or distributed mode. In this paper, we proposed a framework for
distributed nearest neighbor classification. A custom K-Nearest Neighbor
algorithm was developed using Hadoop, an environment for developing and
deploying applications in parallel on a cluster. The algorithm was
implemented on a cluster then tested for accuracy and time of execution. It
was observed that the accuracy depends on the provided k-value and on the
data set, which is to be expected for the K-Nearest Neighbor process. The
time of execution was found to increase logarithmically as the file size,
and thus the amount of data the algorithm must parse, increases
exponentially.
Keywords: Data mining, distributed data mining, classification, K-Nearest Neighbor, Hadoop
