Mining Outliers in Correlated Subspaces for High Dimensional Data Sets

Issue title: Intelligent Data Analysis in Granular Computing

Article type: Research Article

Authors: Leng, Jinsong | Hong, Tzung-Pei

Affiliations: School of Computer and Security Science, Edith Cowan University WA 6050, Australia. E-mail: [email protected] | Department of Computer Science and Information Engineering National University of Kaohsiung, Taiwan. E-mail: [email protected]

Abstract: Outlier detection in high dimensional data sets is a challenging data mining task. Mining outliers in subspaces seems to be a promising solution, because outliers may be embedded in some interesting subspaces. Searching for all possible subspaces can lead to the problem called "the curse of dimensionality". Due to the existence of many irrelevant dimensions in high dimensional data sets, it is of paramount importance to eliminate the irrelevant or unimportant dimensions and identify interesting subspaces with strong correlation. Normally, the correlation among dimensions can be determined by traditional feature selection techniques or subspace-based clustering methods. The dimension-growth subspace clustering techniques can find interesting subspaces in relatively lower dimension spaces, while dimension-reduction approaches try to group interesting subspaces with larger dimensions. This paper aims to investigate the possibility of detecting outliers in correlated subspaces. We present a novel approach by identifying outliers in the correlated subspaces. The degree of correlation among dimensions is measured in terms of the mean squared residue. In doing so, we employ a dimension-reduction method to find the correlated subspaces. Based on the correlated subspaces obtained, we introduce another criterion called "shape factor" to rank most important subspaces in the projected subspaces. Finally, outliers are distinguished from most important subspaces by using classical outlier detection techniques. Empirical studies show that the proposed approach can identify outliers effectively in high dimensional data sets.

Keywords: Outlier Detection, Subspace Outlier Detection, Subspace Clustering, Shape Factor, Dimension Reduction

DOI: 10.3233/FI-2010-217

Journal: Fundamenta Informaticae, vol. 98, no. 1, pp. 71-86, 2010