Affiliations: Florida Atlantic University, Boca Raton, FL, USA
Correspondence:
[*]
Corresponding author: Taghi M. Khoshgoftaar, Empirical Software Engineering Laboratory, Department of Computer Science and Engineering, Florida Atlantic University, Boca Raton, FL 33431, USA. Tel.: +1 561 297 3994; Fax: +1 561 297 2800; E-mail: [email protected].
Abstract: Relatively little attention has been given in the data mining literature to noise handling procedures that deal specifically with a continuous dependent variable. We present a novel procedure that addresses the problem of detecting and correcting noise when the outcome variable is continuous. Our technique uses a procedure for handling missing data called multiple imputation, a well-known statistical methodology based on sound theoretical principles. We demonstrate the utility of our procedure using a real-world dataset with inherent noise and multiple levels of injected noise in numerous carefully designed controlled experiments. Further, we present a comparison with noise correctors developed using five well-known estimation procedures, providing good coverage of the commonly-used classes of estimation techniques such as linear regression, decision trees and neural networks. The results presented in this work demonstrate conclusively the strong noise detection and correction results of our procedure, which outperforms the five competing noise correctors.
Keywords: Data cleaning, data quality, quantitative noise correction, multiple imputation
