Improving data quality with label noise correction

Article type: Research Article

Authors: Li, Benchong^* | Gao, Qiong

Affiliations: School of Mathematics and Statistics, Xidian University, Xi’an, Shaanxi, China

Correspondence: [*] Corresponding author: Benchong Li, School of Mathematics and Statistics, Xidian University, Xi’an, Shaanxi 710126, China. E-mail: [email protected].

Abstract: Data gathered from real world often contains label noise, which is harmful to the quality of data. Moreover, any data mining process suffers a deterioration when it is applied on noisy data. In this paper, a new approach is proposed to improve data quality by correcting mislabeled data. The proposed method employs a procedure to estimate the level of the noise in the data and combines this noise estimation with a correction process. A clustering method and k nearest neighbors approach are applied in the correction process. Extensive experimental results using real-world data sets from UCI machine learning repository are provided. The empirical study shows that our approach successfully improves data quality in many cases and outperforms several correction methods.

Keywords: Label noise, noise correction, noise rate estimation, classification

DOI: 10.3233/IDA-184024

Journal: Intelligent Data Analysis, vol. 23, no. 4, pp. 737-757, 2019