Affiliations: Department of Statistics and Quantitative Methods, University of Milan-Bicocca, Milan, Italy | CRISP Research Centre, University of Milan-Bicocca, Milan, Italy
Note: [] Corresponding author: Fabio Mercorio, CRISP Research Centre, University of Milan-Bicocca, Milan, Italy. Tel.: +2 6448 5864; Fax: +2 6448 2182; E-mail: [email protected]
Abstract: Nowadays Information Systems generate a lot of data for supporting the activities of firms, organisations, and state agencies. While on the one hand such data are primarily collected for realising domain-specific services (e.g., state agencies use data for managing healthcare and retirement contributions) on the other hand domain analysts aim at using such data for studying the dynamics of subjects' behaviours or phenomena over time. Thus, the quality of data plays a key role in ensuring the effectiveness of the overall knowledge discovery process. In such a context, most of the research on data quality is aimed at automatically identifying cleansing activities, namely a sequence of actions able to cleanse a dirty dataset, which are often developed and coded manually requiring a relevant effort for domain-experts. This work is concerned with using AI Planning for both modelling data quality requirements and automatically identifying cleansing activities. To this end, we formalise the concept of cost-optimal Universal Cleanser - a collection of best cleansing actions for each data inconsistency identified - as a planning problem, then we present a motivating government application where data have been cleansed accordingly, making both the source and cleansed datasets publicly available for download.
Keywords: Data quality, data cleansing, data quality on KDD
