Data Preprocessing and Intelligent Data Analysis

Article type: Research Article

Authors: Famili, A.^{a; *} | Shen, Wei-Min^{b; 1} | Weber, Richard^{c; 2} | Simoudis, Evangelos^{d; 3}

Affiliations: [a] Institute for Information Technology, National Research Council Canada, Ottawa, Ontario K1A 0R6, Canada | [b] Information Sciences Institute, University of Southern California, 4676 Admiralty Way, Marina del Rey, CA 90292, USA | [c] Management Intelligenter Technologien GmbH, Promenade 9, 52076 Aachen, Germany | [d] IBM Almaden Research Center, 650 Harry Road, San Jose, CA 95120-6099, USA

Correspondence: [*] Corresponding author. E-mail: [email protected], URL: http://ai.iit.nrc.ca/~fazel/.

Note: [1] E-mail: [email protected], URL: http://www.isi.edu/isd/shen/.

Note: [2] E-mail: [email protected], URL: http://www.mitgmbh.de/.

Note: [3] E-mail: [email protected].

Abstract: This paper first provides an overview of data preprocessing, focusing on problems of real world data. These are primarily problems that have to be carefully understood and solved before any data analysis process can start. The paper discusses in detail two main reasons for performing data preprocessing: (i) problems with the data and (ii) preparation for data analysis. The paper continues with details of data preprocessing techniques achieving each of the above mentioned objectives. A total of 14 techniques are discussed. Two examples of data preprocessing applications from two of the most data rich domains are given at the end. The applications are related to semiconductor manufacturing and aerospace domains where large amounts of data are available, and they are fairly reliable. Future directions and some challenges are discussed at the end.

Keywords: Data preprocessing, Data analysis, Data mining

DOI: 10.3233/IDA-1997-1102

Journal: Intelligent Data Analysis, vol. 1, no. 1, pp. 3-23, 1997