Data cleaning survey and challenges – improving outlier detection algorithm in machine learning

1. Outlier Detection: The term “outlier detection” refers to the identification of values that stand apart from the norm. Although the exact definition of an outlier varies depending on the application, commonly acknowledged explanations, as articulated by Hawkins and al, include “an outlier is an observation which deviates so much from other observations as to arouse suspicions that it was generated by a different mechanism” and “an outlier observation is one that appears to deviate markedly from other members of the sample in which it occurs.” Outlier detection finds utility across various domains of application. It is seamlessly applicable to data, software, and images. Primarily, the techniques of anomaly detection and misuse are employed to filter out erroneous data and generate an accurate dataset [21,22].

2. Data Deduplication, As defined by Chu and al, it is alternatively referred to as duplicate detection, record matching, record linkage, or entity resolution, encompasses the process of identifying tuples within one or more relationships that correspond to the same real-world entity. For instance, a customer might be registered multiple times in a customer database if the customer used various names when purchasing; an item might be appeared many times in an online shopping site; and duplicate records might represent after a data integration project because that record had various representations in original data sources. A data deduplication process usually involves many phases and choices, including designing similarity metrics to evaluate the similarity for a pair of records, training classifiers to determine whether a pair of records are duplicates, clustering all records to obtain clusters of records that represent the same real-world entity, consolidating clusters of records to unique representations, designing blocking or distributed strategies to scale up the deduplication process, and involving humans to decide whether a record pair are duplicates when machines are uncertain [4].

3. Data Imputation, Little and al have elucidated that data imputation pertains to the method of substituting alternative values to fill in for missing data. Most statistical software defaults to rejecting any instance with a missing value, which may add bias or reduce the generalisability of the results. By replacing missing information with an estimated value relying on other available information, imputation maintains all cases. The data set can be analyzed using methods applied for the whole data once all values have been imputed. Scientists have taken on various ideas to clarify missing data, but the bulk of them makes a bias. Data imputation is the process of retaining the majority of the dataset’s data and information by replacing missing data with another value. These methods are used because it would be unsuitable to delete data from a dataset each time. in addition, doing so would minimize the dataset’s amount, raising questions about bias and impairing analysis [23].

4. Data Transformation, After cleaning, the data is transformed into a appropriate format. As outlined by Han and al Data transformation aids to treat the data readily. Data transforming can be simple or complicated relying on the required alterations on the data. Standardizing data, character set transformation, encoding processing, merging or splitting fields,enforce standard patterns or trim long strings, conversion units of measurements into a suitable format, aggregation and consolidation. After completing the data transformation, the data is loaded for processing. Finally, the data analysts and senior management can make logical and beneficial decisions based on the processed data [24].

1. Time-Consuming Process; duplicate data must be deleted, missing entries must be corrected or added, values that were entered wrongly must be corrected, formatting must be consistent, and other activities that take time must be completed.

2. Overwhelming Amount of Data; applications such as Data Warehouses load massive amounts of data from various sources continuously and further, they carry a notable amount of dirty data. In such a situation, the role of data cleansing becomes both formidable and significant at the same time.

3. Data Formats Lack Consistency; Domain format errors happen when the value for a particular attribute is valid but does not comply with the format of the domain. For instance, a specific database demands first name and surname to be detached with a point but the input is without a point. In this situation, while the input may be true but it does not conform with the domain format.

4. Select the Appropriate Tool or Approach; Data cleaning is much more effective when we chose our approach or tool wisely.

5. Outsourcing the Process is expensive given that data cleaning is complex work, and many businesses think of outsourcing the task to third-party vendors.

6. Scalability; Scaling data cleaning techniques to the huge and speedily growing datasets of the Big Data era will be an important challenge, which may be restricted in some domains including medicine and finance.

7. User Engagement; although various research has been done about involving humans to execute data deduplication, for instance, through active learning, participation of humans in other data cleaning tasks, such as reforming IC violations, and taking user feedback in detecting data quality rules, is yet to be explored, and other challenges.

8. Data Privacy and Security: The cleaning process should ensure that sensitive information is not revealed, and data privacy is maintained. It is challenging to balance data utility with data privacy and security.

1. Point Outlier: Bansal and al defined Point Outlieris as a single data point that falls outside the range of the rest of the dataset. There may be an evident trend, pattern, or grouping in the dataset, and an outlier as a data point will typically differ from this. Point outliers can usually be attributed to mistakes in input or measurement of the data. For example, an outlier may occur in health data if there is an error in recording a unit of measurement for a patient’s data. Missing a value while recording the information of a patient can also create a marked point outlier within the dataset. This kind of outlier can be relatively straightforward to detect through visual means. Point outliers are visible if the dataset is plotted in two or more dimensions, where the outlier as a data point would sit far apart from the rest of the dataset [27].

2. Contextual Outlier Pahuja and al State contextual Outlier as a data point that is remarkably dissimilar from the rest of the dataset, but only within a particular context. The context of a dataset may vary seasonally or due to broader economic behavior or trends. A contextual outlier will be apparent when the context of the dataset changes. This could be due to economic fluctuations, seasonal weather changes, or variations in customer behavior during essential holidays. For this reason, a contextual outlier may appear like a usual data point in another context [11].

3. Collective Outlier Divya and al explain Collective Outlier as a group of data points that deviates significantly from the trends in the remainder of the dataset. The individual data points in a group outlier may not appear to be contextual or point outliers. It is only when the data points are considered as a collection that anomalous trends are identified. As a result, identifying a set of outliers can be a challenging type of outlier to recognize. Collective outliers are crucial in detecting concept deviation in machine learning. It indicates that a series of data has deviated from the expected behavior within the model. For instance, if a time series plot showing the number of subscribers and unsubscribers to an email marketing list displays daily fluctuations or seasonal trends, a collective outlier could be indicated if the number of subscribed users remains completely stable for many weeks with no variation. Since new users subscribing and individual users unsubscribing are a common occurrence, a stable count would be flagged as an outlier [13].

As per Wang and al Outliers can be categorized into different types, such as vector outliers, sequence outliers, trajectory outliers, and graph outliers, depending on the types of data being analyzed [15].

1. Intrusion Detection specifies all of the doubtful patterns that may indicate a system or network attack from someone attempting to compromise a system (hackers). There are different types of intrusion systems based on network, system calls and kind of packets.

2. Fraud Detection is at alarming rate and in consequence becomes a big threaten for the banks and institution using a credit card transactions, Fraud is reported under crime activities that includes mobile phones fraud detection, banks, commercial etc. Outlier is basically used to detect a noisy data that is being presented in the original data.

3. Medical Diagnoses and Public Health, the patient data is to be collected from the various features of patient like height, weight, blood test, patient age. Many methods of outlier detection is applied in medical diagnoses which aids to detect critical diseases at early stage for preventing it to become a severe and life-taking disease. Outlier detection plays a key role in identifiying different kinds of cancers.

4. Image Detection, images can be of any kind, the key aim of outlier detection is to identify an abnormal behavior of the images. Each data is composed of the various features of the image that contained image coordinates, texture, color and brightness.

5. Text Data Detection, many text corpora contain heterogeneous documents, some of which may be anomalous and worth understanding more. The key aim of outlier detection is to handle the dirty data that is presented in the text.

6. Sensor Network is a grouping of specialized transducers with ability of communication which aids to supervise and record conditions like pressure, vibrations, humidity, intensity of sound, level of pollution and concentration of chemicals etc. at various positions. A sensor network have many detection station called sensor node. Each node is less weighted, portable, and very small in size. Fundamentally via the outlier detection techniques faulty sensor networks are identified so that communication level is to be augmented. Reliability in wireless sensor networks is influenced by many causes such us environment condition, using low quality sensors etc. that leads to corrupted data generations by sensor containing missing values.

7. Data Logs and Process Logs often contain a large volume of unstructured and semi-structured data that can be challenging to analyze. Data logs usually capture all interactions and events occurring in a system, while process logs provide a detailed record of the steps taken in a particular process or workflow, which can be useful in identifying errors, inconsistencies, redundancies, or irrelevant information. These issues must be identified and removed to produce clean data.

8. Cyber Security, security and surveillance are crucial considerations. Specifically, ensuring secure logging and log administration for computer networks is of utmost importance, as they enhance authenticity and security intelligence. Additionally, detecting anomalies in surveillance videos is an area of research that is both practical and exciting.

9. The Internet of Things (IoT) involves numerous sensors that continuously monitor environmental conditions and are combined to gather information about a particular area or region for a specific purpose. However, before utilizing this data, it is crucial to ensure its quality, as it may contain outliers that can negatively impact the overall efficiency. Therefore, detecting and identifying these outliers is essential to maintain the integrity and accuracy of the IoT data.

1. Supervised Outlier Detection Methods use a training dataset that contains labels for both normal and outlier data objects. When a new test data object is received, it is classified into one of the two classes based on its similarity with the normal data set. If the data object fits with the model, it is classified as normal, otherwise, it is treated as an outlier. The classification is done based on two labels, normal and outlier, and the test data is classified into one of these two categories. Some commonly used supervised outlier detection methods include Support Vector Machines (SVM), Decision Trees, Logistic Regression, Naive Bayes and Random Forests.

2. Unsupervised Outlier Detection Methods do not use any pre-existing labels for the data objects. Instead, the data objects are clustered, and those objects that are far away from each cluster are considered as outliers. When a new test data object is received, clustering is performed, and outliers are detected based on this mechanism. Some commonly used unsupervised outlier detection methods include clustering-based methods (e.g. k-means), density-based methods (e.g. LOF), and distance-based methods (e.g. kNN).

3. Semi-supervised Outlier Detection Methods assume that labeled instances are only available for one class. It is often difficult to collect labels for the other class. Techniques that assume availability of only the outlier instances for training are not very popular. Normal behavior is typically welldefined, and hence it is easier to construct representative models for normal behavior from the training data. Unsupervised outlier detection methods that are commonly used include Self-Training, Co-training, Generative Adversarial Networks (GAN),Semi-Supervised Active Learning for Anomaly Detection(SALAD).

1. Statistics-Based. Assume that the normal behavior of the records (data points) would appear in high probability of a stochastic model while outliers occurs at a low probability regions of a stochastic model, there are two commonly used categories of approaches for statistics-based outlier detection [30]. First one is based on hypothesis testing when they basically calculate a test statistic based on observed data points, which is used to determine whether the null hypothesis (there is no outlier in the dataset) should be rejected. There are many tests such as the Grubbs Test [31] and the Tietjen-Moore Test [32]. The second category of statistics-based outlier detection techniques aims at fitting the distribution or inferring a probability density function (pdf also known as density of a continuous random variable) based on the observed data. Fitting a distribution can be done based on parametric approaches or non-parametric approaches. Parametric approaches for fitting a distribution assume that the data follows an underlying distribution and aim at finding the parameters of the distribution from the observed data. For example, assuming the data follows a normal distribution, parametric approaches would need to learn the mean and variance for the normal distribution. In contrast, non-parametric approaches [33] make no assumption about the distribution that generates the data; instead, they infer the distribution from the data itself.

Statistics based techniques have many advantages such as:

2. Distance-Based. Outlier detection approaches that employ distance between data points frequently specify a distance between data points that is used to characterize typical behavior. A typical data point, for example, should be near to many other data points, and data points that depart from such regular behavior are labeled as outliers. Distance-based outlier identification methods are further classified as global or local based on the reference population utilized to determine whether a point is an outlier. A global distance-based outlier identification approach assesses whether a data point is an outlier based on its distance from all other data points in the dataset. A local approach, on the other hand, analyzes the distance between a location and its neighboring points for determining outliers.

There are many advantages of distance-based techniques:

3. Model-Based. Model-based outlier identification approaches first train a classifier model using a collection of labeled data points before applying the trained classifier to a test data point to evaluate whether it is an outlier. Model based methods presume that utilizing the supplied feature space, a classifier can be trained to discriminate between normal and anomalous data points. If none of the trained models categorize a data point as normal, they mark it as an outlier. Model-based approaches are further classified into two types based on the labels used to train the classifier: multi-class model-based techniques and one-class model-based techniques. Multiclass model-based methods are based on the assumption that the training data points comprise labeled examples from several normal classes. One-class model based approaches, on the other hand, presume that all of the training data points belong to a single normal class.

The advantages of model-based techniques include:

– First Stage: The assignment step Assign each observation to the cluster with the nearest mean: that with the least squared Euclidean distance (Mathematically, this means partitioning the observations according to the Voronoi diagram generated by the means)

– Second Stage: Update step Recalculate means (centroids) for observations assigned to each cluster. mi(t+1)=1|Si(t)|∑xj∈Si(t)xj