Floating-point histograms for exploratory analysis of large scale real-world data sets

Article type: Research Article

Authors: Boullé, Marc

Affiliations: Orange Labs, Lannion, France | E-mail: [email protected]

Correspondence: [*] Corresponding author: Orange Labs, Lannion, France. E-mail: [email protected].

Abstract: Histograms are among the most popular methods used in exploratory analysis to summarize univariate distributions. In particular, irregular histograms are good non-parametric density estimators that require very few parameters: the number of bins with their lengths and frequencies. Although many approaches have been proposed in the literature to infer these parameters, most existing histogram methods are difficult to exploit for exploratory analysis in the case of real-world data sets, with scalability issues, truncated data, outliers or heavy-tailed distributions. In this paper, we focus on the G-Enum histogram method, which exploits the Minimum Description Length (MDL) principle to build histograms without any user parameter. We then propose to extend this method by exploiting a new modeling space based on floating-point representation, with the objective of building histograms resistant to outliers or heavy-tailed distributions. We also suggest several heuristics and a methodology suitable for the exploratory analysis of large scale real-world data sets, whose underlying patterns are difficult to recover for digitization reasons. Extensive experiments show the benefits of the approach, evaluated with a dual objective: the accuracy of density estimation in the case of outliers or heavy-tailed distributions, and the effectiveness of the approach for exploratory data analysis.

Keywords: Density estimation, histograms, model selection, minimum description length, exploratory analysis

DOI: 10.3233/IDA-230638

Journal: Intelligent Data Analysis, vol. 28, no. 5, pp. 1347-1394, 2024