An overview of advances in reliability estimation of individual predictions in machine learning

Article type: Research Article

Authors: Bosnić, Zoran^{; *} | Kononenko, Igor

Affiliations: University of Ljubljana, Faculty of Computer and Information Science, Tržaška 25, Ljubljana, Slovenia

Correspondence: [*] Corresponding author. Tel.: +386 1 4768459; Fax: +386 1 4768498; E-mail: [email protected].

Abstract: In Machine Learning, estimation of the predictive accuracy for a given model is most commonly approached by analyzing the average accuracy of the model. In general, the predictive models do not provide accuracy estimates for their individual predictions. The reliability estimates of individual predictions require the analysis of various model and instance properties. In the paper we make an overview of the approaches for estimation of individual prediction reliability. We start by summarizing three research fields, that provided ideas and motivation for our work: (a) approaches to perturbing learning data, (b) the usage of unlabeled data in supervised learning, and (c) the sensitivity analysis. The main part of the paper presents two classes of reliability estimation approaches and summarizes the relevant terminology, which is often used in this and related research fields.

Keywords: Predictions, reliability, prediction accuracy, data perturbation, unlabeled examples, supervised learning

DOI: 10.3233/IDA-2009-0371

Journal: Intelligent Data Analysis, vol. 13, no. 2, pp. 385-401, 2009