A model selection approach for local learning

Article type: Research Article

Authors: Bontempi, Gianluca | Birattari, Mauro | Bersini, Hugues

Affiliations: Iridia – Université Libre de Bruxelles, 1050 Bruxelles, Belgium E‐mail: {gbonte, mbiro, bersini}@ulb.ac.be

Abstract: Local learning techniques, for each query, extract a prediction interpolating locally the neighboring examples which are considered relevant according to a distance measure. As other learning approaches, the local learning procedure can be conveniently decomposed into a parametric identification and a structural identification. While parametric identification is reduced to a linear regression, structural identification requires that the designer perform a certain number of choices. In this paper we focus on an automatic query‐by‐query selection of the bandwidth, a structural parameter which plays a major role in the final performance. We propose a local method where, for each query, different model candidates are first generated, then assessed and finally selected. We introduce in the context of local learning the recursive least squares algorithm as an efficient way to generate local models. Moreover, local cross‐validation is used as an economic way to validate different alternatives. As far as model selection is concerned, the winner‐takes‐all strategy and a local combination of the most promising models are explored. The method proposed is tested on six different datasets and compared with state‐of‐the‐art approaches.

Journal: AI Communications, vol. 13, no. 1, pp. 41-47, 2000