Variable selection using support vector regression and random forests: A comparative study

Article type: Research Article

Authors: Ben Ishak, Anis

Affiliations: Université de Tunis, ISGT, LR99ES04 BESTMOD, 2000, Le Bardo, Tunisia. Tel.: +216 97 549 940; Fax: +216 71 568 767; E-mail: [email protected]

Abstract: Variable selection is crucial for improving interpretation quality and forecasting accuracy. To this end, it is very interesting to choose an effective dimension reduction technique suitable for processing data according to their specificity and characteristics. In this paper, the problem of variable selection for linear and nonlinear regression is deeply investigated. The curse of dimensionality issue is also addressed. An intensive comparative study is performed between Support Vector Regression (SVR) and Random Forests (RF) for the purpose of variable importance assessment then for variable selection. The main contribution of this work is twofold: to expose some experimental insights about the efficiency of variable ranking and selection based on SVR and on RF, and to provide a benchmark study that helps researchers to choose the appropriate method for their data. Experiments on simulated and real-world datasets have been carried out. Results show that the SVR score ∂ Gα is recommended for variable ranking in linear situations whereas the RF score is preferable in nonlinear cases. Moreover, we found that RF models are more efficient for selecting variables especially when used with an external score of importance.

Keywords: Variable importance score, variable selection, support vector regression, random forests, nonlinearity, stepwise algorithm, curse of dimensionality, selection bias

DOI: 10.3233/IDA-150795

Journal: Intelligent Data Analysis, vol. 20, no. 1, pp. 83-104, 2016