Penalized logistic regression to improve predictive capacity of rare events in surveys

Issue title: Special section: Fuzzy Systems in Management and Information Science

Guest editors: José M. Merigó, Salvador Linares-Mustaros and Joan Carles Ferrer-Comalat

Article type: Research Article

Authors: Pesantez-Narvaez, Jessica | Guillen, Montserrat^{; *}

Affiliations: Department of Econometrics, Riskcenter-IREA, University of Barcelona, Barcelona, Spain

Correspondence: [*] Corresponding author. Montserrat Guillen, Tel.: +34934037039; Fax: +34934021821; E-mail: [email protected].

Abstract: Logistic regression as a modelling technique of rare binary dependent variables with much fewer events (ones) than non-events (zeros) tends to underestimate their probability of occurrence. The vast literature devoted to the prediction of rare binary data identifies several ways to improve predictive performance by making modifications to the likelihood estimation. We propose two weighting mechanisms for incorporation in a pseudo-likelihood estimation that improve the predictive capacity of rare binary responses in data collected in complex surveys. We multiply sampling weights by specific correctors that lead to lower root mean square errors for event observations in almost all deciles. A case study is discussed where this method is implemented to predict the probability of suffering a workplace accident in a logistic regression model that is estimated with data from a survey conducted in Ecuador.

Keywords: Survey data, sampling design, uncommon events, weighting, pseudo-likelihood

DOI: 10.3233/JIFS-179641

Journal: Journal of Intelligent & Fuzzy Systems, vol. 38, no. 5, pp. 5497-5507, 2020