Five shades of gray: Conceptual and methodological issues around multivariate models of performance validity

Issue title: Thematic Issue: The Science and Practice of Neuropsychological Assessment in Neurorehabilitation

Guest editors: Daniel Klyce, Ana Mills and Paul Dukarm

Article type: Research Article

Authors: Erdodi, Laszlo A.

Affiliations: Department of Psychology, University of Windsor, Windsor ON, Canada. Tel.: +1 519 253 3000 / Extension 2202; Fax: +1519 973 7021; E-mail: [email protected]

Abstract: OBJECTIVE:This study was designed to empirically investigate the signal detection profile of various multivariate models of performance validity tests (MV-PVTs) and explore several contested assumptions underlying validity assessment in general and MV-PVTs specifically. METHOD:Archival data were collected from 167 patients (52.4%male; MAge = 39.7) clinicially evaluated subsequent to a TBI. Performance validity was psychometrically defined using two free-standing PVTs and five composite measures, each based on five embedded PVTs. RESULTS:MV-PVTs had superior classification accuracy compared to univariate cutoffs. The similarity between predictor and criterion PVTs influenced signal detection profiles. False positive rates (FPR) in MV-PVTs can be effectively controlled using more stringent multivariate cutoffs. In addition to Pass and Fail, Borderline is a legitimate third outcome of performance validity assessment. Failing memory-based PVTs was associated with elevated self-reported psychiatric symptoms. CONCLUSIONS:Concerns about elevated FPR in MV-PVTs are unsubstantiated. In fact, MV-PVTs are psychometrically superior to individual components. Instrumentation artifacts are endemic to PVTs, and represent both a threat and an opportunity during the interpretation of a given neurocognitive profile. There is no such thing as too much information in performance validity assessment. Psychometric issues should be evaluated based on empirical, not theoretical models. As the number/severity of embedded PVT failures accumulates, assessors must consider the possibility of non-credible presentation and its clinical implications to neurorehabilitation.

Keywords: Performance validity, embedded validity indicators, multivariate models, Erdodi Index

DOI: 10.3233/NRE-218020

Journal: NeuroRehabilitation, vol. 49, no. 2, pp. 179-213, 2021