Performance validity testing via telehealth and failure rate in veterans with moderate-to-severe traumatic brain injury: A veterans affairs TBI model systems study¹

1.1Performance validity

Unfortunately, there exist myriad factors that threaten the validity of neuropsychological evaluation results. The behavior of examinees as it relates to the credibility of neuropsychological test results is broadly referred to as performance validity; however, the terms response bias, effort, and engagement have also been used in the literature that has accumulated during the past 20 years (Lippa, 2018). Invalid performances can be driven by external gains (e.g., malingering to secure financial benefit or avoid responsibility), internal gains (e.g., factitious disorder), unconscious drives (e.g., somatization, conversion disorder), diminished neurogenic drive (e.g., abulia, akinesia), or simply impacted by sensory impairments (e.g., visual/auditory deficits). Importantly, performance invalidity across its forms is a prominent and prevalent concern, accounting for as much as 50%of the variability in neuropsychological test scores (Green, Rohling, Lees-Haley, & Allen, 2001). Historically a focus of forensic neuropsychologists, the investigation (and measurement) of performance invalidity is of clinical concern as well. Base rates of performance invalidity during neuropsychological assessment vary across settings (i.e., litigation, clinical/rehabilitation) and approaches 40%in the presence of external incentives or when referrals specify medical conditions commonly encountered in rehabilitation settings (e.g., mild TBI, chronic pain, neurotoxin injury; Mittenberg, Patton, Canyock, & Condit, 2002). Failure to recognize this prevalent problem in the rehabilitation setting results in the inaccurate interpretation of one’s neurocognitive functioning, potentially leading to inappropriate decision-making.

Accordingly, considerable research has been dedicated to the development of performance validity tests (PVTs) which are objective measures designed to distinguish between valid and invalid neuropsychological test performance. Because of the prevalence and impact of performance invalidity, the inclusion of PVTs during neuropsychological evaluations has become a practice standard (Chafetz et al., 2015; Heilbronner et al., 2009; Sweet, Benson, Nelson, & Moberg, 2015).

1.2Neuropsychological testing and telehealth

The COVID-19 pandemic brought about significant change to standard practice, as neuropsychologists across the nation (in rehabilitation hospitals as well as traditional outpatient settings) were forced to suspend traditional face-to-face neuropsychological evaluations (Bilder et al., 2020). Practitioners were faced with a challenging decision: suspend evaluation services entirely or provide service through alternative modalities such as telehealth which the field of neuropsychology has been slow to adopt. However, the provision of teleneuropsychology (TeleNP) services would provide a means to safely deliver (albeit in a limited form) services to persons who need of neuropsychological evaluations while maintaining travel restrictions and distancing recommendations per the Centers of Disease Control and Prevention vis-à-vis the coronavirus pandemic.

In the rehabilitation setting, the decision to halt services would risk delaying diagnoses of possible neurocognitive and neuropsychiatric disorders, initiation of interventions such as cognitive rehabilitation and evidence-based therapies for psychological disorders, and monitoring of recovery (e.g., following pharmacological treatment or neurosurgical interventions). The effects of delayed diagnosis and treatment have a substantial psychosocial and economic cost (Rosado et al., 2018; Glen et al., 2020; Donders 2019). In contrast, the decision to provide services through innovative modalities via TeleNP requires a mental shift from the practice standard including the way that neuropsychological tests were historically developed and standardized.

Neuropsychology practice has been criticized for failing to adopt telehealth and other technological advances embraced by other health care disciplines (Miller & Barr, 2017). Investigations into the feasibility, validity, and clinical utility of teleNP began emerging in the late 20^th century, although this research was limited in scope, infrequently appeared in scientific journals, and failed to meaningfully impact the field. However, the coronavirus pandemic has forced the field of neuropsychology to prioritize TeleNP in both research and clinical practice. Indeed, in response to the global health crisis caused by the coronavirus pandemic, the Inter-organizational Practice Committee (comprised of the American Academy of Clinical Neuropsychology/American Board of Clinical Neuropsychology, National Academy of Neuropsychology/Division 40 of the American Psychological Association, the American Board of Professional Neuropsychology, and the American Psychological Associates Services, Inc.) issued guidance on TeleNP practice (Bilder et al., 2020).

There now exists a growing body of research supporting the reliability and validity of neuropsychological assessment delivered remotely via telehealth (e.g., Brearly et al., 2017; Marra, Hamlet, Bauer, & Bowers, 2020). Notably, the majority of this evidence base utilized subjects who were tested under the controlled conditions of a remote clinic, oftentimes with the use of a technician to establish videoconference connection and/or assist with task administration (Bilder et al., 2020). Much of the burgeoning research on TeleNP has used web-conferencing platforms as the service delivery method; while innovative, concerns about web-conferencing neuropsychological service delivery include access, particularly among rural or economically disadvantaged persons (i.e., the digital divide; Ramsetty & Adams, 2020; Van Dijk, 2017). Even among persons with internet access, criticism of TeleNP has been raised based on research showing that reliability and validity of TeleNP are impacted by patient characteristics (e.g., age, comfort and familiarity with computers) and technology factors such as monitor size, screen resolution, and internet speed (Brearly et al., 2017; Hirko et al., 2020). These access barriers have led to deeper consideration of TeleNP via telephone (Caze et al., 2020; Lacritz et al., 2020). Indeed, TeleNP via telephone has been effectively utilized in research settings dating back to the 1980s and continuing to present day (Brandt et al., 1988; Bunker et al., 2017; Castanho, Amorim, Zihl, Palha, Sousa, Santos, 2014; Castanho et al., 2016; Matchanova et al., 2020; Nelson et al., 2020; Pendlebury, Welch, Cuthbertson, Mariz, Mehta, & Rothwell, 2013; Plassman, Newman, Welsh, & Helms, 1994).

1.3TeleNP and PVTs

Regardless of neuropsychology service delivery method, including TeleNP, performance validity remains a primary concern. Unfortunately, research into the use of PVTs in TeleNP is sparse. A vital step in PVT development is establishing adequate specificity in groups with known neurocognitive impairment. In this context, specificity reflects the proportion of participants with valid performance accurately identified as such by the PVT [Specificity = True Negatives ÷ (True Negatives+False Positives)] with adequate specificity defined as values > .90 by convention. In contrast, sensitivity reflects the proportion of participants with invalid performance (e.g., poor effort or task engagement) accurately identified by the PVT [Sensitivity = True Positives ÷ (True Positives+False Negatives]). In clinical contexts, specificity is prioritized over sensitivity because false-positive errors (i.e., incorrectly labeling impaired cognitive performance as invalid) are considered more detrimental to the patient than false negative errors (i.e., incorrectly labeling invalid performance that appears to show cognitive impairment as valid). As with traditional in person PVT development, a critical first step in establishing PVT utility in TeleNP is to demonstrate a low rate of false positive classification errors among examinees, especially among those with bona fide cognitive impairment. Thankfully, there exist a number of candidate PVTs that are robust to the effects of genuine neurocognitive impairment, demonstrating excellent specificity across a range of clinical groups (Lippa 2018). Surprisingly, there is a paucity of research on PVTs administered via TeleNP using telephone-based methods of communication. As such, the primary objective of the current study was to report the specificity of two well-established PVTs administered via TeleNP in patients with traumatic brain injury (TBI) admitted for inpatient neurorehabilitation. We selected two PVTs (See Methods) that rely solely on verbal communication given the concerns noted above as well as the finding that verbally mediated neuropsychological tasks translate more reliability and validly to TeleNP than visually dependent tasks (Brearly et al., 2017). It was hypothesized that these PVTs would maintain adequate specificity, consistent with previously published research utilizing comparable clinical groups.

2.1Participants

This study was a secondary analysis of the Veterans Affairs TBI Model Systems (VA TBIMS) dataset. The VA TBIMS is a prospective longitudinal multi-center study of rehabilitation outcomes following inpatient TBI rehabilitation (Lamberty et al., 2014; Ropacki et al., 2018). This study included participants enrolled at two study sites, the VA Polytrauma Rehabilitation Centers (PRCs) in San Antonio, Texas and Tampa, Florida. The CARF-accredited VA PRCs are rehabilitation programs that provide comprehensive inpatient rehabilitation services for veteran and active duty service members incurring injuries affecting the brain and/or other bodily systems. VA TBIMS inclusion criteria as follows: (1) TBI diagnosis per case definition: a self-reported or medically documented traumatically-induced structural brain injury or physiological disruption of brain function due to external force evidenced by onset or worsening of: loss/decreased consciousness; mental state alteration; memory loss for events immediately before or after the injury; transient or stable neurological deficits; or intracranial lesion; (2) age≥16 years at time of TBI; (3) admission to a PRC for TBI rehabilitation; and (4) informed consent by the participant or legally authorized representative. Additional inclusion criteria for this secondary analysis was (5) data collected after March 01, 2020 (which corresponded with the time that performance validity testing was added to data collection at the San Antonio and Tampa PRCs), and (6) medically documented moderate or severe TBI based on established metrics (see section 2.2.3 TBI Characterization, below). Excluded were participants not followed after March 01, 2020, or participants were too cognitively impaired to participate in neuropsychological testing, or who were missing one or more PVT metrics. See Fig. 1 for flow diagram. Study design and procedures were approved by each site’s Institutional Review Board.

Fig. 1

Flow diagram.

2.2Measures

2.3Procedures

Demographic characteristics were collected via structured interview (Lamberty et al., 2014; Ropacki et al., 2018) upon enrollment into the VA TBIMS. Injury information (e.g., TBI severity, cause of injury) were collected via review of medical records from emergency departments and acute care hospitalizations. VA TBIMS procedures include quarterly audit of research records, in which a research team member other than the research assistant initially collecting the data reviews data collection forms and source materials (i.e., hospital records) to ensure accuracy of coded data. Any discrepancies that arise are reviewed by the team’s research coordinator and coded data are corrected if necessary.

PVTs were administered alongside neuropsychological tests via telephone with participants following inpatient rehabilitation discharge at regularly scheduled follow up time points anchored to TBI anniversary date. PVT failure rates at selected cutoffs were presented as percentages. Demographic and injury characteristics were presented as means and standard deviations for continuous data and as percentages for categorical data.

3.1Participants

Data on PVT performance were collected between March 01, 2020 and September 30, 2020. Of 93 participants due for follow up in this time frame,64 had medically documented moderate-to-severe TBI. Of these, 89%were followed. However, 7%of those followed were unable to participate in testing because they were too severely cognitive impaired (e.g., prolonged disorders of consciousness), and 4%of those who participated in testing were missing PVT data. This resulted in a final sample size of 51 participants, which was 96%of those with TeleNP neuropsychological testing during follow up (see flow diagram, Fig. 1). Participants were 100%male, with an average age of 34 years (SD = 14) at time of TBI, over 55%had more than a high school education, and 71%identified as White. Cause of injury was most often vehicular related (61%). See Table 1 for further details of study sample demographic and injury characteristics. TeleNP neuropsychological testing took place between 1 to 7 years post-TBI.

3.2PVT performance

21-FC-TC scores ranged from 7 to 21 (M = 16.4; SD = 3.1). At four of the five cutoffs, failure rate was less than 10%with a range of 2.0%to 7.8%. At the liberal cutoff of < 12, failure rate was 13.7%(see Table 2). RDS scores ranged from 5 to 16 (M = 10.5; SD = 2.4). At three of the four cutoffs, failure rate was less than 10%with a range of 2.0%to 9.8%. However, a liberal cutoff of < 8 resulted in a failure rate of 19.6%(see Table 2).

4.1Future directions and study limitations

Selection bias may have influenced the results of our study. We relied on a sample of veterans who may be more motivated to perform well when compared to the general clinical population, as evidenced by the sample’s willingness to participate in the TBI Model Systems study and to voluntarily complete additional TeleNP and PVT measures. Another limitation is the lack of criterion PVTs beyond those investigated in the current study. As such, it is possible that some of the performances below PVT cutoffs observed in the current sample were indeed invalid (as opposed to false positive classification errors). However, this risk is thought to be minimal given the voluntary nature of the study and the lack of external incentives for poor performance. The veterans in our study were informed their responses to exams would be deidentified, kept confidential for research purposes and, therefore, not be part of the clinical record that could impact future compensation or veterans’ benefit determinations. Moreover, none of the veterans in the sample performed below reported cut-scores on both PVTs (i.e., none met criteria for definite malingering). Lastly, the current study utilized TeleNP via telephone. As such, it is unclear the extent to which findings may generalize to other telehealth modalities, such as video-based evaluations.