Virtual Reality Functional Capacity Assessment Tool (VRFCAT-SL) in Parkinson’s Disease

INTRODUCTION

Parkinson’s disease (PD) is a progressive neurodegenerative disorder involving both motor and non-motor symptoms. Cognitive deficits are common in PD, with an approximate 40%meeting criteria for mild cognitive impairment (PD-MCI) and 80%or more progressing to Parkinson’s disease dementia (PDD) over the course of illness [1, 2]. Impaired cognition is highly associated with loss of independence and caregiver burden, and the strongest predictor of need for placement in assisted living facilities [3, 4].

In movement disorder clinics, neuropsychological evaluations are useful for determining whether subjective reporting of decline in memory and cognition by the patient is supported by objective findings of deficits on standardized tests [5]. Test performances determine the magnitude of impairment with cognitive domains, and the overall profile is interpreted with respect to differential diagnosis of changes typical for PD versus other etiologies. Neuropsychological tests also serve as a proxy for impairment in everyday life. For instance, if an individual performs poorly on standardized tests of attention, they may have difficulty performing tasks such as balancing a checkbook or counting pills [6]. Impairment on a test of verbal fluency may relate to word-finding difficulties in conversation [7]. However, such associations are certainly not perfect, and the need for ecologically valid measures of functional ability have been emphasized in the literature [8–10]. Such measures have also become increasingly emphasized by the Food and Drug Administration (FDA) as outcome measures in clinical trials [11].

Several informant-based questionnaires have been validated for use in assessing cognitive functional capacity in PD. However, knowledgeable informants are not always present during clinical evaluations, and these ratings can be biased by caregiver mood, burden, and other factors [12, 13]. Anosognosia of cognitive deficits [14, 15] and motor impairment [16] in PD also present limitations on the reliability of patient self-report.

Several standardized tests of real-world functioning have been developed and explored in PD populations. The Everyday cognition battery assesses medication use, financial planning, and food preparation and nutrition skills through a series of tasks that involve reading, episodic memory, working memory, and inductive reasoning [17]. It has been shown to be valid in assessing older adults with subjective cognitive decline and demonstrated sensitivity to treatment with rivastigmine in a clinical trial with PD [18]. However, administration procedures are rather complex and completion time of one hour limits application in both clinical evaluations and research. The University of California Performance-based Skills Assessment (UPSA) assesses abilities on tasks related to finances, communication, planning/organization, travel, and household chores and was developed to be used as a functional outcome measure in schizophrenia trials [19]. It takes about 30 minutes to administer and was recently validated in a PD population against a battery of standardized neuropsychological tests. Performance on the UPSA was found to be significantly related to standardized neuropsychological tests even after controlling for demographic factors and PD motor symptoms and discriminated PD with normal cognition from PD-MCI and PDD [20]. The Complex Task Performance Assessment (CTPA) requires examinees to execute the role of librarian in a structured format [21]. This includes managing inventory, listening and responding to telephone messages, and planning. It was developed for use in older adults and also explored in a recent study of PD patients without cognitive impairment. The investigators found that completion time discriminated PD patients from healthy controls, but error rates were similar [22]. Moreover, PD patients on average took about 40 minutes to complete the CTPA, which is near the published cutoff time.

The Virtual Reality Functional Capacity Assessment Tool (VRFCAT) is a tablet-based task that requires examinees to complete a series of real-world tasks in a virtual-reality environment [23]. The task was developed with grant support from the National Institute of Health (NIH) and the FDA to serve as a measure of functional improvement in clinical trials. Like the UPSA, it was originally standardized for use in schizophrenia, with subsequent studies demonstrating validity for assessment of functional capacity in older adults with cognitive impairment [24, 25]. The VRFCAT was developed to measure four different functional abilities: meal preparation, using transportation, shopping, and managing currency. These scenarios were developed using immersive “first-person” gaming technology. The realistic, interactive environment challenges the examinee to a series of tasks related to making a food dish. First, they explore a kitchen to see what items they have for the recipe. Then, they use a bus schedule to find a bus that will take them to a grocery store. They find and purchase the necessary items at the store and use the schedule once again to find the bus that will take them home. Exact change is needed for both bus trips and to purchase the food. Patients sequentially complete the scenarios through a progressive storyboard design. Across the 4 scenarios, there is total of 12 different tasks or “objectives”. For each objective, participants who are unable to complete the objective within a pre-specified time period are “pushed” to the next objective (referred to as “Forced Progression”). The assessment takes approximately 30 minutes to complete. The primary VRFCAT end point is Total Time. Secondary endpoints are Total Errors and Total Forced Progressions.

An abbreviated version of the test, the VRFCAT-SL was recently developed that reduces completion time to less than 15 minutes in healthy older adults. Aside from technical assistance that might be needed from the examiner, the task is self-administered and scoring is automated. Normative data with gender, age, and education correction are available from the publisher. These features, along with elimination of additional test materials and manipulatives, make the VRFCAT-SL particularly attractive for assessing functional abilities within the context of a comprehensive neuropsychological evaluation. There are also multiple versions available to reduce practice effect with serial testing.

We recently began administering the VRFCAT-SL to PD patients referred for clinical neuropsychological evaluations to assess functional capacity. The present study is a retrospective chart analysis to explore utility of the VRFCAT-SL as measure of functional capacity in clinical neuropsychological evaluations of subjective cognitive decline in PD. Specifically, we sought to determine whether VRFCAT-SL performance measures were associated with scores on standardized neuropsychological tests and sensitive to cognitive impairment in PD. We also wanted to assess the potential impact of motor disturbances on ability to complete the VRFCAT-SL. Finally, we examined whether the use of a standardized self-report scale for impact of cognition changes on instrumental ADLs, the Parkinson’s disease Cognitive Function Rating Scale (PD-CFRS) [26], would provide similar information regarding real-world functional ability.

METHODS

This retrospective chart review was approved by the Human Subjects Protection Program at the Medical University of South Carolina, Pro00098834.

Participants were 30 consecutive patients followed in a tertiary care Movement Disorders clinic for idiopathic PD at an academic medical center who were referred for neuropsychological evaluation. PD diagnosis was made according to UK Brain Bank criteria by a fellowship-trained movement disorder neurologist. All patients were referred for neuropsychological evaluation for concerns of cognitive deficits as noted by the patient, informant, or referring neurologist. The VRFCAT-SL was administered to all patients seen during this time period (i.e., none were excluded from analysis). Administration was paused after these 30 assessments to perform these analyses related to quality assurance. One participant was unable to complete the VRFCAT-SL due to visual impairment (advanced age-related macular degeneration); this individual was also not able to complete other tests with small visual stimuli, the PD-CFRS, and other self-report inventories.

The first author (THT) performed all neuropsychological evaluations and administered the VRFCAT-SL. A core battery of tests was administered that included: Mattis Dementia Rating Scale (DRS), Brief Visuospatial Memory Test [27] (BVMT-R, Total Learning and Delayed Recall), Hopkins Verbal Learning Test [28] (HVLT-R, Total Learning and Delayed Recall), Judgment of Line Orientation (JOLO) [29], Neuropsychological Assessment Battery (NAB) [30]: Numbers & Letters A (Time and Errors), Digits Forward and Digits Backward, and Naming subtest; Hayling Sentence Completion Test [31] (Direct Time, Inhibition Time, Inhibition Errors), Controlled Oral Word Association Test (FAS and Animals) and Trail Making Test (Trails A, Trails B) from Halstead-Reitan Battery [32]. Other tests were administered as indicated. The Geriatric Depression Scale (GDS) [33] and Geriatric Anxiety Inventory (GAI) [34] were also administered. Raw scores were corrected for demographics according to test manuals (BVMT, HVLT, and Hayling corrected for age; NAB corrected for age and education; COWAT and Trails for age, gender, education, and ethnicity) and converted to scaled scores. Diagnosis of cognitive status was based on clinical history and performance on standardized neuropsychological testing in keeping with Movement Disorders Society Task Force Level II criteria for PD-MCI [35]. Specifically, the presence of two or more scores within one domain falling more than 1.5 SD below normative expectations were interpreted to reflect MCI.

All standardized testing was completed first, followed by the PD-CFRS, and then the VRFCAT-SL. Performance on the VRFCAT-SL was interpreted qualitatively with respect to implications for functional independence but not was not considered for diagnosis. For all analyses, VRFCAT-SL performances were corrected for age, gender, and education using normative data from the test publisher. For a subset of the sample (n = 20), the Movement Disorders Society’s version of the Unified Parkinson’s disease rating scale (MDS-UPDRS) [36] Part III ratings made by the referring movement disorder neurologist within 3 months of assessment while subjectively “on” medication were available.

Measures of central tendency were examined for VRFCAT-SL outcomes. Corrected scores were strongly skewed (Time = –0.968, Errors = –1.564, Forced Progressions = –2.921) and did not meet assumptions of normal distribution (see Supplementary Figure 1). Notably, only 4 patients had any forced progressions, and just one patient had 2 forced progressions. Accordingly, non-parametric tests were used for all analyses (i.e., Spearman-Brown for correlations and Mann-Whitney U-tests for group comparisons).

To examine association between VRFCAT-SL performance and overall cognition, a global cognitive index for each participant was derived based on the average rank (higher = better) for all standardized neuropsychological tests excluding the DRS-2. Similarly, a global score on the VRFCAT-SL was derived based on the average rank for its primary outcome measures (time, errors, and forced progressions). In exploring associations between VRFCAT-SL measures and neuropsychological test performances, the Bonferroni method was used to control for increased Type-I error rate associated with multiple comparisons (i.e., 17 measures, p < 0.002). SPSS version 25 ® was used for all statistical analyses.

RESULTS

Measures of central tendency for demographic, clinical characteristics, and test performances are provided for the entire sample, and separately by cognitive status group in Table 1. Boxplots illustrating ranges for VRFCAT-SL outcome measures relative to standardized neuropsychological test performances are provided in Supplementary Figure 2. For the overall sample, the average completion time for the VRFCAT-SL was 791 seconds (SD = 242) or about 13 minutes, with a minimum completion time of 504 seconds, and maximum completion time of 1461 seconds.

Association with standardized neuropsychological tests

Spearman-brown correlation analysis assessed relationship between overall VRFCAT performance measure (i.e., corrected Time, Errors, and Forced Progressions) and standardized neuropsychological testing. As shown in Fig. 1, there was a modest association between overall ranks, rho = 0.404, p = 0.037. The relationship was not driven by outliers. Overall standardized neuropsychological test performance was associated VRFCAT-SL time, rho = 0.532, p = 0.004, but not Errors or Forced Progressions.

Fig. 1

Association between VRFCAT-SL performance and standardized neuropsychological test performance.

Relationships between VRFCAT-SL outcome measures and primary neuropsychological outcome measures were also explored. As shown in Table 2, T-score for VRFCAT-SL Time was positively associated with T-scores for NAB Numbers & Letters A Time, rho = 0.457, p = 0.043, Trails B Time, rho = 0.472, p = 0.013, and BVMT-R Delayed Recall, rho = 0.444, p = 0.02. The T-Score for VRFCAT-SL Errors was inversely correlated with Inhibition Errors (raw) on the Hayling, rho = –0.561, p = 0.005, indicating greater errors on the VRFCAT-SL were associated with increased errors on the Haylling. T-score for VRFCAT-SL forced progressions was positively correlated with T-score for NAB Naming, rho = 0.499, p = 0.013. All correlations were thus in the expected direction that improved VRFCAT-SL performance was associated with improved neuropsychological test performance. No other correlations were significant at the p < 0.05 level, and none of the associations exceeded Bonferroni correction for multiple comparison (i.e., 17 measures, p < 0.002). Raw total score from the DRS-2 was not significantly associated with any VRFCAT-SL measure.

Table 2

Spearman-Brown correlations (p value, n) between demographically-corrected scores on standardized neuropsychological tests and VRFCAT-SL measures

	VRFCAT-SL Time	VRFCAT-SL Errors	VRFCAT-SL Forced Progressions
BVMT Total Learning	0.345	0.152	–0.095
	0.078	0.449	0.637
	27	27	27
BVMT Delayed Recall	0.444*	0.135	–0.105
	0.02	0.503	0.602
	27	27	27
HVLT-R Total Learning	0.26	0.258	0.143
	0.199	0.203	0.485
	26	26	26
HVLT-R Delayed Recall	0.206	0.195	–0.115
	0.314	0.339	0.575
	26	26	26
NAB Digits Fwd	0.192	0.266	0.047
	0.339	0.179	0.816
	27	27	27
NAB Digits Bkwd	0.237	–0.169	0.178
	0.235	0.4	0.375
	27	27	27
NAB Numbers &Letters A Time	0.457*	–0.001	0.264
	0.043	0.997	0.26
	20	20	20
NAB Numbers &Letters A Errors	0.279	0.151	0.013
	0.233	0.524	0.957
	20	20	20
NAB Naming	–0.06	0.201	0.499*
	0.781	0.346	0.013
	24	24	24
NAB Figure Copy	–0.224	0.115	–0.019
	0.316	0.611	0.933
	22	22	22
FAS	0.245	–0.133	0.218
	0.219	0.508	0.274
	27	27	27
Animals	0.224	0.321	0.072
	0.26	0.103	0.722
	27	27	27
Trails A	0.325	–0.26	0.09
	0.098	0.19	0.654
	27	27	27
Trails B	0.472*	–0.23	0.063
	0.013	0.249	0.755
	27	27	27
Hayling Initiation	–0.202	–0.168	0.109
	0.356	0.443	0.62
	23	23	23
Hayling Inhibition Time	–0.026	–0.158	0.181
	0.907	0.473	0.407
	23	23	23
Hayling Inhibition Errors	0.096	0.561 **	–0.09
	0.663	0.005	0.683
	23	23	23

DISCUSSION

Current results support use of the VRFCAT-SL for providing an efficient global assessment of functional cognition in PD patients with subjective cognitive decline. A modest correlation was observed between composite ranks for overall VRFCAT-SL and standardized neuropsychological measures. VRFCAT-SL performance measures also discriminated patients meeting MDS Task Force Level II criteria for PD-MCI from PD patients with normal cognition. Importantly, there were no indications that motor impairment significantly impacts task performance. Taken together, our findings using the VRFCAT-SL are consistent with those previously reported using other functional capacity tools. As with the CTPA, the VRFCAT-SL identified mild cognitive impairment in PD [22]. This study also replicated findings using the UPSA, which not only showed that performances differed between PD normal and PD-MCI, but those performances were associated with cognition independent of motor symptoms [21]. However, the average completion time of about 13 minutes for the VRFCAT-SL was found to be much lower than reported times for the UPSA (30 minutes), CTPA (40 minutes), and ECB (60 minutes) [17]. Thus, in addition to tablet-based administration and automated scoring, reduced time demands associated with the VRFCAT-SL facilitates inclusion in more comprehensive clinical and research batteries.

Findings from this study did not support the use of patient’s self-report on the PD-CFRS for assessing impact of cognitive deficits on instrumental ADLs. While Kulisevsky et al. suggest use of the PD-CFRS for self-report [26], in our sample PD-CFRS ratings were not associated with VRFCAT-SL performance or standardized neuropsychological measures and did not differ between cognitive diagnosis groups. Rather than reflecting cognition and motor function, PD-CFRS ratings in this sample appeared to capture anxiety and depressive mood symptomatology. This finding is in keeping with a number of prior studies describing anosognosia for both cognitive and motor disturbances in PD [14–16].

One limitation of the VRFCAT-SL with respect to providing a comprehensive functional capacity index in PD is the lack of a medication management task. Most patients with progressed PD follow a complex regimen involving variable doses of one or more dopamine replacement therapies throughout the day, as well as medications that might be taken for non-motor symptoms and other health conditions. The ability to reliably take medications as prescribed has presents a significant cognitive challenge and obvious relevance to overall functioning. Lack of a medication condition is also a weakness of the UPSA and CTPA. The ECB includes several items that require participants to remember information contained on a pill bottle and answer multiple choice questions but does not measure ability to actually follow a medication schedule. As such, this would appear to be more of a test of attention, learning, and memory with medication-themed stimuli, rather than a medication adherence task, per se. The Medication Management Abilities Assessment (MMAA) [37] was specifically designed to assess to evaluate this domain. The MMAA requires examinees to learn instructions for taking 4 different medications, retain this information over a one-hour delay, and properly distribute the correct number of pills along with instructions (e.g., with or without food) to the rater. In a study by Pirogovsky et al., PD patients meeting criteria for mild cognitive impairment based on standardized neuropsychological evaluation performed worse than those with normal cognition and a healthy control group [10]. The ecological validity of the VRFCAT-SL in PD might therefore be improved if medication adherence were assessed, perhaps by requiring examinees to observe the passing of time and take virtual pills as directed within a certain time window on several occasions throughout the task.

A methodological limitation of this study is that the provider (THT) performed neuropsychological testing, administered the VRFCAT-SL, and interpreted test performances. While cognitive status was based on clinical history and standardized neuropsychological testing, the possibility of diagnostic bias from observing VRFCAT-SL performance must be considered with respect to sensitivity and specificity estimates for discriminating PD-MCI. This study also included a relatively small sample of PD patients, and only a subset of participants had MDS-UDPRS Part III scores thus limiting generalizability. Future studies with neuropsychological evaluation independent of VRFCAT-SL administration are therefore recommended.

In the course of this study, no patients were found to meet MDS Task Force criteria for dementia in PD. This is not particularly surprising given that referring Movement Disorder neurologists generally request neuropsychological testing to identify mild cognitive impairment, while diagnosis of dementia is based on more obvious deficits and loss of independence. Future studies are thus needed to determine whether this can be used effectively in PD dementia populations.

Taken together, the results from this retrospective chart review provide provisional support for the VRFCAT-SL as an efficient tool for obtaining a global estimate of cognitive functioning in moderately advanced PD. Compared to other standardized functional capacity tools that have been used in PD, the tablet-based VRFCAT-SL eliminates the need for manipulatives, reduces administration and scoring complexities, and appears to be less burdensome with respect to time. Further investigation is required to evaluate test-retest reliability and determine whether it provides an objective measure of functional cognitive capacity across the full spectrum of disease progression.