Cognitive Performance and Learning Parameters Predict Response to Working Memory Training in Parkinson’s Disease

INTRODUCTION

Cognitive decline is a common, debilitating non-motor symptom in patients with Parkinson’s disease (PD), which is the second most frequent neurodegenerative disorder of older age [1]. Working memory (WM), executive functions, and attention are frequently impaired cognitive domains across various disease stages [2–4]. WM training (WMT) is the most empirically investigated cognitive training approach across the lifespan [5] due to the strong association of WM with fluid intelligence [6] and daily functioning [7]. WMT delivers reliable medium to large direct training effects and elicits small to medium near-transfer effects [5, 8, 9]. However, positive far-transfer effects by WMT are strongly debated nowadays as considerable heterogeneity in training effects suggests that WMT may not be equally efficient in all participants [8].

Recent randomized controlled trials (RCT) investigating WMT in PD [10, 11], addressed heterogeneity of WMT response by identifying potential predictors of the treatment response including, e.g., cognitive performance at baseline, age, and fluid intelligence. However, large proportions of variance in training response remain unexplained when using traditional statistical methods [10, 12]. In the context of recent machine learning studies, learning performance during cognitive training has recently been used as a behavioral proxy in combination with neuroimaging to monitor response to cognitive training in psychosis [13–15]. In healthy aging, higher gains in tasks trained during WMT are associated with higher transfer effects in both near- and far-transfer outcomes [16–19], yet training recommendations to patients with PD lack an individual signature needed to maximize patients’ benefit from WMT. The informed identification of patients with a positive response to WMT may allow early, personalized interventions (‘precision medicine’ instead of ‘one-size-fits-all’) to delay the onset and progression of cognitive decline in the domain of WM in PD [20].

However, it remains unclear to what extent demographic and clinical characteristics, cognitive performance at baseline, and training-inherent learning performance are useful in predicting WMT response in patients with PD. In this study we aim 1) to elucidate the predictive pattern joining data from several domains (demographic combined with clinical variables, cognitive data, and training-inherent learning performance) using a random forest regression algorithm in a sample of PD patients [11] and 2) to compare predictiveness of single and combined data domains to identify the most informative combination of features. The latter may help to pave a more efficient path to the real-world therapeutic setting, as the availability and accessibility of information from these data domains also highly varies between data domains.

METHODS

2.4Learning, neuropsychological, and clinical parameters

Training-inherent learning performance was tracked via NeuroNation for each of the five trained tasks per training day. For each task, a performance score was given, which represented the sum of all correct answers given within the exercise adjusted for the initial difficulty of the exercise during the training session (the higher the initial difficulty, the more points are given). For the present analyses, we used the min-max normalized scores of the NeuroNation output, which allow to compare task performance across the different WM exercises and training sessions, as the initial performance scores are transformed to a similar range across exercises. Three equally weighted composite scores were built according to the previously described classification of WM paradigms into simple span, complex span, and n-back tasks (see Supplementary Material 1). As applied in previous research [16], two learning parameters per composite score of trained task performance were extracted for each participant based on a linear model fitted to the trained task performance: the ‘intercept’ corresponding to the initial performance level in the first training session and the ‘slope’ corresponding to the change in performance over time (Fig. 1).

Fig. 1

Linear models fitted to learning performance of each participant for each composite score. To extract the learning parameters for each individual, we fitted a linear model to the learning performance curve of each participant (different colors) for training sessions 1, 4, 6, 9, 11, 14, 16, 19. Individual slope and intercept parameters were used as predictors in the random forest regression models.

Outcome raters were blinded for group allocation, patients were not. The WM domain as a near-transfer measure was assessed by several single tests with parallel test forms, if available, including both paper-pencil assessments and computer tests. Two WM composite scores were computed as the average of the corresponding equally weighted single test z-scores based on published normative data of age-, sex-, and education-matched healthy controls. Verbal WM as the primary outcome of the present analyses was assessed by the digit span forward and backward task (Wechsler Memory Scale revised) [28], and the n-back verbal task (Wiener Testsystem) [29]. For details on further cognitive composite scores (non-verbal WM, executive functions, attention, verbal memory, visuo-cognition), which were used as predictors in the present analyses, see Ophey et al. [11]. The reported composite scores can be interpreted as regular z-scores with M = 0 and SD = 1, i.e., a score of 0 represents average performance, negative scores represent worse performance and positive scores represent better performance than the normative population—independent of age, sex, and education, respectively.

The Levodopa Equivalent Daily Dose (LEDD) [30] and years of PD drug therapy, which can be regarded as a proxy for disease duration, were recorded. During all assessments, patients were on their regular dopaminergic PD medication, and the majority of assessments was conducted in the patients’ medication “ON” state. Motor impairment was assessed with the UPDRS-III [31] and the H&Y [25] scale. Furthermore, patients’ questionnaires examining depressive symptoms (GDS) [24] and self-efficacy expectancy (SWE) (German version [32], English version [33]) were considered in the set of predictors (Table 1).

Predictive modeling

We built four single models based on single or combined cognitive, learning, clinical, and demographic data domains to evaluate their individual contribution to the prediction of verbal WM at POST and FU. The ‘cog’ model consists of variables assessing baseline performance in verbal and non-verbal WM, executive functions, attention, verbal memory, visuo-cognition, fluid intelligence, and global cognition. The ‘learning’ model includes slope and intercept parameters for each composite score (simple span, complex span, n-back). The ‘cog/learning’ model describes the combination of the ‘cog’ and ‘learning’ variables and the ‘all’ model additionally includes clinical and demographic information (Table 1). The ‘cog’ model could be estimated based on a standard face-to-face cognitive assessment performed by a neuropsychologist and the ‘learning’ model could be estimated entirely remotely during the WMT. The ‘all’ model could be estimated based on an elaborated face-to-face clinical and cognitive assessment performed by physicians and neuropsychologists additionally including data from the remote WMT.

We used random forest regression as implemented in the R package ‘ranger’ [34] (version 0.13.1) to build the prediction models and to evaluate their performance. In the random forest regression, decision trees are generated based on random subsets of observations in the data set (= bagging). In each decision tree the algorithm repetitively splits the high-dimensional data set at each node based on their most predictive variable within a random subset of the predictors until a certain criterion is reached. A prediction is achieved by integrating the individual predictions of each decision tree. We specified the number of trees generated (N = 1000; sample fraction = 1; sampling with replacement) and the number of possible splitting variables at each node (mtry = square root of number of predictor variables; minimum node size = 5) in accordance with previous recommendations [35]. Predictors were inspected for collinearity to reduce feature space and to avoid biases in model interpretation (Supplementary Figure 2) prior to random forest regression. We excluded the variables ‘years since PD diagnosis’ and ‘intercept simple span’ due to high collinearity to other predictors (Spearman’s rho > 0.80).

Model performance was quantified by root mean square error (RMSE) prediction error, which was calculated by the root of the squared average difference of predicted value based on the model and the actual value. To assess the generalizability of the model, we further calculated the out-of-bag (OOB) prediction error OOB, which represents the performance of the random forest model with respect to observations unseen during the model generation process (for details, see Supplementary Material 2). For both model performance metrics, RMSE and OOB, lower scores represent a better fit of a given model to a dataset.

Predictors were ranked by importance as quantified by the reduction in variance when this predictor was used for splitting the data set [34]. Higher reduction in variance suggests higher importance for regression. The feature importance was estimated using the impurity approach implemented in ‘ranger’ [34]. To assess the stability of our prediction models we used a resampling approach (N_iteration =1000; without replacement) selecting a random subset of 80% (N = 30) of the total sample per iteration. Missing data across all predictor variables (∼0.5%) was imputed within each subset of the resampling scheme by median as it shows less bias for extreme values.

Finally, we calculated pairwise permutation tests to compare performance between the different predictor models generated by the resampling using the ‘rcompanion’-package [36] in R and corrected p-values for multiple comparisons using Bonferroni correction [37]. Cohen’s d was reported as the effect size of the pairwise permutation tests indicating small (d≥0.2), moderate (d≥0.5), or strong (d≥0.8) effects. The random forest regression analysis procedure is visualized in Fig. 2.

Fig. 2

Random forest regression analysis procedure. Random forest regression models were generated embedded in a resampling scheme (left). For each iteration of the resampling that contained 80% of the total sample, variables were median imputed prior to running the random forest algorithm. Each generated model was evaluated based on two model performance indicators and with respect to their feature importance using the ‘impurity’ measure of the ranger package (right).

RESULTS

Details on prior analyses regarding the general efficacy, as well as neural correlates of WM and WMT are reported in Ophey et al. [11], Giehl et al. [38], and Giehl et al. [39]. Table 2 presents the descriptive data for the cognitive domains across baseline, POST, and 3-month FU assessments for the WMT group and the CG. As reported in Ophey et al. [11], repeated-measures, linear mixed-effects models revealed positive training effects for the WMT group compared to the waiting list CG in verbal WM with a small relative effect size 0.39 (95%-Confidence Interval CI = [0.05;0.76]) for the 3-months follow-up only. No other reliable training effects on cognitive and clinical variables were found for either point of time.

Predictive modeling

The ‘all’ model predicted verbal WM with the lowest RMSE prediction error (‘all’ vs. ‘cog/learning’: p < 0.001) followed by the combination of the cognitive and the ‘learning’ model (‘cog/learning’ vs. ‘cog’: p < 0.001), the cognitive solely model (‘cog’ vs. ‘learning’: p < 0.001) and the ‘learning’ solely model (Fig. 3; Table 3). This result is common to both timepoints (POST, 3-month FU). The RMSE prediction error of the ‘all’ model was 0.184 (95% -CI=[0.184;0.185]) at POST and 0.216 (95% -CI=[0.215;0.217]) at FU. Results of the OOB prediction error are presented in the Supplementary Material 2 and visualized in Supplementary Figure 3. RMSE prediction errors of the four evaluated models at POST and FU, as well as predictor ranks of variables in the ‘all’ model including clinical, demographic, cognitive, and learning variables are visualized in Fig. 3.

Fig. 3

Random forest regression model prediction for verbal working memory at POST and 3-month FU. We used random forest regression to evaluate the predictive performance of different subsets of predictor variables: cognitive (‘cog’), ‘learning’, clinical, and demographic (‘all’). The graph shows the performance of the models generated through resampling (N = 1000) measured by root mean square error (RMSE) and the feature importance (impurity approach) of the ‘all‘ model for the prediction of verbal working memory at timepoints (A) POST and (B) 3-month FU. For both time points each Bonferroni-corrected pairwise model comparison was significant (p_bonferroni <0.001, for effect sizes, see Table 3). PRE, baseline; WM, working memory.

Table 3

Comparison of model performance (RMSE) by pairwise permutation tests

	Permutation Test				Effect Size
	POST		3-month FU		POST	3-month FU
	statistic	p*	statistic	p*	Cohen’s d	Cohen’s d
’all’ vs. ‘cog/learning’	–5.68	<0.001	–8.49	<0.001	0.26	0.39
’all’ vs. ‘cog’	–12.13	<0.001	–29.43	<0.001	0.56	1.75
’all’ vs. ‘learning’	–38.37	<0.001	–30.82	<0.001	3.34	1.90
’cog/learning’ vs. ‘cog’	–6.33	<0.001	–24.99	<0.001	0.29	1.35
’cog/learning’ vs. ‘learning’	–37.00	<0.001	–27.02	<0.001	2.94	1.52
’cog’ vs. ‘learning’	–36.07	<0.001	–4.78	<0.001	2.73	0.21

*Bonferroni corrected.

The ten most important predictors for POST WM comprised four cognitive baseline predictors (verbal WM, non-verbal WM, executive functions, fluid intelligence, rank 1-4), three learning parameters (intercept n-back, intercept complex span, slope simple span, rank 5-6, 9), two clinical variables (dopaminergic medication, drug therapy years, rank 7 + 10), and one demographic characteristic (age, rank 8). The ten most important predictors for FU WM included three cognitive baseline variables (verbal WM, non-verbal WM, attention, rank 1 + 3+6), the same three learning parameters as for POST WM (intercept n-back, intercept complex span, slope simple span, rank 2 + 4+8), three clinical variables (dopaminergic medication, drug therapy years, motor impairment, rank 7 + 9-10), and one demographic characteristic (education, rank 5).

At POST and FU effect sizes between the ‘all’ model and the ‘cog/learning’ model were small (Cohen’s ds≤0.39), and medium to large in comparison to the ‘cog’ and ‘learning’ model (Cohen’s ds≥0.56). The combined ‘cog/learning’ model significantly outperformed the ‘cog’ model with a small effect size at POST-test (Cohen’s d = 0.29), however, with a large effect size at FU (Cohen’s d = 1.35).

DISCUSSION

The present study generated and compared multivariate models that aim to predict responsiveness to WMT in patients with PD using learning parameters in addition to cognitive, clinical, and demographic characteristics. General WMT effects in our previous study [11] were limited to the domain of verbal WM, while no other cognitive domains improved in the WMT group relative to the passive waiting list CG. For this reason, in the current study we applied the random forest model to predict response to WMT in patients with PD. The model using the entire set of prognostic variables reached the best performance among other models using the cognitive and/or learning subset only. While the learning model could not outperform the combined model including all the variables, the training-inherent learning characteristics further refined the performance of the cognitive model.

Cognitive variables at baseline, especially the verbal WM score, had the highest feature importance among the investigated set of predictors for verbal WM immediate and 3-month FU intervention scores. Moreover, executive functions at baseline and fluid intelligence followed in the hierarchy of the feature importance at immediate FU, delineating how general cognitive ability complements task-specific requirements represented by verbal WM [6]. A recent large-scale analysis of cognitive training data from more than 36,000 individuals revealed a latent general ability factor determining the improvement in later stages of training, whereas several task-specific factors determined initial training performance [40]. Notably, only the WM composites stay within the most important feature ranks at 3-month FU, possibly indicating a less strong predictiveness of general cognitive abilities at 3-month FU.

In our analyses, we evaluated the single feature importance of our investigated learning parameters, i.e., intercepts and slopes of three composite scores summarizing performance across training sessions of nine training tasks. Our results reveal higher importance of the composite intercepts than the corresponding slopes for both immediate and 3-month FU WM. The intercepts may represent a general proxy for WM performance at baseline, which is why their feature importance appears comparable to that of cognitive baseline scores. However, the intercept parameters do not contain information about the training improvement over time, which is inherently represented in the slope parameters. Our results indicate that baseline performance in either standard cognitive assessments or training-inherent tasks can strongly predict immediate and 3-month FU performance. In turn, it could be derived that the improvement in performance of the training tasks itself was less heterogeneous and therefore less predictive across participants than their individual baseline performance.

Importantly, the predictiveness of learning parameters seems to change over time. For immediate and 3-month FU verbal WM performance, models containing the entire set of behavioral predictors yielded the best model performance, followed by the model combining cognitive baseline performance and learning parameters. While the model using cognitive parameters alone and the model combining cognitive and learning parameters show almost comparable predictive performance immediately after the intervention, the combination of cognitive and learning parameters largely outperformed the model using cognitive baseline parameters alone at 3-month FU. This indicates a stronger predictive importance of learning parameters for long-term WMT outcome compared to immediate WMT outcome, which may indicate that task-general processes acquired during WMT influence performance beyond the training period [40]. The increased predictive value of the learning parameters at FU is also reflected in higher ranks of feature importance. This is in accordance with findings of previous correlational WMT analyses [12] that revealed how individual differences in demographic, cognitive, and clinical variables become less important from immediate to 3-month FU examination. The present study including learning parameters therefore may contribute to closing the gap in understanding the factors influencing response to WMT in patients with PD by revealing associative relationships between individual patient characteristics and cognitive performance. Furthermore, in healthy aging, Bürki et al. [16] showed that near- and far-transfer outcomes were linked to both learning intercepts and slope beyond common proxies for general functioning such as fluid intelligence and the potential to engage in cognitive plasticity, which is potentially reflected in age.

Among clinical variables, the dopaminergic medication (measured as the LEDD [30]), years of drug therapy, and PD motor impairment (measured with the UPDRS-III [31]) were among the most important features in the model including the entire set of features. Cognitive functioning in general but also the ability to engage in cognitive plasticity is to some extent determined or even facilitated by centrally active drugs including dopamine replacement therapy in PD [41, 42]. As stated in the Dopamine Overdose Hypothesis, both dopamine depletion as well as dopamine overdoses might lead to adverse effects in cognitive tasks, resulting in an inverted U-shaped relationship between dopamine levels and cognitive performance [43, 44]. Following, dopaminergic treatment may foster dopamine depleted neural circuits but at the same time overdose relatively intact circuits, resulting in differential effects of dopaminergic supplementation on different cognitive domains relying on different neural circuits [4, 45]. Furthermore, along with the progression of neurodegeneration in PD [44], the optimal level of dopamine replacement therapy for cognitive functions may change over the course of the disease, as it does for motor and other non-motor symptoms [46]. The effects of dopaminergic supplementation on learning and the responsiveness to cognitive interventions in both healthy older adults and patients with PD is heterogeneous as reviewed in Lebedev et al. [45] and requires further investigation. The majority of studies on Levodopa and training responsiveness in PD, however, reveals a positive relationship between LEDD and training outcome.

As the random forest regression does not allow to evaluate the direction of the relationship between predictor variables and outcome, e.g., if more or less years of education are associated with higher WM improvement, we analyzed the bivariate associations between predictors and outcomes, which should, however, be interpreted cautiously (Supplementary Figures 4-6). The WMT literature suggests two mechanisms leading to training effects, which can be discussed in terms of the direction of the relationship between predictor variables and training outcome: on the one hand, compensatory mechanisms [47, 48], i.e., those individuals having more room for improvement benefit most, and on the other hand processes of magnification [47, 48] yielding larger improvement for those with better prerequisites in terms of cognitive and neurobiological reserve [10, 12, 49].

The main limitation of the present analyses is the lack of a validation cohort, which is why the results should be treated cautiously despite the methodologically strong and novel analytical approach. Due to the low sample size we were not able to evaluate our models within a cross-validation scheme as previously recommended [50, 51]. However, stability of the model performances and the feature importance of the investigated predictors was assessed within a resampling approach which decreases the likelihood that results were obtained by chance. Note, that one cannot readily account for model complexity in (non-likelihood-based) random forest regression analyses, as the number of input parameters does not reflect the effective degrees of freedom [52]. This phenomenon is called “paradox of ensembles” by Elder [53]. Nevertheless, the advantage of random forest regression is that is does not make any prior assumptions about the association between predictor variables and response variable (e.g., normal distribution) [35], and it is more suitable than traditional likelihood based approaches such as multiple linear regression when the number of predictors is large [54]. The second limitation refers to the composition of the applied cognitive test battery, including tests that might not have been sensitive enough to monitor different levels of improvements appropriately (e.g., ceiling effects) [55]. Third, the 3-month FU period should be extended by longer FU periods to evaluate predictors of long-term preventive benefits of WMT. Another limitation may be that patients were instructed not to train between POST and 3-month FU; however, it might also be possible, that our participants engaged in further cognitive training activities despite our recommendation.

Finally, future studies with PD patients may want to apply our model to other cohorts and test its reproducibility and generalizability. The present cohort of patients with PD without cognitive performance was not only well-specified regarding its cognitive status, but rather homogeneous regarding motor PD severity. Future research definitely needs to strike a better balance between samples recruited according to specific criteria and the generalizability to broader patient population with PD. Future studies might also apply and evaluate prediction models with predictors from different domains in the context of far-transfer training effects. The present analyses focus on the prediction of verbal WM as a near-transfer measure following WMT. However, in the context of single-domain cognitive intervention approaches such as WMT, those far-transfer effects are of special interest, as training effects in WM are assumed to transfer to other cognitive domains and potentially clinical outcomes [8].

The consideration of WMT-inherent learning parameters allows clinicians and researchers to utilize the information of the WMT-inherent learning dynamics rather than focusing on a single snapshot of cognitive performance obtained from standard cognitive testing [40]. Especially the latter might be biased in patients with PD, who frequently suffer from both motor and—in this context even more important—non-motor fluctuations [56]. Even though non-motor fluctuations may influence training-inherent learning performance as well, their influence will be more balanced across time and patients with PD might have chosen ON phases to train resulting in overall more reliable learning parameters. Furthermore, utilizing the information on training-inherent learning parameters directly extracted from a computerized, remotely delivered WMT, provides a low-threshold possibility to extract a proxy of baseline levels of cognitive performance. Prospectively, the collection of information on potential predictors of training outcome could potentially be performed without resource-intensive face-to-face cognitive assessments reaching even people in more rural areas with less health-care infrastructure. Future studies should evaluate how many training sessions are needed to extract reliable and valid proxies for the proposed learning parameters.