Using Machine Learning to Predict Dementia from Neuropsychiatric Symptom and Neuroimaging Data

INTRODUCTION

Machine learning (ML) identifies complex patterns in high-dimensional data, which are then used to make clinical predictions in new datasets. In dementia, the focus of ML models has been neuroimaging and cognition-centric [1]. Well-established imaging biomarkers of Alzheimer’s disease (AD) include hippocampal and entorhinal cortex volume reduction, and basal forebrain nuclei loss [2]. Additionally, patterns of atrophy in regions including the paralimbic areas, parietal association areas, lateral temporal, temporoparietal, and frontal cortices have been identified as imaging markers of AD-related neurodegeneration that can be observed in individuals prior to onset of clinical symptoms [2–4]. More specific to ML approaches, Salvatore et al. used ML to identify morphological abnormalities in the hippocampus, entorhinal cortex, basal ganglia, precuneus, and the cerebellum as important in pre-clinical phases of AD [5]. Increasingly, non-cognitive signs and symptoms are considered potential early indicators of neurodegenerative disease [6–8]. Recent evidence suggests that assessment of neuropsychiatric symptoms (NPS) may be promising to identify a high-risk population for dementia [8]. Mild behavioral impairment (MBI) is a validated neurobehavioral syndrome that describes later-life onset of sustained NPS as an at-risk state for incident cognitive decline and dementia [9]. MBI has been suggested to be the index manifestation of dementia, associated with biomarker positivity, genetic links to Alzheimer’s disease (AD), and may be a novel approach for early identification of neurocognitive disorders. Conventional NPS assessments have not shown predictive value in previous ML models. For example, Korolev et al. used more than 750 variables including clinical, MRI, and plasma biomarkers to predict progression to AD-dementia of individuals with mild cognitive impairment (MCI), identifying cognitive and functional parameters as most predictive [10]. In this case, NPS were measured conventionally with the neuropsychiatric inventory. We hypothesize that MBI scores, characterized by sustained NPS, will improve the prognostic accuracy of ML models. This novel approach to NPS characterization (e.g., emphasizing sustained symptoms), can complement the neuroimaging findings, which have been better explored for this purpose in the past. Developing better models can help in understanding the early manifestations of dementia, which can improve clinical care and assist with development of new interventions. Identifying the relative importance of NPS in predicting incident cognitive decline and dementia can guide clinicians to look for emergent and sustained NPS, supplementing clinical and imaging measures that are well known and routinely collected. Further, this approach to earlier detection of dementia risk can inform clinical trial methodology and screening [11]. Thus, the aims of this study were to: 1) identify the optimal feature subset required to predict follow-up diagnosis for a two-class experiment (normal cognition [NC] versus MCI/AD); and 2) identify the optimal feature subset required to predict follow-up diagnosis for a 3 class experiment (NC versus MCI versus AD).

MATERIAL AND METHODS

Participants

Participants were between 55–90 years of age, English or Spanish speakers, and accompanied by study partners who completed the Neuropsychiatric Inventory Questionnaire (NPI-Q) [12]. All participants provided informed consent to participate. Approval from an ethical standards committee to conduct this study was received at contributing ADNI sites.

For ADNI-1 participants with NC and MCI at baseline, we collected demographic, clinical, structural MRI, and NPI-Q data as well as final clinical diagnosis at the participant’s latest follow-up. All datasets were downloaded on or before December 10, 2018.

MBI diagnosis requires NPS to be emergent in later life and sustained for at least 6 months [9, 13]. The 6-month symptom duration was defined purposefully to exclude transient and reactive symptoms, to improve signal to noise ratio, and to increase diagnostic specificity of MBI compared to the conventional method of assessing NPS. However, ADNI uses the NPI-Q with a reference range of 1 month to capture NPS. The domains from NPI-Q were combined and categorized under MBI domains as follows: drive/motivation (NPI-Q apathy/indifference); emotion regulation (NPI-Q depression, anxiety, elation/euphoria); impulse control (NPI-Q agitation/aggression, irritability, aberrant motor behavior); social cognition (NPI-Q disinhibition); and thoughts/perception (NPI-Q delusions, hallucination) [14]. The MBI total score was calculated by combining the scores on the five MBI domains. To approximate MBI in the ADNI dataset, participants positive for NPS at two consecutive time points 6-months apart (i.e., MBI total score > 0) were classified as MBI-positive (MBI+). Those with no NPS were classified as MBI-negative (MBI-) for comparison. We further narrowed the sample into two non-dementia groups to include individuals with NC and those with MCI based on ADNI’s diagnostic eligibility criteria (For more information: http://adni.loni.usc.edu/wpcontent/uploads/2010/09/ADNI_GeneralProceduresManual.pdf).

Participants were excluded if: 1) NPI-Q assessment < 6 month apart; 2) baseline demographic/clinical data were > 2 months from NPI-Q exam date; 3) baseline structural MRI data were > 6 months from NPI-Q exam date; 4) follow-up diagnostic status was < 12 months from baseline diagnosis; 5) brain volume data were not processed using FreeSurfer version 4.3, to ensure consistency of the data; 6) UCSF core laboratory’s visual quality control identified overall segmentation failure following processing using FreeSurfer version 4.3. Figure 1 shows the step-by-step process for inclusion/exclusion. Participants included for the analysis were 102 NC (38 with MBI and 64 without MBI) and 238 MCI (219 with MBI and 19 without MBI).

Fig. 1

Data extraction protocol of participants included in the ML analysis.

RESULTS

Table 1 shows demographic characteristics and MBI scores of the sample. Across groups, there were no significant differences in age and education. However, there were significant differences between groups in the follow-up period. Individuals with baseline normal cognition were followed for a longer time (M = 68.63 months; SD = 43.46) compared to patients with baseline MCI (M = 53.19 months, SD = 34.72). Additionally, there were significant group differences in all MBI domains and MBI total score, with individuals with MCI scoring higher. Figure 2 shows the distribution of the two groups across MBI domains. At follow-up, there were 83 individuals with NC, 112 with MCI, and 145 with AD.

Fig. 2

Frequency distribution of MBI domains in individuals with normal cognition (NC, n = 102) and mild cognitive impairment (MCI, n = 238).

Table 2 shows the class balance (i.e., number of individuals for each diagnostic status at follow-up) and output metrics for the two classification experiments. For the three-class experiment, the reported measurements are an average of the metrics for each of the three classes (each one being a positive class in its own calculation). Combining the two types of features (clinical+brain morphological features) led to a better prediction model compared to using clinical or morphology features alone and thus are discussed in a greater detail below. For the combined approach, the mean area under the ROC curve (ROC-AUC) for experiment 1 and 2 was 0.86 (CI:0.853–0.867) and 0.73 (CI: 0.725–0.743), respectively.

The binary classification experiment (classifying participants as NC versus MCI/AD) required only two features, achieving a ROC-AUC of 0.86 (CI:0.853-0.867). On average, this model correctly classified 84.9% (CI: 84.4% – 85.4%) of the cases, with a sensitivity of 0.92 and a specificity of 0.63.

For the second ML experiment (classifying patients into three categories based on final diagnosis: NC, MCI, or AD), the optimal model required seven features achieving an overall discrimination (ROC-AUC) of 0.73 (CI: 0.725–0.743) and a mean accuracy of 58.8% (CI: 58.1% – 59.4%). With each class being a positive class in its own calculation, the mean group specific sensitivity and specificity metrics were as follows: NC (sensitivity = 0.87, specificity = 0.66), MCI (sensitivity = 0.81, specificity = 0.31), and AD (sensitivity = 0.69, specificity = 0.76).

Table 3 shows the features selected for the two experiments. In both, the MBI total scores and the left hippocampus volume were selected as the highest ranked features. However, the three-class experiment required additional features including scores on the MBI impulse dyscontrol domain, cortical thickness of the left entorhinal cortex, scores on the MBI emotion dysregulation domain, volume of the left entorhinal cortex, and cortical thickness of the left middle temporal gyrus.

DISCUSSION

ML methods can identify distinctive imaging and clinical features and patterns to make diagnostic predictions. Clinically, this approach can potentially assist in dementia prognostication for those presenting with cognitive complaints, as this is a common question clinicians are facing. NPS are common in prodromal dementia [20], can even present in preclinical dementia [5], and are clinically significant [21–24]. While several studies have used complex ML models to predict stages of dementia, NPS are often ignored as potentially predictive features [1, 10, 25]. In this study, the future diagnostic status of participants was predicted using clinical information, structural MRI, and MBI scores. The ML models combining clinical, MBI scores, and structural MRI features achieved the highest accuracy predicting diagnostic status at follow-up in comparison to the ML models using clinical or brain morphological features only.

In the two-class experiment, the ML model achieved a high accuracy (84.9%) and required only two features to distinguish normal from impaired (NC versus MCI/AD). The three-class experiment resulted in an accuracy of 58.8%, which despite being lower than the accuracy of the two-class experiment is still considerably better than chance level. A more detailed investigation of the results revealed that differentiating the MCI and AD group was the most challenging for this classifier. It is likely that in the ADNI-1 dataset, some individuals with MCI have patterns of structural atrophy similar to AD, suggesting that MCI may be more similar to AD than NC. This similarity between MCI and AD might also make it more challenging for the ML model to distinguish between the two groups, thus, requiring more features. Clinical experience mirrors this challenge in that the differentiation between MCI and dementia is based on functional decline, which can sometimes be difficult to identify or quantify, resulting in an overlap between groups. Thus, the results identifying features to distinguish normal from abnormal are more clinically meaningful, in comparison to those distinguishing between degrees of impairment (MCI versus AD). Combining different types of data could help to further improve the accuracy of the three-class model to differentiate between clinically relevant groups.

In this study cohort, the ML model required only 2–7 features to optimally classify individuals for the two classification tasks. MBI total scores had greater prognostic utility as quantified by the information gain metric compared to all other clinical or volumetric variables for predicting diagnostic status over time. In both classification experiments, the MBI total score was ranked as the most important variable for distinguishing the normal versus abnormal cognition group, as well as for classifying individuals into NC, MCI, and AD groups. Additionally, the two-class experiment included the volume of the left hippocampus as an important predictor. Of the seven features required for the three-class classification, four were imaging markers (left hippocampus volume, cortical thickness and volume of the entorhinal cortex, and cortical thickness of the left middle temporal gyrus) and three were NPS markers (MBI total score, impulse dyscontrol score, and emotional dysregulation score) emphasizing the importance of NPS in the AD process.

The majority of previously described ML models for classification of MCI and AD have focused on a single imaging modality, while only a few multimodal classification models have been developed to predict progression of the neurodegenerative disease [1, 10, 25, 26]. However, researchers have mostly narrowed ML studies to include clinical, cognitive, and structural or functional neuroimaging variables only. In a recent systematic review, it was highlighted that to improve the performance of ML models, additional types of data must be explored [27]. MBI is a novel way of capturing and presenting NPS data. Based on the results in this study, it is evident that including MBI in ML models has the potential to increase the prognostic utility and accuracy. In this work, we demonstrated that these symptoms are amongst the most important predictors, along with left hippocampus volume, cortical thickness and volume of the entorhinal cortex, and cortical thickness of the left middle temporal gyrus.

The volumetric findings are generally consistent with past literature in which one of the earliest imaging marker identified is atrophy in the medial temporal lobe, specifically in the hippocampus and entorhinal cortex, which previously also has been found to be predictive of progression from MCI to AD [1, 10, 28]. With a similar goal of building a clinically-translatable ML model, Grassi et al. focused on inexpensive measurements such as demographics, neuropsychological tests, cardiovascular risk indexes, and visual rating scale of brain atrophy to predict the likelihood of conversion to dementia in pre-MCI/MCI participants [29]. Their best model required 16 features out of a total of 34, which achieved a classification accuracy of 91.3%, with the cognitive test scores, diagnostic status, hippocampal and entorhinal atrophy being within the top 10 features [29]. While differences in sample, ML algorithm used, and features analyzed could explain the difference in performance accuracy when compared to our results, the brain regions identified were similar and thus they are potentially important predictors of cognitive decline and dementia.

Recent evidence also suggests that assessment of NPS such as apathy, mood disturbances, anxiety, agitation, disinhibition, and psychosis may provide a simple and efficient method to identify a high-risk population for dementia [30, 31]. Population-based [7, 32] as well as clinic-based cohort [33] studies provide consistent evidence that NPS in MCI are associated with higher risk for incident dementia, with an estimated annual rate of progression to dementia of 25% for MCI plus NPS in contrast to 10-15% per year for MCI alone [33, 34]. Similarly, NPS in older adults with NC confers a higher likelihood of progression to MCI and dementia [8]. However, investigating these behavioral symptoms in predictive models is underappreciated, in part due to the way NPS are historically measured. Our study highlights the importance of MBI total and domain specific MBI scores as important predictors of progression to worsened diagnosed status.

Our study differs from Korolev et al. who used total and item-scores from the NPI-Q in their prognostic model of progression from MCI to AD. With this approach, NPS were not identified as important features [10]. In our view, the most likely explanation is that by requiring sustained NPS, consistent with the definition of MBI, we likely reduced noise (i.e., false positives) introduced by transient and reactive neuropsychiatric symptoms. In a population-based study, the point prevalence of NPS at a single assessment was quite high at 34.1% within a sample encompassing individuals from normal aging to MCI [15]. However, validation studies of MBI using the full criteria (including the proper 6-month reference range, and using the appropriate MBI case ascertainment tool— the MBI checklist [13]) gave lower estimates of prevalence, 5.8% in subjective cognitive decline [35] and 14.2% in MCI [36], which are probably more specific for persistent changes related to neurodegeneration. This smaller and higher risk group offers an opportunity for more targeted and efficient use of imaging and biomarker studies to capture preclinical disease [37]. Thus, the MBI requirement for later life emergent and sustained NPS may better reflect true NPS that result from neurodegeneration and proteinopathies associated with dementia, removing symptoms that may be a result of non-dementia etiologies [38].

Additionally, beyond the total MBI score, the impulse dyscontrol and emotional dysregulation domains were also observed as important features in the prediction models developed in this work. The prevalence of agitation in MCI varies from 5–25% in population-based studies [39]. Agitation and irritability are associated with greater atrophy in frontal, insular, amygdala, cingulate, and hippocampal regions [40, 41], and with posterior cingulate hypometabolism in preclinical AD [42]. The overall prevalence of depression in MCI is 32% and more frequent in clinical (40%) versus community (25%) samples. Additionally, a recent scoping review has highlighted the importance of the natural history of emotional dysregulation symptoms (i.e., later life onset versus chronic and recurrent symptoms) in preclinical and prodromal dementia, emphasizing the prognostic utility of these symptoms when assessed and measured appropriately [38].

There are several strengths of the present study. A unique aspect is the use of individuals in the preclinical and prodromal phase of AD, i.e., individuals with NC and MCI baseline diagnostic status, to classify participants into the various diagnostic groups at follow-up visits. Additionally, by using the rigorous cross-validation approach, we attempted to avoid the issue of over-fitting the training data, thus, improving the reliability and generalizability of the results. The results of the cross-validation suggest that the ML models are quite robust and reliable as the confidence intervals are quite narrow and decrease the risk of the findings not being reproducible in external datasets or not being clinically applicable. Nevertheless, the developed ML models should be evaluated using prospectively collected independent data in the next step for further validation.