Utility of MemTrax and Machine Learning Modeling in Classification of Mild Cognitive Impairment

Study population

Between January 2018 and August 2019, cross-sectional research was completed on patients recruited from two hospitals in China. The administration of MemTrax [5] to individuals aged 21 years and over and the collection and analysis of those data were reviewed and approved by and administered in accord with the ethical standards of the Human Subject Protection Committee of Stanford University. MemTrax and all other testing for this overall study were performed according to the Helsinki declaration of 1975 and approved by the Institutional Review Board of the First Affiliated Hospital of Kunming Medical University in Kunming, Yunnan, China. Each user was provided an informed consent form to read/review and then voluntarily agree to participate.

Participants were recruited from the pool of outpatients in the neurology clinic at the Yanhua Hospital (YH sub-dataset) and the memory clinic at the First Affiliated Hospital of Kunming Medical University (XL sub-dataset) in Beijing, China. Participants were also recruited from neurology (XL sub-dataset) and internal medicine (KM sub-dataset) inpatients at the First Affiliated Hospital of Kunming Medical University. Inclusion criteria included 1) men and women at least 21 years old, 2) ability to speak Chinese (Mandarin), and 3) ability to understand verbal and written directions. Exclusion criteria were vision and motor impairments preventing participants from completing the MemTrax test, as well the inability to understand the specific test instructions.

Chinese version of MemTrax

The online MemTrax test platform was translated into Chinese (URL: https://www.memtrax.com.cn) and further adapted to be utilized through WeChat (Shenzhen Tencent Computer Systems Co. LTD., Shenzhen, Guangdong, China) for self-administration. Data were stored on a cloud server (Ali Cloud) located in China and licensed from Alibaba (Alibaba Technology Co. Ltd., Hangzhou, Zhejiang, China) by SJN Biomed LTD (Kunming, Yunnan, China). Specific details on MemTrax and test validity criteria used here have been described previously [6]. The test was provided at no charge to the patients.

Study procedures

For the inpatients and outpatients, a general paper questionnaire for collecting demographic and personal information such as age, sex, years of education, occupation, living alone or with family, and medical history was administered by a member of the study team. Following completion of the questionnaire, the MoCA [12] and MemTrax tests were administered (MoCA first) with no more than 20 minutes between tests. MemTrax percent correct (MTx-% C), mean response time (MTx-RT), and date and time of the testing were recorded on paper by a member of the study team for each participant tested. The completed questionnaire and the results of the MoCA were uploaded into an Excel spreadsheet by the researcher who administered the tests and verified by a colleague before the Excel files were saved for analyses.

MemTrax test

The MemTrax online test included 50 images (25 unique and 25 repeats; 5 sets of 5 images of common scenes or objects) shown in a specific pseudo-random order. The participant would (per instructions) touch the Start button on the screen to commence the test and begin viewing the image series and again touch the image on the screen as quickly as possible whenever a repeated picture appeared. Each image appeared for 3 s or until the image on the screen was touched, which prompted immediate presentation of the next picture. Using the internal clock of the local device, MTx-RT for each image was determined by the elapsed time from presentation of the image to when the screen was touched by the participant in response to indicating recognition of the image as one that had been already shown during the test. MTx-RT was recorded for every image, with a full 3 s recorded indicating no response. MTx-% C was calculated to indicate the percentage of repeat and initial images to which the user responded correctly (true positive + true negative divided by 50). Additional details of the MemTrax administration and implementation, data reduction, invalid or “no response” data, and primary data analyses are described elsewhere [6].

The MemTrax test was explained in detail and a practice test (with unique images other than those used in the test for recording results) was provided to the participants in the hospital setting. Participants in the YH and KM sub-datasets took the MemTrax test on a smartphone that was loaded with the application on WeChat; whereas a limited number of the XL sub-dataset patients used an iPad and the rest used a smartphone. All participants took the MemTrax test with a study investigator unobtrusively observing.

Montreal cognitive assessment

The Beijing version of the Chinese MoCA (MoCA-BC) [13] was administered and scored by trained researchers according to the official test instructions. Suitably, the MoCA-BC has been shown to be a reliable test for cognitive screening across all education levels in Chinese elderly adults [14]. Each test took about 10 to 30 minutes to administer based on the respective participant’s cognitive abilities.

MoCA classification modeling

There was a total of 29 usable features, including two MemTrax test performance metrics and 27 features related to demographic and health information for each participant. Each patient’s MoCA aggregate test score was used as the cognitive screening “benchmark” to train our predictive models. Accordingly, because MoCA was used to create the class label, we could not use the aggregate score (or any of the MoCA subset scores) as an independent feature. We performed preliminary experiments in which we modeled (classifying cognitive health defined by MoCA) the original three hospital/clinic(s) sub-datasets individually and then combined using all features. However, all the same data elements were not collected in each of the four clinics representing the three sub-datasets; thus, many of our features in the combined dataset (when using all features) had a high incidence of missing values. We then built models with the combined dataset using only common features which resulted in improved classification performance. This was likely explained by a combination of having more instances to work with by combining the three patient sub-datasets and no features with an undue prevalence of missing values (only one feature in the combined dataset, work type, had any missing values, affecting only three patient instances), because only common features recorded at all three sites were included. Notably, we did not have a specific rejection criterion for each feature that was ultimately not included in the combined dataset. However, in our preliminary combined dataset modeling, we first used all features from each of the three separate patient sub-datasets. This widely resulted in model performance that was measurably lower than the initial preliminary modeling on each individual sub-dataset. Moreover, whereas the classification performance of the models built using all the features was encouraging, across all learners and classification schemes, performance improved for twice as many models when using only common features. In fact, among what ended up being our top learners, all but one model improved upon eliminating non-common features.

The final aggregate dataset (YH, XL, and KM combined) included 259 instances, each representing a unique participant who took both the MemTrax and the MoCA tests. There were 10 shared independent features: MemTrax performance metrics: MTx-% C and mean MTx-RT; demographic and medical history information: age, sex, years of education, work type (blue collar/white collar), social support (whether the test taker lives alone or with family), and yes/no answers as to whether the user had a history of diabetes, hyperlipidemia, or traumatic brain injury. Two additional metrics, MoCA aggregate score and MoCA aggregate score adjusted for years of education [12], were used separately to develop dependent classification labels, thus creating two distinct modeling schemes to be applied to our combined dataset. For each version (adjusted and unadjusted) of the MoCA score, the data were again separately modeled for binary classification using two different criterion thresholds—the initially recommended one [12] and an alternate value used and promoted by others [8, 15]. In the alternate threshold classification scheme, a patient was considered to have normal cognitive health if s/he scored ≥23 on the MoCA test and having MCI if the score was 22 or lower; whereas, in the initial recommended classification format, the patient had to score a 26 or better on the MoCA to be labeled as having normal cognitive health.

Filtered data for MoCA classification modeling

We further examined MoCA classification using four commonly used feature ranking techniques: Chi-Squared, Gain Ratio, Information Gain, and Symmetrical Uncertainty. For interim perspective, we applied the rankers to the entire combined dataset using each of our four modeling schemes. All rankers agreed on the same top features, i.e., age, number of years of education, and both MemTrax performance metrics (MTx-% C, mean MTx-RT). We then rebuilt the models using each feature selection technique to train the models on only the top four features (see Feature selection below).

The resultant final eight variations of the MoCA score classification modeling schemes are presented in Table 1.

MemTrax-based clinical evaluation modeling

Of our three original sub-datasets (YH, XL, KM), only the XL sub-dataset patients were independently clinically diagnosed for cognitive impairment (i.e., their respective MoCA scores were not used in establishing a classification of normal versus impaired). Specifically, the XL patients were diagnosed with either Alzheimer’s disease (AD) or vascular dementia (VaD). Within each of these primary diagnosis categories, there was a further designation for MCI. Diagnoses of MCI, dementia, vascular neurocognitive disorder, and neurocognitive disorder due to AD were based on specific and distinctive diagnostic criteria outlined in the Diagnostic and Statistical Manual of Mental Disorders: DSM-5 [16]. Considering these refined diagnoses, two classification modeling schemes were separately applied to the XL sub-dataset to distinguish level of severity (degree of impairment) for each primary diagnosis category. Data utilized in each of these diagnostic modeling schemes (AD and VaD) included demographic and patient history information, as well as MemTrax performance (MTx-% C, mean MTx-RT). Each diagnosis was labeled mild if designated MCI; otherwise, it was considered severe. We initially considered including the MoCA score in the diagnosis models (mild versus severe); but we determined that would defeat the purpose of our secondary predictive modeling scheme. Here the learners would be trained using other patient characteristics readily available to the provider and performance metrics of the simpler MemTrax test (in lieu of the MoCA) against the reference “gold standard”, the independent clinical diagnosis. There were 69 instances in the AD diagnosis dataset and 76 instances of VaD (Table 2). In both datasets, there were 12 independent features. In addition to the 10 features included in the MoCA score classification, patient history also included information on history of hypertension and stroke.

Statistics

Comparison of participant characteristics and other numerical features between sub-datasets for each model classification strategy (to predict MoCA cognitive health and diagnosis severity) was performed using Python programming language (version 2.7.1) [17]. The model performance differences were initially determined using a single- or two-factor (as appropriate) ANOVA with a 95% confidence interval and the Tukey honest significant difference (HSD) test to compare the performance means. This examination of differences between model performances was performed using a combination of Python and R (version 3.5.1) [18]. We employed this (albeit, arguably less than optimal) approach only as a heuristic aid at this early stage for initial model performance comparisons in anticipating potential clinical application. We then utilized the Bayesian signed-rank test using a posterior distribution to determine the probability of model performance differences [19]. For these analyses, we used the interval –0.01, 0.01, signifying that if two groups had a performance difference of less than 0.01, they were considered the same (within the region of practical equivalence), or otherwise they were different (one better than the other). To perform the Bayesian comparison of classifiers and calculate these probabilities, we used the baycomp library (version 1.0.2) for Python 3.6.4.

Predictive modeling

We built predictive models using the ten total variations of our modeling schemes to predict (classify) the outcome of each patient’s MoCA test or severity of the clinical diagnosis. All learners were applied and the models were built using the open source software platform Weka [20]. For our preliminary analysis, we employed 10 commonly used learning algorithms: 5-Nearest Neighbors, two versions of C4.5 decision tree, Logistic Regression, Multilayer Perceptron, Naïve Bayes, two versions of Random Forest, Radial Basis Function Network, and Support Vector Machine. Key attributes and contrasts of these algorithms have been described elsewhere [21] (see respective Appendix). These were chosen because they represent a variety of different types of learners and because we have demonstrated success using them in previous analyses on similar data. Hyper-parameter settings were chosen from our previous research indicating them to be robust on a variety of different data [22]. Based on the results of our preliminary analysis using the same combined dataset with common features that were used subsequently in the full analysis, we identified three learners which provided consistently strong performance across all classifications: Logistic Regression, Naïve Bayes, and Support Vector Machine.

Cross-validation and model performance metric

For all predictive modeling (including the preliminary analyses), each model was built using 10-fold cross validation, and model performance was measured using Area Under the Receiver Operating Characteristic Curve (AUC). Cross-validation began with randomly dividing each of the 10 modeling scheme datasets into 10 equal segments (folds), using nine of these respective segments to train the model and the remaining segment for testing. This procedure was repeated 10 times, using a different segment as the test set in each iteration. The results were then combined to calculate the final model’s result/performance. For each learner/dataset combination, this entire process was repeated 10 times with the data being split differently each time. This last step reduced bias, ensured replicability, and helped in determining the overall model performance. In total (for MoCA score and diagnosis severity classification schemes combined), 6,600 models were built. This included 1,800 unfiltered models (6 modeling schemes applied to the dataset×3 learners×10 runs×10 folds = 1,800 models) and 4,800 filtered models (4 modeling schemes applied to the dataset×3 learners×4 feature selection techniques×10 runs×10 folds = 4,800 models).

Feature selection

For the filtered models, feature selection (using the four feature ranking methods) was performed within the cross-validation. For each of the 10 folds, as a different 10% of the dataset was the test data, only the top four selected features for each training dataset (i.e., the other nine folds, or the remaining 90% of the entire dataset) were used to build the models. We were unable to confirm which four features were used in each model, as that information is not stored or made available within the modeling platform we utilized (Weka). However, given the consistency in our initial selection of top features when the rankers were applied to the entire combined dataset and the subsequent similarity in modeling performances, these same features (age, years of education, MTx-% C, and mean MTx-RT) are likely the most prevalent top four used concomitant with the feature selection within the cross-validation process.

Key points in clinical practice

Machine learning is often best utilized and most effectual in predictive modeling when the data are extensive and multi-dimensional, that is, when there are numerous observations and a concomitant wide array of high-value (contributing) attributes. Yet, with these current data, the filtered models with only four select features performed better than those utilizing all 10 common features. This suggests that our aggregate hospital dataset did not have the most clinically appropriate (high value) features to optimally classify the patients in this way. Nevertheless, the feature ranking emphasis on the key MemTrax performance metrics—MTx-% C and MTx-RT—strongly supports building early stage cognitive deficit screening models around this test that is simple, easy to administer, low-cost, and aptly revealing regarding memory performance, at least right now as an initial screen for a binary classification of cognitive health status. Given the ever-mounting strain on providers and healthcare systems, patient screening processes and clinical applications should be suitably developed with an emphasis on collecting, tracking, and modeling those patient characteristics and test metrics that are most useful, advantageous, and proven effective in diagnostic and patient management support.

With the two key MemTrax metrics being central to MCI classification, our top-performing learner (Naïve Bayes) had a very high predictive performance in most models (AUC over 0.90) with a true-positive to false-positive ratio nearing or somewhat exceeding 4 : 1. A translational clinical application using this learner would thus capture (correctly classify) by far most of those with a cognitive deficit, while minimizing the cost associated with mistakenly classifying someone with normal cognitive health as having a cognitive deficit (false positive) or missing that classification in those who do have a cognitive deficit (false negative). Either one of these scenarios of misclassification could impose an undue psycho-social burden to the patient and caregivers.

Whereas in the preliminary and full analyses we used all ten learners in each modeling scheme, we focused our results on the three classifiers showing the most consistent strong performance. This was also to highlight, based on these data, the learners that would anticipatedly perform dependably at a high level in a practical clinical application in determining cognitive status classification. Moreover, because this study was intended as an introductory investigation into the utility of machine learning on cognitive screening and these timely clinical challenges, we made the decision to keep the learning techniques simple and generalized, with minimal parameter tuning. We appreciate that this approach may have limited the potential for more narrowly defined patient-specific predictive capabilities. Likewise, whereas training the models using only the top features (filtered approach) informs us further regarding these data (specific to the shortcomings in data collected and highlighting the value in optimizing precious clinical time and resources), we recognize that it is premature to narrow the scope of the models and, therefore, all (and other features) should be considered with future research until we have a more definitive profile of priority features that would be applicable to the broad population. Thus, we also fully recognize that more inclusive and broadly representative data and optimization of these and other models would be necessary before integrating them into an effective clinical application, especially to accommodate comorbidities affecting cognitive performance that would need be considered in further clinical evaluation.

Utility of MemTrax was further edified by the modeling of disease severity based on separate clinical diagnosis. A better overall classification performance in predicting severity of VaD (compared to AD) was not surprising given the patient profile features in the models specific to vascular health and stroke risk, i.e., hypertension, hyperlipidemia, diabetes, and (of course) stroke history. Though it would have been more desirable and fitting to have the same clinical assessment conducted on matched patients with normal cognitive health to train the learners with these more inclusive data. This is especially warranted, as MemTrax is intended to be used primarily for early stage detection of a cognitive deficit and subsequent tracking of individual change. It is also plausible that the more desirable distribution of data in the VaD dataset contributed in part to the comparatively better modeling performance. The VaD dataset was well-balanced between the two classes, whereas the AD dataset with far fewer MCI patients was not. Particularly in small datasets, even a few additional instances can make a measurable difference. Both perspectives are reasonable arguments underlying the differences in disease severity modeling performance. However, proportionately attributing improved performance to dataset numerical characteristics or the inherent features specific to the clinical presentation under consideration is premature. Nonetheless, this novel demonstrated utility of a MemTrax predictive classification model in the role of clinical diagnostic support provides valuable perspective and affirms pursuit for additional examination with patients across the continuum of MCI.

The implementation and demonstrated utility of MemTrax and these models in China, where the language and culture are drastically different from other regions of established utility (e.g., France, Netherlands, and United States) [7, 8, 27], further underscores the potential for widespread global acceptance and clinical value of a MemTrax-based platform. This is a demonstrable example in striving toward data harmonization and developing practical international norms and modeling resources for cognitive screening that are standardized and easily adapted for use worldwide.

Next steps in cognitive decline modeling and application

Cognitive dysfunction in AD indeed occurs on a continuum, not in discrete stages or steps [28, 29]. However, at this early phase, our goal was to first establish our ability to build a model incorporating MemTrax that can fundamentally distinguish “normal” from “not normal”. More inclusive empirical data (e.g., brain imaging, genetic features, biomarkers, comorbidities, and functional markers of complex activities requiring cognitive control) [30] across varied global regions, populations, and age groups to train and develop more sophisticated (including aptly weighted ensemble) machine learning models will support a greater degree of enhanced classification, that is, the capacity to categorize groups of patients with MCI into smaller and more definitive subsets along the cognitive decline continuum. Moreover, concomitant clinical diagnoses for individuals across regionally diverse patient populations are essential to effectively train these more inclusive and predictably robust models. This will facilitate more specific stratified case management for those with similar backgrounds, influences, and more narrowly defined characteristic cognitive profiles and thus optimize clinical decision support and patient care.

Much of the relevant clinical research to-date has addressed patients with at least mild dementia; and, in practice, too often patient intervention is only attempted at advanced stages. However, because cognitive decline begins well before clinical criteria for dementia are met, an effectively applied MemTrax-based early screen could encourage appropriate education of individuals about the disease and its progressions and prompt earlier and more timely interventions. Thus, early detection could support suitable involvements ranging from exercise, diet, emotional support, and improved socialization to pharmacological intervention and reinforce patient-related changes in behavior and perception that singly or in aggregate could mitigate or potentially stop dementia progression [31, 32]. Moreover, with effective early screening, individuals and their families may be prompted to consider clinical trials or get counseling and other social services support to help clarify expectations and intentions and manage daily tasks. Further validation and widespread practical utility in these ways could be instrumental in mitigating or stopping the progression of MCI, AD, and ADRD for many individuals.

Indeed, the low end of the patient age range in our study does not represent the population of traditional concern with AD. Nonetheless, the average age for each group utilized in the classification modeling schemes based on the MoCA score/threshold and diagnosis severity (Table 3) underscores a clear majority (over 80%) being at least 50 years old. This distribution is thus very appropriate for generalization, supporting the utility of these models in the population characterizing those typically affected by early onset and burgeoning neurocognitive illness due to AD and VaD. Also, recent evidence and perspective stress those recognized factors (e.g., hypertension, obesity, diabetes, and smoking) potentially contributing to higher early adult and midlife vascular risk scores and consequent subtle vascular brain injury that develops insidiously with evident effects even in young adults [33–35]. Accordingly, the most optimal initial screening opportunity for detecting early stage cognitive deficits and initiating effective prevention and intervention strategies in successfully addressing dementia will emerge from examining contributing factors and antecedent indicators across the age spectrum, including early adulthood and potentially even childhood (noting the relevance of genetic factors such as apolipoprotein E from early gestation).

In practice, valid clinical diagnoses and costly procedures for advanced imaging, genetic profiling, and measuring promising biomarkers are not always readily available or even feasible for many providers. Thus, in many instances, initial overall cognitive health status classification may have to be derived from models using other simple metrics provided by the patient (e.g., self-reported memory problems, current medications, and routine activity limitations) and common demographic features [7]. Registries such as the University of California Brain Health Registry (https://www.brainhealthregistry.org/) [27] and others with an inherent greater breadth of self-reported symptoms, qualitative measures (e.g., sleep and every day cognition), medications, health status, and history, and more detailed demographics will be instrumental in developing and validating the practical application of these more primitive models in the clinic. Further, a test such as MemTrax, which has demonstrated utility in assessing memory function, may in fact provide a substantially better estimate of AD pathology than biological markers. Given that the core feature of AD pathology is disruption of neuroplasticity and an overwhelmingly complex loss of synapses, which is manifest as episodic memory dysfunction, a measure which assesses episodic memory may in fact provide a better estimate of AD pathological burden than biological markers in the living patient [36].

With all predictive models—whether complemented by complex and inclusive data from state-of-the-art technology and refined clinical insights across multiple domains or those limited to more basic and readily available information characteristic of existing patient profiles—the recognized advantage of artificial intelligence and machine learning is that the resultant models can synthesize and inductively “learn” from relevant new data and perspective provided by ongoing application utilization. Following practical technology transfer, as the models here (and to be developed) are applied and enriched with more cases and pertinent data (including patients with comorbidities that could present with ensuing cognitive decline), prediction performance and cognitive health classification will be more robust, resulting in more effective clinical decision support utility. This evolution will be more fully and practically realized with embedding MemTrax into custom (targeted to the available capabilities) platforms that healthcare providers could utilize in real-time in the clinic.

Imperative to the validation and utility of the MemTrax model for diagnostic support and patient care are highly sought-after meaningful longitudinal data. By observing and recording the concomitant changes (if any) in clinical status across an adequate range of normal through early-stage MCI, the models for appropriate ongoing assessment and classification can be trained and modified as patients age and are treated. That is, repeated utility can assist with longitudinal tracking of mild cognitive changes, intervention effectiveness, and maintaining informed stratified care. This approach aligns more closely with clinical practice and patient and case management.

Limitations

We appreciate the challenge and value in collecting clean clinical data in a controlled clinic/hospital setting. Nonetheless, it would have strengthened our modeling if our datasets included more patients with common features. Moreover, specific to our diagnosis modeling, it would have been more desirable and fitting to have the same clinical assessment conducted on matched patients with normal cognitive health to train the learners. And as underscored by the higher classification performance using the filtered dataset (only the top-ranked four features), more general and cognitive health measures/indicators would likely have improved modeling performance with a greater number of common features across all patients.

Certain participants might have been concomitantly experiencing other illnesses that could have prompted transitory or chronic cognitive deficiencies. Other than the XL sub-dataset where the patients were diagnostically classified as having either AD or VaD, comorbidity data were not collected/reported in the YH patient pool, and the predominant reported comorbidity by far in the KM sub-dataset was diabetes. It is arguable, however, that including patients in our modeling schemes with comorbidities that could prompt or exacerbate a level of cognitive deficiency and a consequent lower MemTrax performance would be more representative of the real-world targeted patient population for this more generalized early cognitive screening and modeling approach. Moving forward, accurate diagnosis of comorbidities potentially affecting cognitive performance is broadly beneficial for optimizing the models and resultant patient care applications.

Lastly, the YH and KM sub-dataset patients used a smartphone to take the MemTrax test, whereas a limited number of the XL sub-dataset patients used an iPad and the rest used a smartphone. This could have introduced a minor device-related difference in MemTrax performance for the MoCA classification modeling. However, differences (if any) in MTx-RT, for example, between devices would likely be negligible, especially with each participant being given a “practice” test just before the recorded test performance. Nevertheless, utility of these two handheld devices potentially compromises direct comparison to and/or integration with other MemTrax results where users responded to repeat pictures by touching the spacebar on a computer keyboard.

Key points on MemTrax predictive modeling utility

These novel findings establish definitive evidence supporting the utility of machine learning in building enhanced robust MemTrax-based classification models for diagnostic support in effective clinical case management and patient care for individuals experiencing cognitive impairment.