Validity and Reliability Study of Online Cognitive Tracking Software (BEYNEX)

INTRODUCTION

Currently, more than 55 million people have dementia worldwide, over 60% of whom live in low-and middle-income countries. It is estimated that global spending on dementia increased by 4.5% annually from 2000 to 2019, reaching $263 billion attributable to dementia in 2019 [1]. Individuals with cognitive disorders frequently avoid visiting the doctor, and it is common that they must be taken by another individual only after these dysfunctions seriously start to affect their daily lives. Currently, it is estimated that 75% of people with dementia globally are not diagnosed, and this rate is likely even higher in some countries [2]. On the other hand, several individuals worry about dementia unnecessarily due to their forgetfulness. In a review on the determination of communication patterns in patients applying to memory clinics, the multidimensional evaluation of patients complaining of forgetfulness is emphasized. Whether the patient attends with a companion, how they participate, give autobiographical history, demonstrate working memory, and make qualitative observations during routine cognitive testing are all useful in building a diagnostic picture [3]. This diagnostic picture is then validated by impartial data such as neuropsychological tests.

Despite recent advances in the identification of mild cognitive impairment (MCI)-related biomarkers, the neuropsychological assessment remains a critical component of evaluation to ensure that cognitive function correlates with biomarker abnormalities to assist in detecting and tracking the progression of MCI to early Alzheimer’s disease (AD) [4]. However, these assessments are not designed for longitudinal tracking of patients in short time intervals. They are designed to be administered by trained professionals to support a diagnosis with high specificity and sensitivity with both the patient and the administrator being present. Unfortunately, given the prevalence of the disease and the number of people seeking professional help, for both addressing this public health concern from a preventive point of view and adapting how the disease is managed, a valid, reliable, accessible, engaging, and affordable digital cognitive screening instruments for clinical use is in urgent demand and rapid cognitive screening of societies is now possible with online applications that have developed rapidly in recent years [5]. Studies have shown that these online applications can be used in other neurological diseases that require cognitive monitoring, such as stroke, in addition to dementia [6]. Significant progress has been made in the field of digital cognitive testing in recent years. There are now digital cognitive test alternatives to Montreal Cognitive Assessment (MoCA) that still require a trained professional to administer such as CNS Vital Signs [7], NIH toolbox for assessment of neurological and behavioral function [8], CogState [9], BrainCheck [10], and many other such tests. These tests, while being digital, still cannot offer a solution on how to address the undiagnosed population and patients that do not seek professional help on time. Other alternatives such as Cognivue [11] are self-administered but still require either a costly device or presence of the patient in the clinic. And finally, tools such as MemTrax [12] and Boston Cognitive Assessment [13] have been developed to assess patients online with a self-administered digital assessment test. Especially, the MemTrax test, has shown in various studies that it can be relied upon to screen for cognitive impairment and other conditions [5, 12]. To address the concerns of healthy individuals and detect early signs of cognitive decline in patients that may lead to AD, it can be said that accurate, self-administered online longitudinal cognitive tracking will be invaluable in the future.

AD care requires cognitive tracking of cognitive performance at the earliest point of concern and monitoring disease progression and treatment when appropriate. Traditionally, a broad range of separate tests is combined into a neurocognitive test battery to measure the full spectrum of cognitive functioning. However, combining enough of these traditional tests into a complete test set can lead to a long time-consuming evaluation. As a result, clinical cognitive monitoring takes a lot of time and causes high costs. Therefore, online cognitive tracking applications are quite feasible. However, these applications should be shown to be valid by clinical tests, and cut-off values should be determined.

With the contribution of today’s digital technology, many online applications have started to be used in the topics of cognitive stimulation, training, and rehabilitation. Research results of programs targeting training and rehabilitation are discussed in various review articles [14, 15]. Most studies reported that users do not need to be technologically skilled to successfully complete or benefit from training. Overall, findings are comparable to or better than reviews of more traditional, paper-and-pencil cognitive training approaches suggesting that computerized training is an effective, less labor-intensive alternative [12]. However, those skeptical of the benefits of cognitive training or rehabilitation are right to argue that positive data are insufficient. However, in our opinion, online cognitive applications can provide longitudinal tracking information to the person by constantly comparing the performance of the users with those of their age groups and similar education levels and by monitoring their daily life activities. As it moves away from the average cognitive values, it may recommend that the person applies to the clinic. With today’s developing artificial intelligence (AI) technology, it can determine the life factors that affect a person’s cognitive level and produce unique messages for that person [16]. In this study, we aim to demonstrate that the Beynex gamified assessment test can be self-administered by its users and provide longitudinally data on users’ cognitive performance in various parameters in line with the current standard of pen and paper based neuropsychological tests.

METHODS

Beynex is an application created by neurologists, psychologists, and computer engineers. The first version was released in January 2020. The application, which can be downloaded in Android and IOS environments, works on tablets and smartphones. It monitors people with cognitive concerns in 7 different categories (speed, flexible thinking, language, memory, visual perception problem-solving, and attention). Each day, the users are prompted to take one of the gamified exercises that tests these parameters in one category to eliminate learning bias and 3 times to standardize the results. Each test lasts about 4 to 6 minutes. Thus, a full cognitive survey of each user is completed every 7 days. Beynex instantly evaluates the user’s cognitive performance in each category, and presents it in a single score, Beynex Performance Index (BPI).

φⁿ: average in n’th category

BPI is a summary of your cognitive level, with respect to the cognitive performance that a person in a population is supposed to have compared to the average.

As for the 7 categories, users are scored based on their accuracy and/or their speed in completing the gamified assessments described below:

Speed: A circle appears on the screen at random intervals, users must then and only then click on them before they disappear.

Flexible thinking: Users are given 2 color names with different font colors; they must answer correctly or incorrectly depending on whether or not the meaning of the first color name is the same as the second color’s written font color.

Language: An image is given to users. They must then write its name.

Memory: Users are given a grid of unlit squares which then light up randomly. Once the grid is once again unlit, they must replicate the lit squares.

Visual perception: Users are given objects with location descriptions and are asked to select if the statements are correct or incorrect.

Problem-solving: There are multiple jars with different colored items. Users must collect all the same-colored items in the same jar.

Attention: A number is shown on-screen. After it disappears the users must rewrite it backwards.

Additionally, Beynex provides a way to track daily activities such as time spent reading or watching TV. There are nearly 20 free exercise games for people who want to exercise more. However, the performance scores of additional games do not affect cognitive follow-up parameters.

This study was performed, reviewed, and approved by the ethical committee of the Maltepe University Medical Faculty in Istanbul, Turkey (2023/900/31). All participants voluntarily provided signed informed consent.

RESULTS

Cut-off score determination

The results of 91 subjects who took the MoCA and the Beynex tests are used to determine the cut-off values. The categorical distribution of the participants based on their test scores is given in the scatterplot can be seen in Fig. 1. The cut-offs for BPI scores were determined by maximizing the optimization function as mentioned in the method section. Two cut-offs were found to be 49.0 and 55.6 and the categories defined by scores of both MoCA and BPI are given in Table 3. According to the MoCA scores, 51.6% of participants had no impairments (>24), 39.6% had mild impairments (24–20), and 8.8% had severe impairments (<20). Of the 91 subjects 55.0% had no impairments, 17.6% had mild impairments, and 27.5% had severe impairments after applying the cut-offs that have been determined for BPI scores.

Fig. 1

Scatterplot of MoCA and BPI scores of participants for cut-off score determination study. The blue linear regression line, represented by the equation y = (0.14±0.02)×BPI + (16.29±1.17), has an R^∧2 of 0.38. Green lines correspond to lower cut-offs of the tests whereas yellow ones represent upper cut-offs. NPA = 0.78 and PPA = 0.88 for the lower cut-offs while NPA = 0.83 and PPA = 0.75 for the upper cut-offs.

Table 3

Descriptive categories and their corresponding MoCA and BPI cut-offs

	Cut-off Scores
	MoCA	BPI
Impaired	<20.0	<49.0
Mildly Impaired	20.0–24.0	49.0–55.6
Unimpaired	>24.0	>55.6

Percentage agreements for the lower cut-offs that maximized the optimization function are NPA = 0.78, PPA = 0.88. Using the upper cut-offs corresponded to NPA = 0.83, PPA = 0.75. The accuracy value is 0.791 for both lower and upper cut-off values which can be considered as significant. When a linear regression line was added, the coefficient of determination R^∧2 was calculated as 0.38.

Clinical validation study

During the clinical validation study, we compared the evaluations of 214 participants using both the MoCA and BPI. Table 4 presents a comprehensive comparison, revealing how the participants’ cognitive states varied between the two tests. In the MoCA assessment, 6.54% of participants were identified as impaired, 40.19% as mildly impaired, and 53.27% as unimpaired. The results from the Beynex BPI evaluation showed that 26.2% of participants were classified as impaired, 16.8% as mildly impaired, and 56.97% as unimpaired.

Table 4

Distribution of participants based on their MoCA and Beynex test scores. The percentage each cell represents is given in parentheses. Here, the number of participants is 214 and this group has been used in the validation study

MoCA∖Beynex	Impaired	Mildly	Unimpaired	Total
		impaired
Impaired	11 (5.1%)	1 (0.5%)	2 (0.9%)	14
Mildly Impaired	35 (16.4%)	22 (10.3%)	29 (13.6%)	86
Unimpaired	10 (4.7%)	13 (6.1%)	91 (42.5%)	114
Total	56	36	122	214

Figure 2 complements these findings by displaying the test scores of 241 study participants in a scatterplot. The best fitting line was obtained by linear regression and the coefficient of determination R^∧2 is 0.36. NPA and PPA were found to be 0.79 and 0.78 respectively for the lower cut-offs while NPA was 0.69 and PPA was 0.8 for the upper cut-offs. These results shows that the data used for validation agrees with the data used for cut-off determination. Accuracy given in Eqn 4 was calculated as 0.77 for lower and 0.75 upper cut-off values.

Fig. 2

Scatterplot of MoCA and BPI scores of participants for validation study. The blue linear regression line, described by the equation y = (0.14±0.01)×BPI + (16.65±0.76), has an R^∧2 of 0.36. Green lines correspond to lower cut-offs of the tests whereas yellow ones represent upper cut-offs. NPA = 0.79 and PPA = 0.78 for the lower cut-offs while NPA = 0.69 and PPA = 0.8 for the upper cut-offs.

In addition to the NPA, PPA, and accuracy, ROC analysis was performed. The AUC was 0.827 (with [0.778, 0.871] 95% CI) when comparing unimpaired to mildly impaired (Fig. 4a) and the AUC was 0.832 (with [0.756, 0.888] 95% CI) when comparing mildly impaired to impaired (Fig. 4b).

Fig. 3

Scatterplot of the comparison between first and second attempts of subjects at Beynex test. The added regression line has a slope of 0.86 with a 95% confidence interval (0.791, 0.935) which is shown as the shaded blue are about the line. The y-intercept is 10. R^∧2 was found to be 0.647.

Fig. 4

ROC curve analysis with AUCs comparing Beynex test performance against MoCA diagnoses. The solid blue line signifies Beynex’s efficacy in differentiating unimpaired from mildly impaired in (Fig. 4a) and mildly impaired from impaired in (Fig. 4b). The gray dashed line is the reference line of no-discrimination.

Cohen’s kappa K was calculated to quantify interrater reliability between Beynex and MoCA tests. Quadratic weighting has been used to emphasize the categorical identifications that differ by 2. An example for such a case is when a participant is categorized as impaired in one of the tests while the other categorizes the same participant as unimpaired. K has been found to be 0.4843 with a 95% CI of 0.3759 to 0.5927. This result demonstrates moderate agreement between the two tests.

To assess the reliability and internal consistency of the 7 subtests of Beynex Cronbach’s Alpha has been calculated as 0.823 with a 95% CI of [0.792, 0.852] which indicates that the test has good internal consistency.

Test-retest reliability of the Beynex test was investigated by comparing two consecutive tests of participants that were at least 2 weeks apart. The scatterplot of the two consecutive BPI scores of the participants is given in Fig. 3 along with a linear regression line. R^∧2 has been found as 0.647 which is quite good considering the learning effect that takes place during the first taking of the test that boosts the second trial scores of the participants.

DISCUSSION

Quality care for people with AD and other dementias starts with an early, documented diagnosis, including disclosure of the diagnosis. However, most people who have been diagnosed with AD are not aware of their diagnosis. Evidence indicates that only about half of those with AD have been diagnosed. Among those seniors who have been diagnosed with AD, only 33% are even aware that they have the disease. Even when including caregivers, 45 percent of those diagnosed with AD or their caregivers are aware of the diagnosis [11].

Detecting cognitive impairment, diagnosing AD and other dementias, and disclosing that diagnosis to the individual are necessary elements to ensuring that people with dementia, together with their families, have the opportunity to access available treatments, build a care team, participate in support services, enroll in clinical trials, and plan for the future. Furthermore, there is growing consensus that novel disease-modifying therapies for AD would be appropriate and more likely to deliver benefit when used at earlier stages in the disease evolution, potentially at the prodromal stage [21].

The 2017 Geriatric Summit on Assessing Cognitive Disorders among the Aging Population, sponsored by the National Academy of Neuropsychology, emphasized the importance of ongoing screening and the need for automated tools for assessing and recording patients’ results over time [22].

We can list the obstacles to evaluation with cognitive tests in the clinic as high cost, timing, and lack of trained personnel in this field. [23–25]. MCI or milder cognitive impairment often accompanies daily living activity change and has been shown in many studies. Assessment of activities of daily living (ADL) is paramount to underpin accurate diagnostic classification in MCI and dementia. Unfortunately, most common report-based ADL tools have limitations for diagnostic purposes [26–29].

Beynex is self-administered by the patient and can be initiated by non-clinician support personnel. Because of these advantages, it was written for the purpose of long-term cognitive follow-up rather than the dementia screening test. The Beynex software, which is the subject of this research, does not claim to be cognitive training or rehabilitation; it only aims to stimulate the user by performing cognitive follow-up. With the experience obtained from 3 years of user data, we observed that the follow-up of both cognitive and daily living activity changes for at least 3 months or longer is much more meaningful than instant evaluations in people using the application.

This study aimed to verify the BPI as a cognitive assessment tool and establish the cut-off values for the BPI. The scatterplots and linear regression analysis showed a correlation between the BPI scores and MoCA scores. The determined cut-off values for the BPI scores allowed for categorizing participants into different levels of impairment, aligning with the categories defined by the MoCA scores. Additionally, the AUCs for both cases when differentiating unimpaired from mildly impaired and mildly impaired from impaired show that BPI can screen for cognitive performance with good accuracy. The validation study further supported the accuracy and reliability of the BPI test. The agreement between the BPI and MoCA tests, quantified by Cohen’s kappa coefficient, indicated moderate agreement between the two assessments. The internal consistency of the BPI subtests was also found to be good, as evidenced by the calculated Cronbach’s alpha value. The test-retest reliability analysis demonstrated consistent performance of the Beynex test over time, indicating its stability and reliability for measuring cognitive function when administered to participants at least 2 weeks apart.

The findings of this study emphasize the importance of early detection and diagnosis of cognitive impairment, including AD and other dementias. Automated tools like the Beynex test can facilitate ongoing screening and tracking of cognitive function, enabling timely access to available treatments, support services, and future planning for individuals with dementia and their families. Overall, the Beynex test shows promise as a self-administered cognitive assessment tool that can be initiated by non-clinician personnel. Its ease of use, reliability, and ability to track cognitive and daily living activity changes make it a valuable tool for long-term cognitive follow-up. Future research of the Beynex test should focus on further validation by widening the sample population to include a more diverse demography of participants, and people of varying severity of cognitive decline.