Situation Model for Situation-Aware Assistance of Dementia Patients in Outdoor Mobility

Mobility in dementia

An active and socially integrated lifestyle protects against cognitive decline in aging [4]. People with mild cognitive impairments have a reduced risk of conversion to dementia when engaged in outdoor and social activities [5, 6]. Furthermore, it allows people with mild cognitive impairments to continue their independent social life and to retain their dignity and self-esteem. In dementia, cognitive rehabilitation shows promising results in delaying further cognitive decline [7]. Outdoor mobility gains importance in this context, both as a prerequisite for an active lifestyle and also as a cognitively demanding activity in itself. Decreased mobility in dementia results from progressive cognitive decline. The complexity of highly dynamic outdoor environments poses an additional obstacle for cognitively impaired people. Indeed, people with dementia report decreased access to socially salient public places and socially stimulating activities [8] due to lack of motivation, previous experiences of errors in complex situations, and feeling of insecurity. The interplay between cognitive decline and decreasing mobility forms a vicious circle: decreased cognitive abilities reduce social activities which in turn impairs cognitive abilities, already affected by the disease process. In this perspective, ATDs that respect the remaining cognitive abilities and provide help for the user when needed might have the potential to break this vicious circle and preserve the autonomy of persons with dementia.

Such an assistive system needs to be able to reason about the user’s situation, as the situation influences the intervention strategy the ATD will follow[9, 10]. To be able to provide situation awareness, however, the system should have access to the knowledge base that represents the various situations related to outdoor mobility. Additionally, a group of individuals with dementia is typically heterogeneous with respect to declined mental abilities, and above that they are very heterogeneous with respect to premorbid characteristics as environmental exposure, desired activities, preferred range and variation of activities, and subjective meaning of particular activities. Thus, an ATD that can make use of situation knowledge has the advantage of providing a standard platform/instrument to quickly develop an individual prediction model or activity profile that is maximally useful for the individual user.

Situation models for situation-aware ATDs

Situation-aware assistance means that the ATD plays the role of a mediator between the person and the environment by observing the person through sensors, predicting his goals, and, only if needed, providing the missing information required to achieve the goal [3, 11, 12]. In other words, the assistance is not limiting the person’s outdoor activities unless necessary. A core element of a situation-aware assistance system is the situation model. It defines the domain knowledge needed to reason about the person’s situation. This knowledge could include personal preferences, social background, external and internal factors influencing the behavior of the person with dementia, elements of the environment he is acting in, etc. [13]. The situation model together with sensor observations is then used to reason about the person’s current situation and goals as well as about possible assistance strategies [3, 14]. It is also used as a codebook for annotating the observed behavior.

Situation models can be obtained in two ways, either through specification-based approaches, or through learning-based approaches [15]. Specifica-tion-based approaches rely on manually incorporating expert knowledge into logic rules that allow reasoning about the situation [15]. On the other hand, learning-based methods can rely on the sensor data to learn the situation [16] or on textual sources to extract the situation related information and its semantic structure [17–20]. In this work, we use specification-based approach to build the situation model. The situation model is often encoded in the form of ontologies. In this work, we consider a situation model to be a special case of ontology that incorporates the domain knowledge required by an assistive system to reason about the user behavior and the possible assistance strategies.

Objective

The objective of this paper is to provide a situation model for situation-aware assistance of people with dementia during their outdoor mobility. This model was developed as part of the SiNDeM project (Funded by the German Federal Ministry of Education and Research (BMBF, reference number: 16SV7091), aiming to enable technology support for outdoor mobility of people with dementia. The model has two purposes: 1) to provide the knowledge base in a situation-aware assistive technology device, namely a device that recognizes situations of disorientation and provides individually tailored guidance in real time; and 2) to be used as a codebook for annotating the data, collected during the outdoor mobility of people with dementia.

As a result, this work provides the basic structure of a situation model of outdoor mobility behavior and the associated cognitive factors that form the basis of knowledge-based situation-aware assistance and that can be used to code (transcribe) behavior traces. The situation model is implemented in the form of an OWL ontology.

Development process for situation models

To build the situation model, we followed a development process based on the general ontology process proposed in [21]. It consists of four phases; Fig. 1 shows the process. The proposed process consists of domain analysis, conceptualization, implementation, and maintenance. The domain analysis involves analysis of different scenarios, discussion of relevant problems, observation of use cases, analysis of the requirements of the system, and review of existing solutions. The second step of the process is the conceptualization. It involves the conceptualization of the model and the integration of existing ontologies or relevant information. The last step in the development process is the model implementation. It deals with implementing the formal model in a concrete representation language. In addition, it has the explicit phase “Maintenance” which involves adapting and extending the developedmodel. In parallel to these phases, there are two additional processes. These are the evaluation of the results of each phase and their documentation.

Fig.1

The proposed development process for situation models. Figure adapted from [21]. The process consists of domain analysis, conceptualization, implementation, and maintenance. In parallel to these phases, there are two additional processes. These are the evaluation of the results of each phase and their documentation.

Domain analysis

Outdoor mobility study

People with dementia were observed during their outdoor mobility. Ethical approval was granted by the ethics commission at the medical faculty of the University of Rostock (Approval #A2014-0154). The study had two purposes: 1) to collect sensor data that is used for training a model of human behavior during outdoor mobility; and 2) to collect behavior traces for the construction of a situation model. During the initial study, the psychologist followed the participants in 20-min walks, observed their behavior, and later discussed the observed problems with the ontology developer. Finally, the psychologist, together with a computer scientist responsible for the later situation model and ATD development, analyzed the video log of the walks to identify the observed behavior.

Additionally, a life-space assessment was conducted (see Fig. 2). There, the study participants took part in an at least 4-week long assessment of their outdoor mobility. It was recorded through sensors and mobility diaries. The data from the long-term assessment has not been fully analyzed. However, initial analysis provided some insights about the behavior patterns and user preferences.

Fig.2

Life-space levels as proposed in [29]. Life-space of 0 is limited to the bedroom; Life-space of 1 is limited to the home of the person; Life-space of 2 is limited to the immediate outside of the home; Life-space of 3 is limited to the neighborhood; Life-space of 4 is limited to the town where the person is living; Life-space of 5 is unlimited.

Study participants. Our study participants were patients from the local memory clinic who were diagnosed with amnestic mild cognitive impairment or clinically probable Alzheimer’s disease dementia, according to the national guidelines [26]. Cognitive testing had been performed using the CERAD cognitive battery [27]. Severity of dementia was graded using the Mini-Mental State Exam (MMSE) [28]. We only included people with an MMSE score of 18 and above (mild to moderate dementia). Participants were interviewed about their current level of mobility. Patients who reported to have independently reached a Life-space level [29] of at least three (Level 3 indicates that the person has been outside the house, visited places in the neighborhood) within the last month (see Fig. 2 for details on the levels of Life-space) were eligible to further participate in the mobility assessment. Patients were only included after oral and written information on the study procedures, and after they had given their written informed consent. The study had been approved by the local institutional review board of the University Medicine Rostock.

The mobility assessment study comprises two separate setups, a guided walk and a long-term assessment.

Guided walk. The fifteen participants that answered the above criteria were invited to a 20-min guided walk from which thirteen completed the walk. The sensor system consisted of sensors to measure electrodermal activity, electrocardiography, GPS, and accelerometric data. Data about electrodermal activity and electrocardiography were intended to infer the stress level and possible states of disorientation during the walk. For later analysis and annotation, the walk was videotaped and events were protocoled by a trained psychologist. The motion sensors were attached to the left ankle, the chest, and the left wrist. The sensor data was annotated according to the schema identified in the situation model and it was shown that the data conforms with the behavioral findings in the situation model. Detailed information about the sensors and their placement can be found in Supplementary Material B.

Prior to the recorded and videotaped part of the walk, the participant and the psychologist met at a defined public location (see Fig. 3). The psychologist guided the participant to a tram station approximately one kilometer away. The route was designed to be cognitively challenging for patients with dementia. Aside from the usually crowded tram station, the route contained a four-way crossing of two main streets, as well as narrow streets in a residential neighborhood. At the tram station, the participant was asked to walk back on his/her own. On the way back, the psychologist stayed in the background, observed and protocoled the behavior, and only interacted with the participant in case of unresolvable disorientation or potentially endangering situations. The study contained two potential risks for the participants: increased stress level and risk associated with the surrounding traffic. To address the first risk, the participants were briefed before the study and the route was clearly explained. It was also made clear that the psychologist will be following the participants at all times, so that they will not be left alone. This also addresses the second problem, where to avoid any risks from the traffic, the psychologist stayed near the participants at all times, so that in case of risk she could physically or verbally intervene.

Fig.3

The guided walk. The person with dementia is guided by a psychologist to a tram station, which is 1 km away. The person is then asked to find the way back to the start point on their own.

Life-space assessment. The guided walk was followed by a long-term assessment over at least 4 weeks, called life-space assessment. Here, daily outdoor mobility activities of 11 out of the initial 15 study participants were recorded. The sensor system for the long-term assessment was only GPS andaccelerometer. In addition, the participant kept a mobility diary. It was developed based on experts’ brainstorming. After the first prototype was developed, it was tested on two of the study participants and optimized before being used in the study itself. We asked the participants to wear a bandage containing GPS and accelerometer and to add an entry to their mobility diary every time they left their home. The diary was filled in twice for each outing, once for the way to their goal, and once for the way back home. Each entry in the diary had a standardized form and contained date, time, destination, stopovers, means of transportation, incidents, and eventually company (e.g., relatives, friends, or a dog).

Conceptualization

Implementation

During the implementation phase, the Web Ontology Language (OWL) [33] and OBO Edit (http://oboedit.org/) were used to encode the ontology. OWL was chosen, as it is a semantic web language that can represent rich and complex knowledge about things, groups of things, and the relations between things. Moreover, it is a widely-used language that allows the use of the ontology across different applications. That also allows the easy conversion of the ontology in another targeted language.

Evaluation

To validate the situation model, the OBO Edit tool and its reasoner were used:

Furthermore, domain experts were asked to revise the situation model. The experts’ revision of the semantic structure is considered to be genuine evaluation for ontologies [34] and as the situation model is a special case of ontology, we also use the experts’ revision for evaluation. Apart from that, two annotators were asked to annotate the video logs from the guided walk in the mobility study using the same codebook extracted from the situation model. The annotation serves as a ground truth for evaluating the performance of the ATD when recognizing the user behavior. In this case, it was also used to evaluate whether the concepts, identified in the situation model, are recognizable in the video log of the mobility study. This provides information about the objectiveness of the situation model. In order to measure this, the interrater reliability in terms of Cohen’s kappa between the two annotators was calculated. Cohen’s kappa between 0.00–0.20 indicates slight agreements between annotators, between 0.21–0.40 fair agreement, between 0.41–0.60 moderate agreement, between 0.61–0.80 substantial agreement, and between 0.81–1.00 almost perfect agreement[35, 36]. We have to mention that there are no established benchmarks for this methodology. Landis and Koch [35] addresses the annotation of dance movement, while [36] comes from the field of radiology. Regardless, to our knowledge, existing benchmarks from different fields use the same scales to measure the annotation reliability with Cohen’s kappa. As the aim was to evaluate whether the concepts in the situation model can be recognized in the observed behavior by different annotators, a training phase was not conducted with the annotators. This is opposed to other approaches aiming at training the annotators to always code a given behavior with the same label. In the latter case, the name of the label could be irrelevant, as the annotators are already trained to associate the label with the corresponding behavior, regardless of the fact that the label could trigger other behavior associations in untrained annotators.

Model structure and contents

The system requirements evaluation and the review of existing ontologies yielded a knowledge base divided into four main groups describing the factors influencing the behavior of people with dementia during their outdoor mobility. These are “interaction with the environment”, “extrinsic factors”, “intrinsic factors”, and “long-term intrinsic factors”. The results from the requirements evaluation and the literature review can be found in Table 1. More details about the rationale behind the obtained results can be found in Supplementary Material A.

During the outdoor mobility study, four challenging behaviors, associated with disorientation during outdoor mobility were identified. These are “spatial reasoning”, “topological reasoning”, “communication”, and “wandering”. Apart from them, different normal actions were identified. The list of identified behaviors can be found in Table 3. The outdoor mobility study also produced a set of goals the participants were following as well as different transportation modes. These are listed in Table 4 under “Intention / Goal” and “Transportation preferences”. More details about the outdoor mobility study can be found in Supplementary Material B.

The interviews with people with dementia showed three different types of technology users: Type 1: people that have technical understanding and interest in technology based on their professional experiences; Type 2: people that have technical understanding and interest in technology based on their personal interests and hobbies; Type 3: none or small interest in modern technology. The type of users affects the way in which the ATD can provide assistance. More details about the outdoor mobility study can be found in Supplementary Material B.

The knowledge extracted from the domain analysis and IUA was divided into different sub-categories and the relations between the different elements were identified. This resulted in 101 concepts categorized in a hierarchical structure where the uppermost categories are those identified from the system requirements examination, while the middle and bottom layers are based on the literature review, the interviews, the mobility study, and the interaction unit analysis. The concepts and their hierarchical structure can be seen in Table 4. More details about the definition of each concept can be found in Supplementary Table 1.

Apart from the concepts, 11 relation types were identified, divided into five categories. These are “relations describing the abstraction hierarchy of concepts”, “relations describing the interaction of the user (person) with the environment”, “relations describing locational dependencies”, and “relations describing the causal dependencies between concepts”. The complete list of relations can be seen in Table 5. Some additional details and examples about the relations can be found in Supplementary Material C.

Figure 4 shows a conceptualization of the situation model with the corresponding relations between the different concepts. The situation model implemented in OWL is publicly available for download at the Library of the University of Rostock [37]. The concepts’ definitions can be found in Supplementary Table 1.

Fig.4

The identified concepts and the relations between them. Orange indicates the person with dementia, green the interaction the person can execute. The behavior can be normal or challenging (both in dark grey). The rest of the concepts are in light grey, while their subconcepts or concepts identified during the mobility study and interviews are in white. The full list of concepts can be found in the Supplementary Material C and Supplementary Table 1. The implemented situation model can be found in [37] (https://doi.org/10.18453/rosdok_id00000150).

Evaluation

Advantages

The situation model provides the code raster needed for annotating the user behavior. It also provides the semantic meaning behind the annotated behavior, such as reasons for the observed behavior and relation to other annotated behaviors. The situation model is also essential for detecting the user situation and reasoning about the appropriate intervention in case of abnormal behavior. It is the first systematically developed situation model for assistance during outdoor mobility for people with dementia. Existing situation models concentrate on assisting people with dementia in indoorsettings.

Our work is also the first to conduct a mobility study for people with dementia in order to obtain the sensor data and the context information necessary for developing assistive systems for outdoor mobility for people with dementia. It is also specifically tailored to address the needs of people with dementia. For that reason, it contains detailed information about errors in behavior during outdoor mobility and the corresponding causes of these errors.

The annotation of the guided walk also showed that using the code raster from our situation model produces substantial [35] agreement between two different annotators (Cohen’s kappa of 0.61). This stands to show that the proposed model is not only a subjective interpretation but that it also reflects the perception of people who did not take part in the model development. Finally, the situation model incorporates the knowledge from the literature, from personal experiences of people with dementia, and from the requirements needed to develop an ATD for outdoor mobility. This makes it a good starting point for any assistive system that aims at outdoor mobility for people with dementia.

Limitations

Apart from the advantages listed above, there are some limitations associated with the proposed model.

The model does not integrate the knowledge from the long-term assessment and the mobility diary as at the time of development they were not analyzed. We were, however, able to conduct an initial analysis of the long-term assessment study and to integrate our first impressions in the situation model. In the future, we plan to annotate the long-term study with the code raster from the situation model, which will provide additional evaluation of the model’s functionality.

Clinical implications

The situation model for outdoor mobility for people with dementia has two clinical application fields.

1) It tests hypotheses on disoriented behavior in real world settings with sensor based behaviorassessment.

The phenomenology of functional decline in dementia is poorly described. The majority of studies use rating instruments by proxy or self-rating instruments to determine impairments in (instrumental) activities of daily living including way-finding, e.g., the Bayer ADL scale. Such scales are usefully employed to determine the effect on aggregated functional outcomes in clinical trials, but even here they have been criticized for limited sensitivity to functional change and limited validity in respect to underlying behavior features [38]. Another approach is to assess the degree of impairment of cognitive functions that are involved in the maintenance of navigation abilities using psychometric scales. This approach cannot directly assess the effect of cognitive change of everyday function in a given patient, but is based on inference from a stochastic model on the association between cognitive ability and everyday function in groups of patients. In acknowledgment of this fact, major stakeholders, such as the Alzheimer’s Association, the Food and Drug Administration, pharmaceutical industry, and the European Research Council (ERC) have adopted the concept of real world evidence. In 2015, the ERC asked for the “collection and analysis of high quality real world evidence (RWE)”, stating that “many factors prevent the reliable prediction of the natural course of [Alzheimer’s disease] AD in real-world settings... [Including]... lack of clarity on how to best model the natural history of the disease using real world data, particularly as disease stages are difficult to define outside of highly qualified assessment centers or research cohorts.” (http://www.imi.europa.eu/sites/default/files/uploads/documents/IMI2Call6/IMI2_Call6_FINAL.pdf, p. 29). Sensor based automated assessment of disorientation using an adequate situation model can be regarded a major patient-powered real world endpoint to directly and objectively assess the effect of disease progression or interventions on patient function.

2) An adequate model of disorientation in dementia can serve to inform a situation-aware ATD to select an intervention that meets the actual need of a person in a given phase of disorientation. The inferred type of orientation error would drive the selection of the intervention allowing for a user-tailored set of intervention. This in turn provides the user with as little information as necessary to find his/her way back again rather than a generic maximal intervention, such as calling the caregiver every time the user leaves a predefined route, irrespective of potentially underlying decisions or needs of the user.

In the future, we plan to integrate the situation model into a situation detection system. We will evaluate its performance by applying it to the sensor data collected during the mobility study. This will provide us with evidence of whether the system is able to detect cognitive problems and disorientation based on state of the art sensors.

Later, a new study with people with dementia will be conducted where the aim is to test the system’s ability to provide adequate intervention strategies and assistance cues.