A validation & verification driven ontology: An iterative process

3.1.Case study description

The ontology supports a medical team composed of neurologists, who are specialized in rehabilitating patients with head injuries, in the context of the National Rehabilitation Institute (Instituto Nacional de Rehabilitación – INR, http://www.inr.gob.mx/r08.html), a Mexican hospital and leader in attending patients in rehabilitation services. INR also promotes research projects in several medical areas about rehabilitation. The medical team conduct one of the INR’s research protocols called the Cerebrolysin Research Protocol (CRP; INR, 2013). CRP focuses on investigating the functional, cognitive, psychological, and physical effects of the Cerebrolysin^® drug, as adjuvant treatment for several sequelae of Traumatic Brain Injury (TBI). The CRP team is constantly seeking patients with specific diagnosis conditions, who meet the CRP’s requirements and analyses whether they are candidates for neurological rehabilitation in an in-hospital program that lasts for a year. The medical team, who is specialized in neurology at INR and manages the CRP, needs to perform a specialized evaluation of the patients, who are sent from other hospitals with a preliminary diagnosis of TBI. However, these hospitals are the first-contact medical place after the patient has suffered an accident, and commonly the general physicians in charge are not specialized in making such specific diagnoses for the criteria evaluation. Currently, two situations can happen: 1) INR’s medical staff need to travel to the first-contact hospitals to perform the diagnostic analysis and considering the high number of hospitals with A&E services in Mexico City and the time it takes to get there, this becomes an unpractical activity; 2) The first-contact hospitals’ general physicians perform a preliminary diagnosis to send potential patients to the INR for the Cerebrolysin treatment. However, as they are not specialists in medical rehabilitation, such preliminary diagnostic, could be incomplete or wrong. Both situations cause the loss of candidate patients to be enrolled in the CRP, which affects the medical research and the rehabilitation of current and future patients.

Thus, in the Case Study, it is important to analyze whether the patient fulfills a set of criteria, which is followed by the INR’s medical team, to determine if the patient can be treated under the CRP (INR, 2013). Table 2 shows these criteria as a list of Inclusion Criteria and a set of Exclusion Criteria. The application of the inclusion/exclusion criteria follows an order which is determined by the INR’s medical team. In this Case Study, some of these criteria might be business policies (e.g., that the patient must be at least 18 years old), others might be business constraints (e.g., that the TBI is considered “severe” if the lesion time is between 1–6 months. Thus, the CRP (including the inclusion/exclusion criteria) along with the INR organization, will be taken as Business Information, which is used to identify the Business Policies (BP) and Business Constraints (BC; Fig. 2, Activity 2).

Fig. 2.

Phase I flow.

In this scenario, the ontology will be the means to organize and model the medical rationale required to process patients to be treated under the CRP. The KB (based on the ontology) will support the first-contact GPs at the hospital to perform the specialized diagnostic analysis, through the implementation of the medical criteria as knowledge rules. It is worth mentioning that before starting the ontology design, research was performed to identify processes that are oriented to developing application-oriented ontologies. As commented in Sections Introduction and Related Work, even there are several well-known processes, we were not able to find one which focused on developing ontologies to be used in an application setting, with frequent activities to perform quantitative validation (for measuring if the ontology and KB are truly answering the CQs, from the SMEs’ perspective), with a guide to applying the process, and that the resulting ontology can be used to set up a knowledge base, able to execute queries from a software application. Therefore, CODEP is inspired by many of these processes but fills the gap by proposing a new process that is structured, includes V&V, and can be driven by the CQs to create an application-oriented KB.

In the following Section 3.2, the description and application of CODEP in every phase are explained.

3.2.CODEP phases and activities

The objective of Phase I is to understand the domain to obtain the list of Competency Questions and Knowledge Rules that will be used to drive the ontology (see Fig. 2). The activities for this phase are as follows:

The problem identification of an ontology is key for avoiding the modeling of irrelevant aspects of the domain application, and the scope states the expectations that the ontology must be compliant. The problem identification and scope are defined by conducting several meetings between the knowledge engineer and the SME to identify what are the needs for the ontology. The SMEs are the experts of the domain and they can be end-users of the ontology. From this step, the KE identifies the actors, end-users, and stakeholders of the ontology. The Input is the Business Information related to the domain and context that describes problem elements. The Output is a Vision and Scope that clearly describes the subject, the problem boundaries, and the roles involved.

Case Study: Vision and Scope contains the ontology scope, which is “to support the INR’s medical staff in the analysis of patients who have just suffered an accident causing a TBI and who are being attended at first-contact hospitals”. The aim is to: 1) assist the first-contact hospital’s general physician during the patient evaluation to obtain a close diagnostic and to determine the CRP candidates; and 2) support the INR’s neurology researcher to identify candidate patients of the CRP protocol without physically attending the first-contact hospitals. The involved actor is the neurology specialist (SME). After stating the ontology scope, the target ontology was named CErebrolysin Research PRotocol Ontology (CERPRO).

The objective is that the KE gets deep knowledge about the concepts that need to be included in the ontology and the level of expressiveness. Specifically, 1) the application domain context (the institution features, the company’s business policies and constraints from relevant standards, and a glossary of terms) is studied; the doubts should be solved with the involved people in the domain (staff, customers, or technicians); 2) the set of scenarios (including the tasks that are performed by the scenario’s actors) where the ontology is applied are identified; here each actor can have a set of scenarios or many actors can share them. The Input is the Vision and Scope, from Activity 1. The Output is three elements: 1) the Business Features that describes the institution description and a list of significant domain terms; 2) the Business Scenarios, which includes the scenario set per role; and 3) the Business Policies and Constraints.

Case Study. In this activity, the KE obtains knowledge from the Cerebrolysin Research Protocol (CRP; INR, 2013) and from the Business Information to get the Business Features: medical terms, specific diseases and pathological conditions that a patient could present, the supplied drugs, specific tests, accident types, and cognitive/physical disabilities and the scales to measure them. The KE obtained the Business Policies and Constraints, from the INR documentation and the inclusion/exclusion criteria from the CRP. The Business Scenarios were obtained through several meetings with the Chief Medical Doctor of the CRP team to identify the ontology end-users (actors) and usages (scenarios), based on the Ontology Vision and Scope, from Activity 1. The meetings were structured in a question-answer format, where the questions were sent to the medical team before the meetings. After the meetings, specific inquiries were further addressed via email. As a result, the set of scenarios and actors are summarized in Table 3. These scenarios will be helpful to state the end-user requirements (CQs) in further activities.

This activity focuses on capturing the set of CQs. The CQs are the questions that SMEs need to answer to support their daily work. They are usually the queries that need to be supported with the ontology, and they are defined as focusing on obtaining knowledge (explicitly or implicitly) from the KB. Usually, CQs are in the mind (tacit) of SMEs, and the KE works with SMEs to make them explicit. To be able to explicitly define the CQs, the KE analyses as Inputs: The Business Features, Business Scenarios, and the Business Policies and Constraints from Activity 2. The Output of this Activity is the List of Competency Questions (this includes the refined CQs in iterations).

Case Study. Applying this activity results in the specification of CQs as the medical team needs to identify patients with specific diagnosis conditions, after analyzing the Business Features and Business Scenarios from Activity 2. The CQs have been defined to answer whether a patient satisfies the Business Policies and Constraints (the inclusion/exclusion criteria from the CRP). Thus, the SP will use the CQs responses to support the evaluation of the medical condition of the patients, who were initially diagnosed with TBI in the first-contact hospital by the GP. Table 4 shows an excerpt of the 14 CQs (List of Competency Questions) dictated by the SME, indicating dependencies among them. The Dependency column indicates which CQ/CQs must be modelled in the ontology, before the CQ indicated in the ID-CQ column; this is simply the sequential order for modeling the CQs. For example, CQ-1 must be implemented before CQ-11, because this latter question asks for previous pathologies to the “current” TBI. This implies firstly implement the CQ to find out if a TBI has occurred as an accident result, in another way, CQ-11 does not make any sense outside this context.

This activity aims to identify constraints or restrictions in the knowledge domain, such as regulations, the vision, and scope document (from Activity 1), and relevant domain documentation provided by the SMEs. Constraints are defined as Knowledge Rules to implement the business logic in the ontology domain, and they are defined as axioms in the ontology, which can be verified by a reasoner (an intelligent software application). In CODEP, the Knowledge Rules are extracted from the Input: Business Scenarios, Business Features, and Business Policies and Constraints (from Activity 2); but it is recommended that the KE refines them along with the SME. The Output is the Knowledge Rules, which are implemented in the KB in Activity 12, to apply the ontology quality metrics (e.g., for testing accuracy of the CQs responses). The knowledge rules identification can be done by defining use cases, identifying the flows in such use cases, and complementing them with activity or business process modeling diagrams (e.g., with UML or Business Process Modeling Notation).

Case Study. The Knowledge Rules for the case study are extracted from the Business Policies and Constraints, for the identification of the patient diagnostic to be enrolled in the CRP. There are several strict conditions in the CRP (Inclusion/Exclusion Criteria, IC/EC), which prevent a patient from being considered as a candidate; such conditions are the Business Policies (BPs) and constraints (BCs) from Table 2, which drives the application of the Cerebrolysin treatment to patients. Thereafter, the neurologist applies the IC/EC criteria following a specific order, as a checklist to the patients for identifying candidates for the CRP. Thus, the Knowledge Rules are based on the IC/EC, e.g., the knowledge rule KR-3 describes that the patient diagnosis must present sequelae originating from the TBI (IC-3), but with no previous pathologies to the current TBI (EC-3). The medical reason is that the rehabilitation process could be affected by the Cerebrolysin drug or the functional performance evaluation. Thereafter, KR-3 states to verify these two criteria simultaneously, as the GP does in the first-contact hospital; and if one of these two criteria fails, then the patient is not included in the CRP; otherwise, the knowledge rule application carries on to the next knowledge rule, which is KR-4.

Before the creation of the ontology, we recommend defining the ontology validation criteria. The validation criteria aim is to check whether the CQs respond to the SMEs’ expectations, once an ontology is set up in a knowledge base. For this purpose, the KE will get the CQs’ responses through querying the KB, and the SME will indicate if they believe whether the validation criteria are met or not after evaluating such CQs’ responses. Thus, the KE and SMEs work together to set these criteria, and they should reflect the functional achievement (e.g., completeness and accuracy of the CQs) but also the non-functional one (e.g., comprehensibility and efficiency). During the validation criteria definition, the communication between the KE and the SME must be very active. Depending on the ontology objective and the SME expectations, different validation criteria can be defined as a set of metrics. In the literature, there are already metrics that can be reused for quality assessment, such as the external quality and quality in use from the SQuaRE model in the series ISO/IEC 2502n (ISO-25023, 2016; ISO-25022, 2016). However, the KE must adapt them to reflect that the requirements specification for CODEP is represented as CQs. The input for this activity is the List of Competency Questions (from Activity 3); and the Output is the List of Validation Criteria, which will be used for the ontology validation (Activity 15).

Case Study: The INR’s medical team is the target audience and the SME is the neurologist specialist, who will perform the validation. The List of Competency Questions (from Activity 3) is used to create the validation criteria for CERPRO, as follows: 1) the KEs define a set of possible validation metrics which are presented to the INR’s medical team, 2) the medical team selected the relevant metrics based on their needs to obtain automatic support for the simultaneous identification of patient’s diagnosis from several medical datasets, and 3) the SMEs were asked to set expected values for each metric, based on their priority of satisfaction (Expected Medical Team’s value). To make this communication fluent, emails, recordings, and minutes from one-hour meetings were used, which were held for each version revision.

The List of Validation Criteria for CERPRO included: CQ completeness (since the medical team is interested in implementing the whole CQs set), CQ response accuracy (since the medical team expects accuracy in the responses when identifying diagnostics), and CQ response comprehensibility (since the medical team needs to comprehend the responses from the knowledge base with no complication in the technical argot). Each metric has a definition, a metric to collect values to prove whether the ontology expectations are fulfilled, the range of meaningful values (e.g., 0 the worst value and 1 the best value), the number of CQs needed to calculate the metric and the medical teams’ expected value.

aCQ: Zero the worst accuracy, one the best. If aCQs is near zero then the medical team’s satisfaction is under reasonable expectations, on the contrary, if it is near 1, the satisfaction is high.

i: Number of the CQ being measured.

n: Number of the evaluated CQs (14 for this case study) in a given iteration.

ACQi= Accuracy measurement for each CQi, from 0 to 1, being 1 the better.

TCQs: The total number of CQs, which were provided by the medical team.

Expected Medical Team’s value: 100% that is normalized to 1 for this case study, since the medical team expects a high accuracy.

cCQ: If cCQ is near zero then the CQ implementation is poor, on the contrary, if it is near 1, it is the best.

i: Number of the CQ being measured.

n: Number of the evaluated CQs (14 for this case study in the last iteration) in a given iteration.

ICQi: The score for the CQ answer (provided by the ontology/KB). This is given by the end-user (0 or 1).

TCQs: The total number of CQs, which were provided by the medical team.

Expected Medical Team’s value: 100% that is normalized to 1 for this case study, since the medical team expects getting responses from the KB for the whole set of CQs that they defined in conjunction with the KE.

rCQ: If rCQ is near zero then the CQs’ responses’ comprehensibility is poor, if it is near to 1 it is high.

i: Number of the CQ being measured.

n: Number of the evaluated CQs (14 for this case study in the last iteration) in a given iteration.

RCQi: Measures how comprehensible is the CQ’s response through the query to the doctors.

Expected Medical Team’s value: 90% that is normalized to 1 for this case study, since the medical team can accept a moderate technical argot in the provided responses from the KB.

Thus, in Activity 15, these metrics are applied to obtain feedback about CERPRO’s quality from the SME’s perspective, to identify deviations and to correct them, till aCQs,cCQ and rCQ become nearly to the end-users expected values.

Milestone I: The Ontology Vision and Scope are obtained.

The objective of this phase is to build an ontology model and its axioms (see Fig. 3). The activities for this phase are as follows:

Fig. 3.

Phase II flow.

This activity aims to extract the relevant domain nouns and business actions from the outputs of previous activities. The Input to this activity is the Business Scenarios, the Vision and Scope document, the list of Competency Questions, and the Knowledge Rules. Nouns in these inputs will be the initial list of the ontology concepts and the verbs will be relationships among the concepts. This is a highly creative task and can involve interviewing further the SME and additional information could be provided to the KE related to the identified concepts. The Output of this activity is the Initial Concept List.

Case Study: The following are used: the CQs, the inclusion/exclusion criteria (which are the Knowledge Rules), the scenarios’ list, and Vision and Scope to identify the relevant concepts. A table was created for the initial domain concepts. After obtaining initial concepts, new concepts were introduced which are not present in the inclusion/exclusion criteria. For example, consider the criterion IC-3 (from Table 2):

“Patients with a sequela diagnostic caused by the TBI.”

From this statement, the following nouns can be identified as concepts: Patient, Diagnostic, Sequela, and TBI. Then, more information can be discovered about the features which describe Patient, Diagnostic, and TBI through interviewing physicians. For the Sequela concept, the SP has further required to detail the classification of a series of illnesses produced as a consequence of the TBI. Relevant concepts were also identified for the application domain analysis of the target ontology. For example, the Glasgow Outcome Score (SNOMED International, 2015) which is a scale for measuring the damage caused by brain injuries (e.g., cerebral traumas), divides patients into 5 groups: Death, Persistent vegetative state, Severe, Moderate and Low disability. Thus, the Initial Concept List is produced to be included in the ontology.

As a result, new relationships and concepts not identified in Activity 6 can be discovered and added to the Initial Conceptual List. In this activity, the KE can identify several potential third-party ontologies. In this case, the KE needs to evaluate which one is more appropriate based on the initial concepts identified and the scope. The KE maps the third-party ontologies’ concepts with the initial concepts. The SME needs to work with the KE to ensure that the semantics of the concepts. Therefore, the Input is the Initial Concept List and the Output is the reusable Third-party Ontology/ies, which are selected.

Case Study: An exhaustive search in medical relevant ontologies was performed, by using searching the initial concepts from Activity 6 about the CRP and the required medical concepts for the CQs. As a result, OpenGalen (OpenGalen Foundation, 2012) was identified which provides both: 1) an acceptable taxonomy of medical concepts and procedures and 2) a proper expressivity to support the CQs. Another relevant ontology identified is SNOMED-CT (SNOMED International, 2015). However, the 14 CQs’ goals are not met with SNOMED-CT and its expressivity is poor in terms of the OWL-DL potential. Therefore, several OpenGalen’s modules were used as a basis to incorporate medical terminology already agreed by an extensive medical research team. In the aim of finding the reusable OpenGalen modules, the semantics of the concepts of the Initial Concept List were matched to the terminology available in OpenGalen. The neurologist team at INR was involved in ensuring the semantics of the medical terminology are valid.

This activity aims to synthesize the concepts obtained from the Third-Party Ontologies (Activity 7), with the concepts that have been identified in Activity 6 to eliminate duplicate concepts from the re-used modules. Therefore, the Input of this activity is the Initial Concept List and the reusable Third-Party Ontologies. This activity must be carefully done since third-party ontologies could be exhaustive, and it is necessary to double-check if each of the concepts from the CQs is already included in the Third-Party ontologies selected. We, therefore, recommend mapping the concepts of the reusable ontology to the ones in Initial Concept List. The output is a Refined Concept List which will be used to model the ontology in Activity 10.

Case Study: Table 5 shows an excerpt of the resulting conceptual synthesis, including some potential modeling actions for creating the concepts in the target ontology. For example, for IC-3, Patient, Diagnostic and Sequela concepts can be modeled as classes in CERPRO, where Patient inherits from the class Human, which is already defined in OpenGalen. This axiom connects CERPRO with OpenGalen since Patient must be defined in CERPRO and it is related to Human (by inheritance) from OpenGalen. Similarly, TBI is already included in OpenGalen through the concept HeadTrauma. This analysis strategy was followed for each concept as in Table 5 (Refined Concept List for CERPRO).

This activity aims to allow the KE to set quality indicators about the ontology design to be verified in later activities. The KE sets quality indicators related to the design of the ontology. In this way, the design of the ontology is driven by these quality indicators and checked after the modeling stage. Quality indicators are set to assure that the ontology follows quality standards. It allows the KE to verify that the ontology has been designed and implemented according to the quality indicators and if not, the KE can refine the ontology. Quality indicators are different from validation metrics defined in Activity 5. Validation metrics are end-user-oriented and are checked with users to achieve end-user expectations. Quality Indicator Metrics can be adopted from already defined ones available in the literature such as by Gangemi, Catenacci, Ciaramita, & Lehmann (2005) and by Tartir, Budak Arpinar, Moore, Sheth, & Aleman-Meza (2005). However, these could need to be adapted to be defined in terms of CQs. The Output of this activity is the Quality and Metric Indicator List, including the metrics per indicator, to be used for measuring the quality of the ontology.

Case Study: Several quality indicators were defined such as Modelling Completeness, Semantic Consistency between Models, and Model Expressiveness:

comCQ: The average of total concepts in the ontology from the n evaluated CQs, <1 the worst, ⩾1 the best.

i: Number of the CQ being measured.

n: Total number of CQs.

MCi= Number of modelled concepts in the ontology from the CQi.

CQCi: Number of concepts that are identified from the CQi.

SemCM: 1 the best (none of the concepts are modelled in more than one ontology),

2 the worst (concepts are modelled in both ontologies).

i: Number of the CQ being measured.

n: Total Number of CQs.

MCCi: Number of concepts of the CQCi that are modelled in the Ontology (CERPRO).

MCOGi: Number of concepts from the CQCi that already exists in the third-party ontology (OpenGalen).

CQCi: Number of concepts that are identified from the CQi.

pR: 0 the worst, ⩾1 the best, indicating that the concepts have at least a relationship (a data or an object property, or both). The condition: Oi or Di or Hi⩾1, restricts that each CQi has at least one relationship of any kind, avoiding leaving unconnected classes (unless the domain requirements specifically ask this).

i: Number of the CQ being measured.

n: Total Number of CQs.

Oi: Number of modelled object properties originated from the CQi.

Di: Number of modelled data properties originated from the CQi.

Hi: Number of modelled inheritance relationships originated from the CQi.

MCi: Number of the modelled concepts from the CQi description.

The metrics will be used in Activity 13 to measure the CERPRO’s quality, to improve the quality design through the iterations, till the 3 metrics are near 1. For example, if pR is near 1 then it implies that a variety of possible elements (such as object, data properties, inheritance, etc.) have been strongly used to express the identified elements from the CQs (since this metric is based on the identified CQs’ elements). If pR is near 0, then this would mean that the resulting ontology model is poor in terms of modeling expressiveness for the CQs’ elements.

As this activity is part of an iterative process, the KE selects a set of CQs that will create an increment of the ontology model. Therefore, the input of this activity is the List of Competency Questions, the Knowledge Rules and the Refined Concept List, and the Ontology Model from previous iterations. Each time an increment of the model is generated that satisfies a set of CQs, another set of CQs are selected from the Refined Concept List in the next iteration and the previous increment is extended to support a new set of CQs. In the first iteration, this activity includes selecting the language for the ontology modeling and design, and a tool for an ontology graphical representation. A graphical representation is advisable as it allows the KE to discuss the model with SMEs. An example of an ontology design language is OWL-DL (W3C-OWL 2, 2012), and examples of ontology modeling tools are yED (yWorks Software, 2020), which has a plug-in to model OWL-DL ontologies, and Enterprise Architect (SparxSystems, 2013), which uses UML to depict the OWL-DL axioms. Each iteration generates an increment, which is called the Ontology Model (Output), and the last iteration will generate the increment that satisfies all the CQs.

During the activity, concepts and their relationships are represented in a graphical model, before designing the ontology, to create a consensus among the participants about the ontology axioms, knowledge rules, and expressiveness level. Also, the graphical ontology model is a means to visually verify if all the concepts and their relationships from the Refined Concept List (Activity 8) are being considered in the model.

Due to the iterative nature of CODEP, Activity 10 can be iteratively be updated with Activity 11–12, for modeling, and designing the ontology, and defining the knowledge rules.

Case Study: For this Activity, we selected yED (yWorks Software, 2020) as the tool for the ontology graphical representation, since this tool has a plug-in for modeling OWL-DL ontologies. Fig. 4 and Fig. 5 show the CERPRO v5 model (as the Output of Activity 10), generated based on the List of Competency Questions, the Knowledge Rules, and the Refined Concept List. It can be observed that there is an asterisk in several of the classes (e.g., in Human); this notation is used to indicate that such a class belongs to OpenGalen. From iteration 2, this model was used to improve the CERPRO modeling in each iteration.

Fig. 4.

CERPRO model in yED, showing the INR Inclusion Criteria.

Fig. 5.

CERPRO model in yED, showing the INR Exclusion Criteria.

In this activity, the axioms already specified in the model from Activity 10 are implemented. The Input for this activity is the Ontology Model and the Ontology Beta (from iteration 2, which includes the axioms in a formal ontology language, implemented through an ontology editor). The objective is to constantly verify that the designed ontology contains all the model expressiveness (inheritance, equivalence, disjoining, etc.), stated in the model. Several graphical modeling tools used in Activity 10 can automatically generate the ontology axioms; however, the KE must manually check that the generated axioms contain all the required expressiveness. This manual check must be done each time the model or ontology is edited. This expressiveness could consist of iteratively defining a set of axioms (the TBox in the ontology OWL language), the domain concepts and the properties (relationships) between them, cardinalities, equivalences, functions, and inheritance. It is possible to use some already known Ontology Design Patterns (ODP; Gangemi & Presutti, 2009).

In this step of this Activity, the ontology consistency checking must be done as many times as the reasoner reports inconsistencies in the ontology until there are no more reported errors. The Output of this activity will be an Ontology Beta version.

Due to the iterative nature of CODEP, Activity 11 can be iteratively be updated with Activity 10 and 12, for modeling, and designing the ontology, and defining the knowledge rules.

Case Study: We have used Protégé 5 (Protégé, 2020) which includes an editor for OWL-DL (W3C-OWL 2, 2012), the selected ontology language, and yED for the graphical modeling (since this allows creating graphical ontology models for OWL-DL ontologies). Since there are 2 different tools, each time the model or axioms are edited, special attention had to be taken to check the consistency between the graphical and axiom models. Fig. 6 shows an excerpt of the CERPRO v5 ontology (Output), in Protégé, describing the OWL axioms for the class Patient. The classes for Diagnostic, Sequela, the Glasgow, and RLAS scales can also be observed (the classes from the Refined Concept List from Activity 8); the axioms for these classes are displayed when the class is selected in Protégé. All the CERPRO ontology axioms are defined in the underlying ontology file.

Fig. 6.

CERPRO in Protégé, displaying the OWL axioms for the patient class, and patients as instances (CP_P1, CP_P2, etc.).

This activity aims to receive feedback on the ontology design as it is easier and less costly to make any changes/updates at this stage than in later activities in the process. In this activity, the KE shows the ontology model and axioms to the SME. For this purpose, it is recommended to use a graphical ontology representation to facilitate the comprehension of the SME. Thus, this is a qualitative validation – as it captures the SME’s comprehension about missing or misrepresented concepts, attributes, and relationships.

After performing this activity, a decision can be made either, to perform more iterations to other activities in this phase, specifically activities 6, 10, and 11; or not. The input of this activity is the Ontology Beta and the output is the decision about performing more iterations (when some concepts are still missing in the ontology) or going forward to Activity 13 to implement the knowledge rules.

Case Study: The KE walked through the ontology model with the neurologists. During this exercise, many misconceptions were corrected in the model. For example, in iteration 5 during this validation activity, the neurologist detected that the class TraumaMechanism (which indicates how trauma can occur e.g., downfall, ballistic, physical aggression, etc.) had missed concepts. Specifically, more classes were needed to model whether the trauma occurred through an accident (which might be either intentional – e.g., in a downfall, or not intentional – e.g., physical aggression and whether it has an internal origin due to a pre-existent sickness – e.g., epilepsy, heart attack, etc., or an external one – e.g., by a car accident.

Fig. 7 shows how the ontology was changed after this validation. The trauma accident concept was not detected by the KE from the CQs and domain analysis. This situation can happen because SMEs are not aware that they need this in the model early on, until the KE walks them through the model.

Fig. 7.

The (a) shows the previous version of the TraumaMechanism class; (b) shows the same class but it was renamed as AccidentMechanism, with additional classes for modeling whether the trauma was an accident that might be intentional (e.g. a third-person pushed the patient) or NonIntentionalAccident (e.g., a person falls due to a sickness).

In this Activity, the KE implements the Knowledge Rules (from Activity 4) in a rule language (e.g., for OWL ontologies), by selecting the best strategy according to the objective of the ontology (e.g., to be used for inferencing knowledge, or querying). Three strategy options are proposed to implement KRs: Option 1) for evaluation with rule engines, such as Jess (Sandia National Laboratories, 2020) or RuleML (RuleML Inc., 2020); Option 2) as injected axioms in the target ontology, which will be dynamically and periodically evaluated with a reasoner, e.g., Pellet (Clark&Parsia, 2011) with Semantic Web Rule Language (SWRL; W3C-SWRL, 2004); and Option 3) as queries to the KB, e.g., in Jena Fuseki (The Apache Software Foundation, 2020) by using a query language, such as SPARQL Query Language for RDF (SPARQL; W3C-SPARQL, 2008). Options 1 and 2 can be set up for an automatic evaluation, whilst option 3 is only executed by direct requests to the KB, through a SPARQL end point, or with an information system. For any selected strategy, the Knowledge Rules will be implemented in the Ontology Beta, per each CODEP iteration. Thus, the Input of this activity is the Knowledge Rules and the Ontology Beta (this only from iteration 2); and the Output is the Ontology Beta (with the implemented knowledge rules).

Due to the iterative nature of CODEP, Activity 13 can be iteratively be updated with Activity 10–12, for modeling, and designing the ontology, and defining the knowledge rules.

Case Study: The knowledge rules from Table 6 were implemented as SPARQL queries to be run in the KB. This decision was made because the INR’s medical team required to observe the CQs’ responses when performing the validation (later on, in Activity 15); otherwise if the rules were run in the background through a rule engine or a reasoner, it would not be possible to observe the results in real-time by the SME (however, the rule implementation as in Options 2 is part of the on-going work, further the validation task). To illustrate this, R-3 (from Table 6) is included in Table 7, where both IC-3 and EC-3 implemented in SPARQL, need to be simultaneously executed and evaluated in true since R3 dictates this (Table 6). At the end of this Activity, CERPRO Beta contained all the Table 6 knowledge rules, implemented as SPARQL queries. It is pointed out that these queries were reviewed and edited in each of the 11 iterations for CERPRO, which implied reviewing/edit CERPRO as well.

Milestone II: It is obtained the Ontology Beta.

The objective of Phase III is to implement a Knowledge Base from the ontology that is verified and validated (see Fig. 8). The activities for this phase are as follows:

Fig. 8.

Phase III flowchart.

This activity aims at verifying the ontology design by measuring its quality, according to the defined Quality and Metric Indicator List (Input from Activity 9). The Verification definition in CODEP is based on the one provided in SWEBOK (IEEE-SWEBOK, 2014), as follows: “Verification is an attempt to ensure that the product is built correctly, in the sense that the output products of an activity meet the specifications imposed on them in previous activities”. In this context, this activity verifies the ontology design in terms of the quality indicators that were defined in Activity 9.

The verification is run by applying the metrics included in the indicators to obtain measurements, which will give insights about the ontology design, and then to correct deviations if the results are under the expectations. This will guide the KE to review and improve the design of the ontology and conduct further iterations if needed. Examples of improvements that can be performed are the addition of missing concepts, axioms, and properties and changes in the hierarchies. Metrics that can be measured are model expressiveness, completeness, and semantic consistency between the target ontology and reused ontologies. The Output will be either: The Verification Results and the verified Ontology Increment; or the decision to go back to Activity 10 for a new iteration. Otherwise, if the ontology design quality is reached, then the next Activity 15 is performed.

Short iterations from Activity 10–14 might be done, to adjust the ontology modeling and design, until reaching the expected quality as stated in Activity 9.

Case study: The Quality and Metric Indicator List (from Activity 9) is applied to measure the CERPRO design quality: Model Completeness, Semantic Consistency between Models, and Model Expressiveness. Obviously, as KEs, our ideal expectation is to get these indicators as near as possible to 100%, however, without applying metrics the perception about reaching such expectations would be only subjective. Thus, these selected metrics gave us quantitative data to measure how close are we getting to 100%, and insight into what specific points should the ontology of CERPRO be improved. In the following, we explain how we made use of the Metrics:

Model Completeness measures if all CQ concepts are implemented as concepts in the ontology. For example, for CQ1 (see Table 8), the number of concepts that are identified is CQC1=5 and the number of modelled concepts is MC1=8. Therefore, Completeness for CQ1 is the ratio 1.6 indicating that 5 concepts come from CQ₁ but 8 concepts were modeled. This means that 3 concepts are not traced back directly from the CQ1 description, however, these added concepts are needed to complete the design (as the sub-types, AccidentMechanism, IntentionalAccident, and NonIntentionalAccident). For every CQ (CQi), the completeness measurements are similarly calculated. The last row in Table 8 shows the total of the metric: comCQ=1.1; this indicates that all identified concepts from the CQs were modelled in the ontology, including some additional concepts (required for the CQs modeling as commented). According to the indicator range, it can be concluded that CERPRO is in good shape in terms of completeness (however the result shown was obtained after Iteration 11). Otherwise, when comCQ was near 0 (meaning that CERPRO was poor in terms of the CQs concepts modeling) more CODEP iterations were required from Activities 10–13 until comCQ became near to 1.

Table 8

(Excerpt) metrics application for measuring the model completeness indicator

The advantages of using the Model Completeness metric in an iteration for driving improvements are as follows:

Table 9

(Excerpt) previous model completeness metric calculation, where CQ₂ and CQ₁₃ metrics had lower figures

Table 10 presents the results for Semantic Consistency between Models indicator measurements. In this table we have defined columns for: the CQ description, the Identified Concepts in the CQ (to document the relevant concepts from the CQ text, Modelled Axioms in Open Galen, (to document the axioms that already exist in OpenGalen, which model the concepts from the CQ), Modelled Axioms in CERPRO (to document the axioms that model the concepts from the CQ in CERPRO), and the Metrics (for each of the 14 CQs to calculate SemCM according to Equation 5 from Activity 9). The metric calculation is based on the ratio (MCOGi+MCCi)CQCi, which measures the number of modelled concepts in the reused ontology (OpenGalen) and the target ontology (CERPRO) and takes an average by dividing the identified concepts from the CQi. Thus, if the metric is near 1, then this means that the concepts are not duplicated. If the metric is near 2, then this means that potential axioms are duplicated in the ontology. For example, for CQ9 the Semantic Consistency is 1.23 (MCOG9+MCC9)CQC9, where MCOG9=11, i.e., 11 concepts of the CQ are modeled in OpenGalen (column Modelled Axioms in OpenGalen), MCC9=5, i.e., 5 concepts of this CQ are modeled in CERPRO (column Modelled Axioms in CERPRO), and CQC9=13, i.e., there are 13 concepts identified from the CQ (column Identified Concepts in the CQ). This considers eliminating those concepts that were already counted in previous CQs (e.g. Patient was already counted in the CQ1 calculation). This metric considers that several concepts are introduced into the ontology for modeling purposes such as StimulatingTreatment which is not found in CQ9, but is needed for creating a hierarchy for StimulatingDrug which also requires Citicoline and Cerebrolysin. According to the metric definition in Equation 5, since the value is very near to 0, the resulting ratio 1.23 is good after performing 11 iterations. The rest of the CQi measurements for the 14 CQs are similarly calculated (omitted for space reasons) and the resulting average Semantic Consistency between Models is indicated in the last row in Table 10, which was the resulting number in iteration 11, meaning that this metric indicates that there are not duplicate concepts in the ontology.

Table 10

(Excerpt) metrics application for measuring the indicator: semantic consistency between models

Table 11

(Excerpt) previous semantic consistency between models metrics calculation, where CQ12 has repeated axioms

The advantages of using the Semantic Consistency between Models metric in an iteration for driving the improvements are as follows:

Table 12 presents the results for the Model Expressiveness indicator measurement. The pR measures if the model is presenting at least the same number of relationships against the modelled concepts from the CQs (but it is expected to find this number much larger). For each CQi, the number of object/data properties and the inheritance relationships are counted, then this number is divided by the modeled classes given by the specific CQi, to obtain a ratio that indicates if on average each class has at least a relationship. The result is a balance between the ratio counting for each CQ (one could have no data/object properties just inheritance relationships, but other CQs could compensate this number and have no inheritance relationships), but each CQ’s ratio will give a deeper detail about the balance between the modeled CQ’s concepts and the relationships in the model. Thus, if pR is less than 1, it indicates that the model requires more expressiveness, and therefore more CODEP iterations are required from Activities 10–12 until pR becomes equal to 1 (or greater). For example, for CQ1, there are 3 object properties (O1), 2 data properties (D1), 6 inheritance relationships (H7), and 8 modeled classes (MC1), giving a total of (O1+D1+H1)MC1=1.37. The resulting total pR is 1.13 in the last row in Table 12, meaning that every concept has on average an axiom relationship.

Table 12

(Excerpt) metrics application for measuring the model expressiveness indicator

The advantage of using the Model Expressiveness metric in an iteration for driving the improvements is to identify vertical ontologies (mostly hierarchies). This metric was very helpful in identifying whether a CQi is being modeled with only hierarchies with barely data/properties (very vertical shape), in which case, the ratio for the given CQi will be near 0. This happens when a CQi requires a hierarchy and the class definition lacks axioms as data/object properties, this might be correct if the CQi description requires this, but the metric will put quantitatively the vertically-modelled CQi and make it possible to review that part of the ontology and to correct the model. For example, a previous calculation of the CQ1′s ratio (Table 13), (O1+D1+H1)MC1 gave 0.83, which indicates that there are classes without relationships in that section of the ontology (see Fig. 4, left-bottom side). Initially, in iteration 1 we had the AccidentMechanism and its hierarchy connected to PathologicalPhenomenon, through an object property. However, the result of this metric made us review the CQ description and the modeling. Thus, we decided to create the accident mechanism type hierarchy, connected as well with the inverse object property, AccidentMechanismOf, and to differentiate the NonIntentionalAccident from the IntentionalAccident, as the Downfall concept (since the SME was interested in an intentional downfall, and not caused due to a previous pathology). The resulting CQ1′s ratio, with these corrections in the next iteration, was 1.37, which means a more balanced ontology model (as observed in Fig. 4). Thus, this metric calculation made us review our knowledge in the domain, to figure out if some concepts or relationships were badly understood, but not exactly missing, which would lead to a model correction in terms of expressiveness.

Table 13

(Excerpt) previous model expressiveness calculation, where CQ1 has a low ratio

Therefore, after iteration 11, CERPRO achieved the verification expectations in terms of these 3 metrics.

The aim is to create a KB to test if the ontology is properly answering the CQs. This is entirely a software installation task. Thus, the Ontology Increment is used (Input from Activity 13) to set up the KB platform in servers, for creating a dataset to be populated with selected trial data for creating triplets (ABox for OWL ontologies). This allows the execution of the knowledge rules (from Activity 12) and the queries (created from the CQs) for validation. The Output of this Activity is the Knowledge Base with a dataset.

Case Study: Apache Jena (The Apache Software Foundation, 2020) was chosen as the platform to create the KB, and Fuseki was chosen as the SPARQL End Point to remotely query the KB (from a web browser and a software application). The KB called “CERPRO-KB” was created, which can be found at http://liim.izt.uam.mx:8080/fuseki/ (Fig. 9). For this, the Ontology Increment of CERPRO from Activity 13 was uploaded, producing the dataset, CERPROv-5 (by the time this paper is written). With the support of INR’s medical team, the CERPRO-KB was created with anonymized data from patients (see Table 14). This data describes different relevant diagnostics for the medical team, intending to test the KB’s responses with already known diagnostics. In an upcoming scenario, INR will keep the knowledge base on its premises to preserve personal data confidentiality.

Fig. 9.

The SPARQL implementation of CQ-1 and the CERPRO response.

Table 14

(Excerpt) trial data with patients data

This activity aims to validate whether the Ontology Increment is answering properly the CQs from the SME’s perspective in a quantitative perspective. The Validation definition in CODEP is based on the one provided SWEBOK by (IEEE-SWEBOK, 2014), as follows: “Validation is an attempt to ensure that the right product is built – that is, the product fulfills its specific intended purpose”. In this context, this activity validates whether the ontology fulfills the CQs that were defined in Phase 1, in terms of querying the KB and evaluating the results according to the ontology validation criteria, through the metrics that were defined in Activity 5.

The Input is the KB with a dataset and the List of Validation Criteria (from Activity 5), which are applied to measure whether the SME expectations have been satisfied or not. For each metric, the SME is asked to indicate their Expected value which indicates when they are satisfied with the result of the metric.

The Output of this activity is either: 1) the Validation Results, the Ontology Release version and the Knowledge Base Release, then, ending the CODEP process; or 2) the decision to perform another short iteration from Activity 10–15, to update the ontology according to the validation results if the ontology does not fulfill the criteria. Due to the iterative nature of the process, it must be ensured that any changes performed to the ontology during an iteration do not affect the obtained satisfaction when answering the CQs in the previous iteration. Therefore, the final iteration must include all the CQs, already validated.

Short iterations from Activity 10–15 might be done, to adjust the ontology modeling and design, to respond to the CQs as the SME expects.

Case Study: We defined a protocol for performing the validation activity (see Section Validation Protocol for details of the metrics application and results) to make sure that the SME’s needs about the CERPRO functionality and responses in the medical diagnostic identification are satisfied, according to their expectations stated in the Validation Criteria. This protocol applies the List of Validation Criteria that was defined for the case study in Activity 5, as follows: 1) CQ completeness (since the medical team is interested in implementing the whole CQs set), 2) CQ response accuracy (since the medical team expects accuracy in the responses when identifying diagnosis) and 3) CQ response comprehensibility (since the medical team needs to comprehend the responses from the knowledge base with no complication in the technical argot). The validation results were used for further improvements of CERPRO by undergoing iterations of CODEP (from Activity 10–15), which generated new increments of the ontology, till the validation criteria are satisfied near 100% of the expectations. We finally produced the CERPRO-v5, with the validated quality in iteration 11.

Milestone III: The Ontology and Knowledge Base Release is obtained.

6.1.Comparing CODEP’s features to other ontology development processes

This section discusses the CODEP’s features in comparison to other available approaches in the literature for ontology design. To compare these approaches, we have used the features described in Table 16.

From Table 17, it can be observed that many approaches do not mention involving the user or domain experts in the process. Those that do, involve users in activities acquiring knowledge about the domain. Two approaches in addition to CODEP do involve the user invalidating the ontology or knowledge base to ensure that the user is satisfied and collect/update the ontology and define new CQs. On-To-Knowledge and Diligent allow the user to propose changes to the ontology, therefore they can be viewed as the user to participate in the Ontology Building/validation.

In terms of approaches supporting validation and verification activities, only 3 approaches include both, the rest include one or the other. CODEP is the only process that includes validating the ontology quantitatively using metrics. The validation in CODEP is performed in two phases: one where the ontology model is validated and then where the knowledge base is validated with the users/domain experts.

It can be noticed that there are few approaches where the ontology is driven by CQs (3 approaches and CODEP). From the 3, only 2 support the development of a knowledge base. But it can also be observed that only approaches that are CQ-driven can develop knowledge bases. This demonstrates the importance of these criteria when suggesting processes for creating ontologies that can be applied.

From the evaluation in Table 17, it can also be observed that none of the approaches propose well-structured processes to be followed, with clear activities and roles. This is one of the factors that motivated us to define the CODEP process, as we could not identify any approach from the literature which can be followed.

6.2.CODEP further evaluation plans

CODEP includes 3 main phases, each with several activities. The phases cover the life cycle of ontology development. In addition, the end-user is highly involved in most of the activities. This has been appropriate for developing the CEPRO ontology where the usage of the results of the ontology are critical to the health of humans and ensure that medical doctors (users) trust the quality of the diagnosis produced by the ontology. The effort of following the CODEP activities can be high from the perspective of the knowledge engineer and the users (domain experts). In certain kinds of projects, a trade-off should be considered between applying all CODEP activities and the effort required without influencing the quality of the produced artifacts (ontology, knowledge base, etc.). In some cases, not all activities have to be applied. In our further work, to provide better guidance on the application and adaptation of CODEP, we can design experiments to evaluate which activities can be skipped and which activities must be kept for certain domains or characteristics of end-users.

We also plan on conducting further studies to evaluate other aspects of the performance of the CODEP process and the quality of the ontologies it produces. One of the aspects that would be of interest to study is bottlenecks. Identifying possible bottlenecks, analyze them, and suggest improvements to CODEP activities to handle such issues. One approach we can follow is the process improvement concept (CMMI Institute, 2020), where we can design measurements and adjust CODEP’s activities to solve identified process issues. The process improvement can be performed based on quality attributes of the process such as process comprehension (if the process is well-defined from the knowledge engineer’s perspective), visibility (if the activities produce clear results), acceptance (if the process is understood by the end-users, knowledge engineers, etc.), support (which activities are performed with tool support), reliability (if the process allows identifying failures which can affect the resulting ontology), maintainability (if the process can easily evolve) and velocity (in the speed of performing activities to complete the knowledge base).

Further studies comparing CODEP to existing ontology development approaches would be of interest. For this objective, we would have to define metrics where we can measure aspects of CODEP and compare them with the results of the other approaches. There are challenges we can encounter in performing this kind of study to make these comparisons meaningful. We have to apply the same metrics used and published by other approaches and develop ontologies of similar domains and complexities.