Causal Knowledge Modelling for Agile Development of Enterprise Application Systems

2.1Transformations in MDA/MDD

Model-Driven Architecture (MDA) is a software development approach by the OMG. The MDA approach includes the modelling of these layers: Computation Independent Model (CIM), Platform Independent Model (PIM), Platform Specific Model (PSM), and model transformations (inside each layer and between these layers) (OMG, 2022a).

Traditional MDD is a model transformation process aimed to generate (semi automatically) the target model from the source model (Kleppe et al., 2003). One of the shortcomings of the current MDD methods is that the initial knowledge about the business domain processes is described using empirical methods and models, relying on external observation and experience. Current MDD methods are not focused to understand the causes of internal interactions, to find the so-called deep knowledge. As stated in Gudas and Valatavičius (2020), “real world domain modelling at CIM layer is constructing of black boxes”, therefore including causal modelling is a must from a theoretical point of view.

The conceptual diagram in Fig. 1 highlights the sources of knowledge that influence decisions in the MDA/MDD process – i.e. “Business management strategy”, “Expert knowledge” about the “Business domain” and the “Request to develop EAS”. These elements depict knowledge (preconditions) required for starting the MDA/MDD process of creating models in the CIM layer aligned with the business strategy.

Fig. 1

Conceptual diagram of the MDA/MDD approach.

Computation Independent Model (CIM) is designed to represent business domain processes (e.g. using BPMN and DMN notations, which are an OMG standard, OMG, 2022b), Platform Independent Model (PIM) specifies the EAS design model without regard to the hosting platform. Platform Specific Model (PSM) is the EAS project implementation model and includes concepts of the hosting platforms. Model transformations between the CIM, PIM, and PSM (inside each layer and between these layers) are defined.

MDD is a promising methodology for the development of cyber-physical-social systems (CPSS), cyber-enterprise systems (CES), cyber-physical systems (CPS), and other types of complex systems. Kulkarni et al. over the years have presented several papers aiming to improve the development of their component-based development environment Mastercraft (Kulkarni and Reddy, 2004). Mastercraft is a model-driven development toolset for developing large business-critical applications. It contains a meta-modelling tool to specify an abstract view. This represents the capture of predefined knowledge (i.e. grey-box approach). The MDA/MDD lies in one of the components of the Mastercraft tool – i.e. a set of code generators that transform each view instance to the desired implementation artifacts. Barat et al. has presented a configurable code generator meta-model to transform models to code (Barat and Kulkarni, 2010). Although some information that is required to smoothly conduct transformations are missing (i.e. “various non-functional concerns, namely, design strategies, technology platform, and architecture”) the model described in Kulkarni and Reddy (2004) is well defined to be used for the MDD approach. The model transformation rules specify the mapping between MDA layers and lead to the specification of the computer code for implementation of the solution. The transformations between CIM, PIM, and PSM layers must comply with relevant meta-models (den Haan, 2008).

To summarize, current MDA/MDD methods do not contain the element of domain causality modelling.

2.2Agile MDA/MDD Methodology

Agile Model Driven Development (AMDD) is an attempt to use the benefits of the fast-paced and responsive to change Agile development and the quality-focused and stable structures of Model-Driven Development. Ambler coined the term Agile Model Driven Development (Ambler, 2002). By examining the blended concepts of Agile and MDD it should be noted that MDD is a model-centric approach and values the thoroughness of models which take time for development. Agile, on the other hand, values working software, and the attention to models is little to none. Using the Agile approach, extensive models are considered as not required, and as Ambler states even with the AMDD approach, “an agile model is a model that is just barely good enough” (Ambler, 2004). Agile models need to exhibit a set of traits like fulfilling purpose, being sufficiently detailed, etc. However, it is not explained how the knowledge is obtained and there is no mentioning of performing model transformations as required by the MDD approach. This shows that model transformations are left to the MDD side of the AMDD approach. Furthermore, Ambler states that in order to perform modelling, fundamental information gathering skills are required and names observation as one of them (Ambler, 2004). This leads to the conclusion that Agile modelling is external observation-based (black-box). Ambler also stated that “AMDD is <only> about modelling MDD models more effectively” (Ambler, 2004).

Kulkarni et al. introduced a “meta sprint” based software development approach. Results show that there was a decrease in turnaround time from requirement specification to delivery, the increase of productivity of the development team, and less rework by using their developed tool Mastercraft (Kulkarni et al., 2011). However, this approach is very similar to the dual Scrum or dual Agile approach (Miller, 2005) which later evolved into the so-called “design sprint” (Knapp, 2016). The dual-track Agile and design sprint was inspired by the “double diamond” methodology (British Design Council, 2007) formulated by the British Design Council. Furthermore, the approach by Kulkarni et al. does not describe the causal knowledge that the meta-models should explain.

Matinnejad (2011) has presented criteria how to evaluate the attributes of the fast-paced and flexible approach of Agile and thorough, well-thought, and complete approach of MDD and analysed the AMDD processes of Sage (Kirby, 2006), Hybrid MDD (Guta et al., 2009), MDD-SLAP (Zhang and Patel, 2011) and the AMDD high-level life cycle (Ambler, 2004).

To summarize, after a thorough research to the best of our knowledge, there is no attempt to add the component of intelligence (causal knowledge capture) to Agile MDD methods.

2.3Causal Modelling Driven MDA/MDD Approach

The traditional MDA/MDD method and AMDD method start with the analysis and modelling of the business area using the standard OMG notation BPMN, which allows to construct models from (Input, Process, Output) elements, connect them, create hierarchies (sub-processes). It is essentially an empirical modelling approach that creates black-box models without revealing the causality of the business domain. More advanced is DMN (Decision model and notation), which uses decision tables and is good at specifying complex business rules (OMG, 2022b). DMN models partially reveal causality dependencies and can be classified as a grey box type. Therefore, both the traditional MDA/MDD approach and the AMDD approach does not contain the domain causality aspect, therefore it is missing the key feature of creating an intelligent software development management system.

In order to improve this, we will utilize the causal modelling approach. Specific domain causality awareness is the prerequisite for discovering deep knowledge (i.e. regularities, laws) in a given domain. Causation methods are common in statistics, econometrics, cybernetics, computer science, data science, and other complex sciences to study cause-effect relationships and construct causal models in order to predict and control the possible dynamics of the systems (Gudas, 2021). From the causal modelling viewpoint, an enterprise is a subject domain, considered as a self-managed system driven by internal needs (strategy, management, system goals).

Based on causal modelling viewpoint the MDA/MDD approach was enhanced with a new layer (above CIM layer) of the domain knowledge discovery in Gudas and Valatavičius (2020) and the causal knowledge discovery (CKD) technique tailored for the enterprise domain. The peculiarities of the causal MDA/MDD method is illustrated in Fig. 2. In this block diagram rectangle means component, arrow – interaction.

Fig. 2

The causal MDA/MDD transformations. Based on Gudas and Valatavičius (2020).

We focus only on the stage of building a causal domain model and transforming it into a CIM layer, as these key steps apply to our approach to EAS development management using a modified Agile management hierarchy. As in this article we investigate the content of project management processes, in Fig. 1 the “Business domain“ means project management domain (processes).

Causal domain models in the CIM layer are built using transformation specification rules, which defines mapping of the domain causality meta-model to causal CIM meta-model, and both meta-models are relevant to MT framework (predefined causal knowledge) (Gudas, 2021; Noreika and Gudas, 2021). The same principle is applied in the modified Agile management process (Noreika and Gudas, 2021), when predefined causal knowledge (meta-model of Agile management hierarchy) is used to verify the status of EAS project solutions (project content) which is recorded in the standard Jira tool.

Alfraihi et al. have conducted a thorough systematic literature review on the topic of the integration of Agile development and MDA/MDD transformations (Alfraihi and Lano, 2017). We examined the results to determine if there is a description of how to capture causal domain knowledge. The conclusions are presented in Table 1 and are based on the causal modelling approach as described in Gudas and Valatavičius (2020), Noreika and Gudas (2021). The papers that do not have MDA transformations specified are not included. The last line in italic indicates a gap in the established MDD/MDA transformation approaches where the MDD/MDA methods are converging to a causal knowledge based transformation rules. “DSL” stands for “Domain specific language”. Three types of knowledge are shown in Table 1:

Many tools support the Agile software project management process. Atlassian Jira software is one of the leaders (digital.ai, 2020). Current state of Jira only supports the themes, initiatives, epics, and user stories or similar structure. It does not provide formalities to ensure that the links between the different activities in the levels of the Agile hierarchy are properly identified and justified in the business context. It means that it is not ensured that each and every user story, epic, initiative, and theme links to one of the strategic business objectives. Links are determined by experts working on project delivery and are subjective, which means that the links definition are prone to errors.

Furthermore, as it was presented in Table 1, only a few AMDD methods contain causality modelling, including meta-model, and in those articles there is a lack of definition of causal-based knowledge base or a similar repository, therefore a more thorough and detailed research is required.

3.1Model of the Traditional EAS Development

The traditional EAS development part in Fig. 3 includes real world interactions between business management and operations (nodes M1–M3), and experience–based interactions between nodes (M1–M4–M2), (M3–M4–M2), (M3–M4–M5) and (M2–M5) that are explained in this subsection below. The interactions between nodes (M1–M4) and (M3–M4) shows the causality understood through experience – i.e. performing the real world activities multiple times reveals the peculiarities of the business domain and allows to capture the causality. The interaction between nodes (M4–M2), (M4–M5) means that the expert (business analyst or developer) must understand the business domain, i.e. understand the causality of it.

The traditional MDA methodology relies on the external business domain analysis of activities (M1–M3) and analyst’s experience when the CIM layer models are constructed. There is no requirement that the domain causality M4 must be directly theoretically studied and used to develop M2 and M5.

The domain causality M4 exists as causality is a permanent feature of the real world and has impact on interactions between real world processes M1 and M3.

Business domain modelling is based on external observation, creating business domain models (M2) based on experience and using them to design EAS solutions M5. Content of M2 and M5 based on partly discovered knowledge M4 (through experience). However, this experience is not clearly expressed as a particular model(s) of domain causal knowledge.

The project management is considered as real-world process with its permanent causality (node M7) that needs to be understood thoroughly and modelled (M7^∗) to develop an intelligent project management system. Therefore, we model the Agile project management process as EAS engineering process following MDA approach. Thus, the Agile project management process as a causal process is described in Fig. 2.

The interactions between the aforementioned nodes are described as follows:

Current MDA/MDD models on CIM layer are built through external observation, so the notations used for modelling do not distinguish between what is the control function (i.e. information transformation activity) and the controlled process (i.e. physical/material transformation activity). These notations do not require verification of the feedback loop, which must be created to ensure control causality.

It is not explicitly clear if there are only fragments of the whole system observed in node M2 or the whole components of the system and causal interactions are captured. In real-world EAS development projects this is often the case, because initially a lot of information about the processes are context-based, so the analyst or developer (external observer) cannot perceive the process in full, until it has sufficient knowledge of it and various exceptions, workarounds, etc. Only then proper modelling can be done.

We assume that business domain causality (M4) can be (partially) discovered and specified in the knowledge base. To achieve this goal, we apply the management transaction framework (MT), which is described in detail in previous works (Gudas and Noreika, 2022).

3.2Modified Agile Project Management

Based on the analysis of the traditional Agile development management method, a modification is proposed in Gudas and Noreika (2022), Noreika and Gudas (2021) that is the conceptual background to create an intelligent EAS development management tool.

Conceptual model in Fig. 3 shows that causal Agile management tool M6 (right side of Fig. 3) is interacting with the traditional EAS development nodes (left side of Fig. 3) on the basis of experience and is supported with virtual transitions (M6–M2), (M6– M5) and (M6–M7–M7^∗).

Node M6 indicates the EAS project management tool (e.g. Jira, Azure Boards, Monday.com, Wrike, etc.), with a database about the current state of project execution (data) (state of project).

M7 denotes EAS project management causality between Agile hierarchy of themes, initiatives, epics, user stories. These are real world activities, that we (partly) model using the meta-model of causal Agile management hierarchy (Fig. 6a)

Node M7^∗ refers to project management knowledge base content described in Section 4: traditional Agile hierarchy meta-model, causal Agile hierarchy meta-model, and causal knowledge unit (MT meta-model).

The integration of M6 with M7^∗ enables the creation of an intelligent Agile management environment, based on causal model of Agile management hierarchy (M7).

Due to these knowledge components, the node M7^∗ enables to assess the project state specifications recorded in node M6 (project management tool database) and indicate the gaps against required causal Agile management hierarchy specifications (M7^∗). In our approach towards intelligent Agile management tools development, domain causation is the regularity of the enterprise management, defined as a management transaction (Gudas, 2012) and forms the basis of the knowledge base M7. Management transaction (MT) is considered as a causal knowledge unit. Recognized, discovered domain processes are specified by examining (verifying) whether they conform to the MT framework.

We provide further specifications for the knowledge base as part of node M7 and project management tool database additions as part of node M7 that allows to build an intelligent project management tool. The purpose of displaying M7 here as a component of the EAS development management is to demonstrate that we acknowledge that domain causality must be understood by the analysts, developers and any personnel working with EAS development to make sure that the developed EAS meets the real world processes.

The causal business domain model M4 is defined using the Management Transaction (MT) framework (Gudas, 2012; Gudas and Lopata, 2016). Management Transaction MT = (P, F, A, V) is relevant to some enterprise goal (G), captures knowledge on management function (F), enterprise process (P), informational input flow (state attributes A), informational output flow (controls V). MT includes a feedback loop between F and P composed of A and V flows.

Fig. 4

Conceptual model of Management Transaction. Based on Gudas (2012).

A conceptual management transaction model presented in Fig. 4 also captures the enterprise goal (G), which is not specified explicitly in the formal MT definition.

Knowledge base (KB) is a crucial component of any intelligent system. KB contains the real-world knowledge captured by experts and is an internal model following the Internal model principle defined by Francis and Wonham (1976). The causal MDA/MDD approach uses predefined knowledge to ensure causality-based transformations (as depicted in Fig. 2). We believe that the KB needs to be created to support MDA transformations for developing EAS in an Agile setup.

3.3The Modified Agile Hierarchy

Traditional Agile management hierarchy is based on the TIES hierarchy as explained in the Section 1 of this paper. Basically, concepts of themes, initiatives, epics and user stories represent the requirements for EAS development in different levels of abstraction. The mentioned concepts are activities – meaning actions are required to fulfil them – analysis, development, testing, etc. A set of initiatives form a theme, a set of epics form an initiative and a set of user stories form an epic. The traditional Agile management hierarchy is presented in Fig. 5. It must be noted that we do not investigate the link between strategic objective and theme in this paper, as this is a task of such complexity that it requires separate investigation. However, there are attempts to specify it using enterprise architecture frameworks (Noreika, 2021).

Fig. 5

Agile management hierarchy (traditional), aggregation relationship.

Analysis of traditional Agile hierarchy from the causal modelling perspective revealed a few new aspects to be considered in EAS project management as follows:

By evaluating these characteristics of a real-world Agile process, we defined a modified (causal) Agile hierarchy model. This causal Agile management model is based on the business domain causality model and is focused on the enterprise strategy alignment with EAS solutions.

In the modified Agile hierarchy (Fig. 6a) any activity on each level:

The detailed view of the modified Agile hierarchy is presented in Fig. 6a. An example of different activities in the Agile hierarchy layers is represented in Fig. 6b, here the axis AG (aggregation) indicates the hierarchy of Agile activities (vertical interactions), the axis GE (generalization) indicates the classification of Agile activities into types. T01 in Fig. 6b represents example of themes in Fig. 6a, I01 – initiatives, E01 – epics and S01, S02 – user stories respectively. In our approach, the interaction of activities T01 and I01 is a management transaction that is a complex process and it is defined as a set of information flows specified in the element “Interaction specification: MT(Theme – Initiative) (Fig. 6a) and so on, respectively (I01 – E01, E01 – S01, E01 – S02).

Fig. 6

Modified Agile hierarchy: a) detailed view; b) example of Agile activity hierarchy.

Any vertical interaction between activities of different levels (theme, initiative, epic, user story, …) is conceptualized as management transaction MT(P, F, A, V), where a higher level activity is considered as “management function” (role = F), relevant lower-level activities are considered as “processes” (role = P), also a feedback loop between higher-level activity (F) and lower-level activity (P) is predefined (state attributes flow A and controls flow V). Management Transaction is detailed in Fig. 4.

Vertical interactions of activities on the different Agile layers are overlapping. For example, the internal interaction in MT (initiative, epic) is as follows: the role of the higher-level element “initiative” is “control function” (F), and the role of the lower-level element “epic” is “process” (P). Next, the role of epic in internal interaction MT (epic, user story) changes: the role of epic here is F, the role of the user story is P.

A horizontal interaction between Agile activities must have a coordinating message between two management transactions MT(P, F, A, V) and MT^∗(P^∗, F^∗, A^∗, V^∗). Horizontal interactions differs depending on the management function of the managing function – i.e. software development management and business management.

3.4Verification of the Modified Agile Process

The objective of model verification is determining if the implementation of the model is correct, i.e. conforms to the model requirement specification. The modified (causal) Agile process model must be verified to ensure that it meets causal modelling requirements defined as follows:

The verification rules a), b) and c) define requirements to conceptual model of the causal Agile management process and help to evaluate if the modified Agile process is performing the way it should following causal modelling paradigm.

The verification rule a) defines requirement that internal structure of Agile activities must include all MT elements (F, P, A, V). Verification rule b) defines the requirement that the internal structure of the interaction between the levels of the Agile hierarchy corresponds to the MT definition, i.e. would be a transaction with a feedback loop.

Verification rule c) defines the requirement that the Agile hierarchy must meet the strategic goals of the enterprise. Verification rule d) checks whether the causal elements of the Agile process, whose internal structure corresponds to the MT definition, can be stored in the designed project knowledge repository (Fig. 9).

The verification rule a) is satisfied because all activity types of causal Agile hierarchy (theme, initiative, epic, user story) are considered having internal structure defined as MT(P, F, A, V). For example, starting with the theme (Th), the internal structure is defined as MT(Th) = MT(P, F, A, V) as depicted in Fig. 8. The formal specifications of all causal Agile hierarchy activities, which are based on the MT definition, are discussed in detail in our article (Gudas and Noreika, 2022).

The verification rule b) is satisfied because causal model of each item of causal Agile hierarchy is defined as MT(P, F, A, V) (see rule a), thus by definition it includes vertical interaction between higher level and lower level activities.

In order to specify the content of MT more precisely, we include both identifiers of the higher level and lower level items in the specifications of MT structure. For example, in MT(Th,I) = MT(P, F, A, V), a lower-level activity “initiative” is considered as process (P) associated with a higher level activity “theme”. The role of activity “theme” in this interaction is “management function (F)” (Gudas and Noreika, 2022). Therefore, we have the following specification: theme (Th) is defined as MT(Th, I) = MT(P, F, A, V) and includes a vertical interaction with initiative (I) which is defined as MT(I, E) = MT(P^∗, F^∗, A^∗, V^∗) (Gudas and Noreika, 2022) as illustrated in Fig. 8. The same structure of vertical interaction exists among all causal Agile hierarchy activities. The causal Agile hierarchy interaction specifications are discussed in detail in Gudas and Noreika (2022). Thus, the lower Agile activity satisfies the “needs for information” of the higher level activity – requirements for complete information about its state, i.e. a vertical interaction corresponds to required MT structure.

The verification rule c) is satisfied by the modified Agile process because top-level Agile activities themes are related to the strategic goals of the enterprise represented as capabilities specification in Fig. 6 and attribute ID_CAPABILITY in the table Agile_activity in the knowledge repository (Fig. 9b). At the model implementation level, data storage tables (Agile_activity, MT in Fig. 9) have relationship with the data storage table (CAPABILITY).

Verification rule d) is satisfied because all Agile activities are saved as management transactions, because all internal MT elements (F, P, A, V) are specified in data storage tables (MT, A flow attributes, V flow attributes) in Fig. 9.

In addition, an example of the enhanced Jira tool database record in Table 3 demonstrates that the designed knowledge base meets the requirements of the causal Agile process.

4.1Defining Agile Project Management Data Structures

The causal Agile management repository specification (Fig. 9) is developed following the conceptual model of causal knowledge repository in Fig. 8. The causal Agile management repository in Fig. 9 consists in to two parts: enhanced project management database (Fig. 9b) and project state assement knowledge base (Fig. 9a).

Fig. 9

Causal Agile management repository specification.

Enhanced project management database (prototype is the Jira database) is supplemented with new DB tables CAPABILITY, BUSINESS_OWNER and new attributes required by the conceptual model in Fig. 8. The table CAPABILITY contains identifier (ID_capability) and description of strategic goal item. For ease of visualization, the CAPABILITY table is recreated twice. The table Agile_activity contains new attributes that ensure data storage of causal interactions: ID_MT – identifier of management transaction, ID_CAPABILITY – identifier of capability (i.e. strategic goal item), BUSINESS_OWNERS_ID – identifier of top level manager. The table Sprint is not modified.

EAS “Project state assessment knowledge base” in Fig. 9a is the implementation of the component “Causal Agile hierarchy meta-model” in Fig. 8. The causal knowledge storage structure includes tables MT, A_FLOW_ATTRIBUTES and V_FLOW_ATTRIBUTES with functional dependencies between them. This data structure ensures the storage of causal interactions content as defined in the conceptual model of causal knowledge repository (Fig. 8):

It is important to notice that “Project state assement knowledge base” is linked to capability description in table CAPABILITY. This allowed for tracing of EAS solution activities (theme, initiative, epics, user stories) against strategic objectives.

According to the causal modelling method, project state data on interactions between levels of the Agile hierarchy are normalized by checking their compliance with the causal knowledge unit definition MT = (P, F, A, V). The storage of the received restructuring result is ensured by the specifications of tables MT, A_FLOW_ATTRIBUTES and V_FLOW_ATTRIBUTES, it can be seen in Fig. 9a.

Thus, the specifications of the causal Agile management knowledge repository comply with the definition of the conceptual model of causal knowledge repository.

Fig 9b includes the implementation of the “Strategic capabilities” component of the “Causal knowledge repository” from Fig. 7.

In summary, the EAS “Project Status Assessment Knowledge Base” and “Enhanced Project Management Database” specifications (Fig. 9) meet the constraints defined by the conceptual models (Fig. 7 and Fig. 8).

The Fig. 9 shows how each causal Agile hierarchy interaction (MT(theme, initiative), MT(initiative, epic), MT(epic, user story)) should be defined and saves the actual state of current EAS project state assessment against strategic enterprise objectives.

Records of the Agile activities in the project management database are associated with a causal model of Agile hierarchy and interactions through an additional set of attributes (Fig. 9b), additional fields are capitalized.

We chose to present the fields with longer names for the sake of clarity. Furthermore, such usually encountered service information as date created, updated, user, that created or updated the record are not provided in each of the tables for the sake of simplicity.

Table 3 contains the records for EAS project requirements based on database structures in Fig. 9. It indicates that there is missing theme and initiative information and epic information is missing link to theme (F).

The prototype of an intelligent interaction component is presented in Fig. 10. This view is compiled using the content of the project management database record of the enhanced Jira tool from Table 3.

Fig. 10

Prototype of the enhanced Jira tool report.

The dotted column in Fig. 10 reflects the possible additional fields based on the situation comparing records in the Agile project management database. Below is an example of the semantic information that the Jira tool would store (Jira database records content) and that is used to reflect the gaps of the project state against the modified Agile hierarchy meta-model: