Towards a scientific data framework to support scientific model development

1.Introduction

Scientific repositories allow the collection and distribution of scientific and scholarly data. Among the fundamental factors contributing to their growth is an increasing availability of computing power and storage capacity as well as new big data analysis techniques to manage such vast amounts of data. However, since their initial establishment, some challenges have been highlighted in the literature, such as availability, integration, extensibility and maintainability [34].

The urgent need of improving the infrastructure supporting the reuse of experimental data has been highlighted in the literature and led to general guidelines to create and manage repositories, notably the FAIRness [59] (being findable, accessible, interoperable and reusable) or the “pyramid” of needs for data management that span from being simply saved to being shared to ultimately being trusted [18]. The collection of scientific data is becoming more and more important for the validation of research results. One of the current goals is to improve the capability of sharing experimental data among researchers, in order to enhance their quality and reproducibility and to derive new research results. Based on the Open Science Cloud strategy of the EU,11 every project financed within the H2020 framework has to comply with the FAIR data policy.

Sharing of scientific data has been increasingly promoted over the last decade. However, it was recently discussed how “data sharing practices, especially motivations and incentives, have received far more study than has data reuse” and that data sharing and open data are not final goals in themselves, but the real benefit is in data reuse, which is “an understudied problem that requires much more attention if scientific investments are to be leveraged effectively” [39]. Reuse leverages investments in research and enables large-scale data-driven research progress. However, in order to reuse datasets from multiple sources, challenges, such as integration, must be addressed.

This paper focuses on reuse, and in particular on the design of an integrated system to automatically take advantage of large amounts of scientific experimental data with the goal to support new research. This requires the integration of data management and analysis techniques in the scientific development workflow. If a scientific repository with its tools can improve the efficiency of many tasks, such as searching and exporting selected entries, usage of new data analysis techniques can leverage the large amounts of integrated scientific data to automatically and dynamically discover knowledge normally not obtainable manually or through partial datasets given the large volumes of data required. We consider experimental data published in scientific publications, which are typically delivered as attached supplementary materials or made available in scientific and scholarly data repositories.

One key research activity in many scientific domains is scientific model development, where the goal is to build models to reproduce and predict complex phenomena. Experimental data, typically extracted from scientific publications, have a crucial role within this activity. Indeed, extensive comparison of model simulations with real experimental data is needed in the iterative scientific model development life cycle, to continuously validate and improve the scientific models being developed.

In this paper, we focus on experimental data and model development in the combustion kinetics domain, where a number of initiatives to collect experimental data in a systematic way have already been developed [23,26,54,58], along with initiatives to define better community data reporting standards.22 Even if this paper focuses on a specific domain, the use cases, the requirements and the solutions analyzed are common to many other scientific domains, as discussed in Section 3.

Automatically taking advantage of published scientific data to aid model development sets several challenges:

Starting from a description of the most important use cases, which also emerged from previous works [44,49], this paper describes the development of a new prototype which goes in the direction of supporting them, the new requirements that emerged from the prototype development and an architecture which, while also taking new requirements into account, outlines the proposed framework’s future development.

Therefore, this paper’s novel contributions are as follows

This paper is an extension of [49], presented at the SAVE-SD workshop on Semantics, Analytics, Visualisation: Enhancing Scholarly Dissemination. This paper shares with it the analysis of the combustion kinetics scenario and the first generalized use cases and requirements arising from it. However, with respect to the previous work, the use cases have been refined and detailed, a completely new prototype has been developed, and this led to new requirements and an architectural framework design which has been heavily improved and detailed with respect to those presented in [49].

The paper is structured as follows: Section 2 introduces the domain and the general approach to scientific model development, Section 3 discusses related work, Section 4 describes a set of use cases for the system and the developed extended prototype, Section 5 discusses the emerging requirements and Section 6 outlines an integrated architectural framework to support them. Finally, Section 7 discusses perspectives for future research directions.

2.Scenario

In this section, we describe the scenario – kinetic modelling of combustion processes. Kinetic models determine the reactivity of a given fuel or a fuel mixture, and combustion kinetic modelling has been driving the development of more efficient fuels and combustion technologies for the last 40 years.

The development and update of reliable kinetic models is a rather challenging task, directly reflecting the intrinsic complexity of combustion phenomena, and it is one of the fields of research of the CRECK modeling group.33 Such models typically involve ∼100–1,000 chemical species connected by a network of ∼1,000–10,000 elementary reaction steps. Moreover, a combustion kinetic model hierarchically develops from small chemical species (e.g., hydrogen, methane, and so on) up to heavier compounds typically found in commercial fuels such as gasoline, diesel, jet and alternative fuels. For this reason, any modification in the core model significantly propagates its effects to heavier species making continuous revisions and updates mandatory to preserve the model’s reliability.

OpenSMOKE++ [17] is an example of software developed to execute such models performing kinetic simulations of typical facilities such as jet stirred and flow reactors, 1D–2D laminar flames, shock tubes and rapid compression machines, specifically conceived to decouple purely chemical kinetic effects from fluid dynamics, heat and mass transfer phenomena. Variables of interest are typically ignition delay times, laminar flame speeds, fuel/oxidizer mixtures, fuel consumption, and intermediate product species formation/disappearance at specific conditions of temperature, pressure and dilution.

Fig. 1.

Standard development, validation and refinement cycle of a chemical kinetic model for combustion applications. The process is represented as a flow of data and activities and has multiple starting points. Dashed lines represent auxiliary data flows. In this paper, we mainly focus on the validation phase (dotted rectangle). A validation step could be triggered by the development of a new model – which in turn could be the result of the refinement of an existing one – or by the acquisition of new experimental data in the database.

The standard development, validation and refinement cycle of chemical kinetic models for combustion applications is shown in Fig. 1. From an operational perspective, the iterative validation of such models (dotted rectangle) strongly relies on extensive comparisons of results from model simulations with experimental data covering conditions of interest for real combustion devices. The key step in such procedure is the comparison (assessment) of model performances with respect to experimental data. Experimental data are typically found in supplementary materials attached to scientific papers as CSV, ASCII or Excel files. While this good reporting practice is almost always the case in recent publications, older studies did not include such details. In this case, semi-automated or automated extraction of experimental data and boundary conditions from tables, figures and text reported in the manuscript is necessary.

Fig. 2.

On the left, boundary conditions and output variables of the experiment 10.24388x10000021_x, extracted from [55] and available in the ReSpecTh repository [54]. On the right, the output variable ignition delay time is plotted with respect to the inverse of the temperature in a logarithmic scale, as generally accepted practice in the domain.

From a data perspective, an experiment consists of a set of conditions, which describe the experimental setting, and a set of output variables. The conditions can be categorical values (e.g., the reactor type), constants (e.g., the initial mixture or the initial pressure of the experiment) or, less frequently, data series. The conditions are needed to replicate an experiment or simulate it with a model. Outputs variables are series of (x,y) values and can be represented as discrete curves. An example of an experiment description provided by the ReSpecTh repository [54] is shown in Fig. 2, highlighting the extracted boundary conditions and output variables, which are also plotted. In ReSpecTh, the experiment description – originally in the XML format – includes also other data not shown in the Figure, including a unique DOI (10.24388x10000021_x in this case) and information about the source paper from which data have been manually extracted ([55] in this case, specifying also “Fig. 8., full square” as the source Figure). Supplementary materials of scientific papers, when available, include the same kind of information in Excel or CSV files.

A model’s assessment is typically performed by using plots in which experimental data and curves from model simulations are plotted together. Researchers express their subjective evaluation based on their experience and knowledge, often neglecting experimental and model uncertainties. This standard model validation procedure was recently highlighted as a “poorly posed” problem [56]. Analysis tools (e.g., sensitivity analysis) allow the highlighting of relevant model parameters and drive their refinement by means of more accurate estimation methods.

An example of a plot which compares the output variables of the experiment detailed in Fig. 2 with the same output variables as predicted by some models – given the experimental boundary conditions – is shown in Fig. 3. In general, the concept of “best model” is not easily definable [7]. Historically, the evaluation of model performances has been performed by means of a rather subjective interpretation of the degree of agreement with experimental data as visualized on 2D (x,y) plots. Within the combustion kinetics domain, Turanyi and co-workers reported the sum of squared error based methods to evaluate the performances of different kinetic models [36,37]. New techniques were recently proposed in order to automate a model’s performance assessment with respect to some experimental data, deriving objective and quantitative measures and overcoming manual graphical comparisons. The Curve Matching algorithm [7] allows an objective, quantitative and automatic evaluation of the model’s capability to predict an experiment’s variables of interest, extending the most common, but sometimes misleading, sum of squared error based approach [36].

Fig. 3.

The output variable of the experiment in Fig. 2 is shown against some models which approximate it.

If the model provides satisfactory agreement, subsequent steps of optimization and reduction [53] make the model suitable for large scale computations of interest for industry (i.e. reactive computational fluid dynamics). Indeed, the developed model is not manageable given its size (number of chemical species and reactions), and model reduction, provided a desired degree of agreement with the original detailed model, allows to obtain a smaller model that is suitable for applications. On the contrary, if the model shows deviations outside of experimental uncertainties, relevant pathways that can be identified by means of analysis tools and model parameters are further refined with better estimates. Indeed, recent developments coupling high performance computing and theoretical chemistry allow the automatic generation of highly accurate parameters [11,29].

The efficient integration of the above tools and the process shown in Fig. 1 in a fully automatized system has been recognized as one of today’s challenges in kinetic modelling [29]. Exponential growth in the volume and complexity of scientific information in the combustion community (experimental data, models, theoretical investigations, etc.) and improved accuracy of experimental techniques can be beneficial, at best, only if coupled with efficient tools for acquiring, storing and analyzing such information, and therefore allowing real advances in knowledge. In this paper, we focus on an integrated system mainly targeting the validation phase (dotted rectangle in Fig. 1).

Several initiatives to enable the effective and structured collection of experimental data for combustion science are described in the literature. Starting from the pioneering work of M. Frenklach and co-workers to develop the PrIMe database [23], which is still under continuous update, the ReSpecTh repository [54] largely improved and extended it by means of a more flexible, detailed and user-friendly data structure. At present ReSpecTh has collected more than 1,000 experiments into XML files, which correspond to more than 87,000 data points, and the extent of this collection is expected to increase in years to come. CloudFlame (KAUST) [26] provides an open database for experimental data in a standard CSV format, together with a cloud infrastructure for running simulations based on stored models and data. To improve human readability of structured data, ChemKED [58] has introduced a human and machine readable data standard for chemical kinetic experiments based on YAML, together with a set of validation and conversion tools. The COST Action CM 1404 (SMARTCATs44) established a task force of scientists to define standards for data collection, allowing easy and effective coupling with the above systems.

Over and above the reference repositories mentioned, one should consider a large amount of experimental information stored in less structured formats in many institutional servers belonging to experimental or modelling groups working in the field of combustion. As an example, the CRECK repository is the result of data collection from over ∼40 years of research efforts in modelling combustion kinetic and thermal processes. While data management has relied on manual extraction and organization into unstructured spreadsheets or text files for a long time, a relational database with a local interface for querying and exporting data was recently developed [44]. However, the limitations of that software and new use cases which go towards the direction of a system that not only stores experimental data, but leverages them to support and enhance the entire model development process, led to the design of a new infrastructure which includes not only an experimental database, but also integrated data curation and analysis tools. Requirements for the new infrastructure were investigated starting with [49]. This paper moves towards the same direction and describes, in addition to updated and more complete use cases and requirements, a new framework prototype with its designed architecture.

As discussed at the end of the following section, the need for continuous model validation based on new experiments and use cases arising from this domain are certainly shared by other scientific fields. This makes this development scenario, and its challenges in terms of a supporting infrastructure, a general scenario which is also valid in other domains with similar characteristics.

3.Related work

This section presents a brief overview of related works available in the literature. As the proposed framework includes challenges in data management, data quality and cleaning, each of them is briefly outlined below, before describing other systems with similar goals and other related work which motivated this system. Other papers will be cited in the following sections when discussing requirements on specific issues.

All requirements analyzed in this paper strongly rely on data management efforts. Indeed, it has been highlighted in the literature how value creation is not driven by data itself, but by the whole data management process [63], and that it is enabled by the management of both internal and external data. Data reuse, in particular, requires specific management techniques.

The goals of this work stem from the increasing availability of open and structured scientific data and the resulting needs of new ways to effectively leverage and reuse them. In general, the concept of “open data” is not easily definable and baseline conditions are “fewest restrictions” and “lowest possible costs” [40]. It has been noted how data sharing practices have been investigated more than data reuse [39]. Data reuse is not to be limited to reproducing research, which is an example of independent reuse (that is, reuse of an individual dataset) but should also include data integration “to make comparisons, build new models or explore new questions altogether” [39]. This is the direction of the present work, which focuses on reuse as a means to extract new knowledge not available by analyzing individual instances. To reach this goal, it has been highlighted how standards and formats for data release influence reuse opportunities. Standards are a prerequisite to understand and integrate datasets, but even with these in place, differences in parameters, instrumentation and other factors make data integration non-trivial [8]. Moreover, semi-structured data formats could add ambiguities to standardized datasets.

Challenges related to data reuse and value creation are discussed in [28]. Finding useful datasets is a challenge in itself, since this precedes their interpretation. This requires adequate metadata and often some kinds of standards, but “adequate” has a different meaning when the goal changes, and often a publication includes only the bare minimum required for its specific goal. There are also format issues, which sometimes can be simply a question of performing a schema mapping, but often require resolving more substantial mismatches for an effective integration.

Besides domain-specific repositories, described in Section 2, several general purpose scientific repositories have gained importance in the last years to efficiently share research outputs, including experimental data. Notably, examples are Figshare,55 Zenodo,66 myExperiment77 and Gigantum.88 On top of these, tools to efficiently search the vast amount of published data, such as Google Dataset Search,99 and to efficiently reproduce scientific code, such as Code Ocean1010 have been made available. This work investigates ways to aggregate and analyze published experimental data for a different goal: supporting scientific model development. Collecting and storing experimental data is a means rather than an end for the proposed framework and the goal is to leverage existing efforts in data sharing as much as possible. Overcoming challenges like experimental data interpretation and integration, discussed in this paper, should pave the way for large-scale comparisons on published experimental data that could lead, for example, to indirectly detecting inconsistent results, rather than directly reproducing an experiment.

Two fundamental aspects of the proposed framework are data quality management and data cleaning, which are strongly related to the veracity of big data. Their importance comes from the reuse of data which can be noisy, outdated, misleading and, in general, unreliable. Indeed, incorrect information is often found in shared data [8] and unreliable experimental data have been identified in the chemical kinetics domain even quite recently [27]. Quality issues could be introduced even later in the processing pipeline, for example when experimental data are imported into a repository or converted into a specific format, given also the constraints of the related standard. Independently from the cause, it is important to assess data quality before using data in order to get meaningful results.

Data quality management techniques used in traditional databases are not enough to handle a big data scenario with heterogeneous sources, which instead requires an “adaptive approach able to trigger the suitable quality assessment methods on the basis of the data type and context in which data have to be used” [4]. Among other challenges, this requires context-dependent quality assessment, which takes into account, for example, that a large number of sources can instill confidence in the data’s reliability, and multi-granularity assessment, to evaluate data quality at various aggregation levels [10]. Data cleaning techniques enable the detection and repair of data errors by identifying integrity constraints, duplicates, functional dependencies, and so on. These activities can be automatic or human-guided, and error detection techniques can be applied on the original database or later on in the data processing pipeline [13]. In this paper, the discovery of errors after some processing and integration is of particular importance given the use of large-scale cross comparisons and the integration of external sources. Such “delayed” cleaning has been analyzed and formally described in the literature [12]. Data integration, and in particular the analytical querying of an integrated set of data sources, is normally addressed through data warehousing (DW) and online analytical processing (OLAP) systems. However, in a traditional OLAP system all data sources should be known and described in advance, as the rules used to perform their integration, and external data sources (i.e., sources which evolve outside the system) are not taken into account. Moreover, traditional OLAP systems typically do not deal with semi-structured data and external ontologies. Such problems arise in this paper’s context, given the need of dealing with multiple, variable and external sources of experimental data (scientific repositories) and external ontologies (domain knowledge which should be managed externally by domain experts) and to obtaining integrated results. Semantic-aware exploratory OLAPs have been proposed to face these limitations allowing the exploration “of new data sources, of new ways of structuring data, of new ways of putting data together, of new ways of querying data” [2]. Such solutions provide several benefits with respect to data extensibility and dynamicity and their “open-world” assumption allows analytical querying of new external data.

The framework investigated in this paper shares some features with scientific workflow management systems (WMSs). Workflows are “a set of interrelated computational and data-handling tasks designed to achieve a specific goal” [32] and a WMS aids in the automation of those operations by managing their execution and information exchange. Scientific workflows are typically data-intensive workflows and scientific WMSs have been proven crucial in data-driven science [5]. A notable example is Taverna,1111 an open source and domain-independent WMS which integrates, among other things, specific web services for chemistry-related workflows. In this paper, we focus on use cases, requirements and challenges coming from the reuse and large-scale analysis of available scientific data to extract new insights, in particular, with respect to scientific model development. Unlike WMSs, we do not directly address the development and management of new experiments (“in silico” experimentation) or models, under the assumption that these activities occur externally to the framework. Workflows can become complex and resource intensive, therefore WMSs face problems such as scheduling and resource allocation in distributed environments [32]. A workflow consists of several computational tasks linked by data dependencies and their management by WMSs involves challenges such as resource provisioning, provenance management, performance variation, parallelism and fault tolerance [19,46]. Therefore, our work is complementary to a scientific WMS, since the analysis tasks identified could be managed efficiently as scientific workflows in a large-scale and multi-user scenario.

Scientific data are typically attached to scientific papers or manually extracted from them successively. Therefore, scholarly data repositories, which typically contain information about authors, citations, figures, tables, etc., are often also a source of experimental results [33,60]. Moreover, being able to analyze the sources of experimental datasets, as well as data themselves, may lead to new analysis opportunities. The management of big scholarly data has been surveyed in [60] and currently there are initiatives to publish open bibliographic citation information as linked data, such as Open Citations [42]. The feasibility of tools to explore such data has been demonstrated [22,38], facing challenges related to statistical analysis, semantic exploration and visual analytics. Scholarly data analysis is often based on knowledge graph analysis.

The usage of graph structures as conceptual-modeling allows the storing and querying of data based on the relationships among objects, which is necessary to model inter-dependencies among entities and for data exploration, also through faceted search [45]. Graphs can also be exploited for mining activities [45] and to model varying precision and accuracy. Graph modeling can be accomplished through a graph database [31] or via mappings that enable graph reasoning over traditional relational databases [9,50].

Recent research results on the generation of knowledge from large-scale analysis of experimental data in the combustion kinetic domain have been reported in [27]. Hansen and co-workers presented a collection of 55 experimental datasets in premixed laminar flames extracted from previous publications, showing how their analysis allows the extraction of fuel-specific rules that can be useful to identify inconsistent data. Along the same lines, [41] also collected and reviewed a large set of experimental data (60 datasets) on rich laminar premixed flames of hydrocarbon fuels reported in recent years, and analyzed them by means of the available kinetic models to describe the formation of polycyclic aromatic hydrocarbons. The Curve Matching approach [7] was applied in the latter case to evaluate model performances. This allowed the identification of inconsistencies not only between single models and experiments, but also between different models often describing the same phenomena in significantly different ways, yet still obtaining a comparable degree of agreement. From these recent literature examples, it is clear how a fully integrated framework with large-scale capabilities, as that described in this work, could be beneficial to both model development and experimental evaluation.

The needs which emerge from the scenario described in Section 2, like the continuous validation of models based on new experimental data, are very general and shared by other domains and contexts. Therefore, the solutions discussed in this paper have a general validity beyond the domain of combustion kinetics. Indeed, recent literature has highlighted “cases where datasets are reused in combination with other data, whether to make comparisons, build new models, or explore new questions altogether” [39], in fields such as computational biology [6], gene expression analysis [47], biomedical research [35], cell biology [21] or predictive structural materials science [1]. All these fields are characterized by the mathematical, physical and computational modeling of complex systems and behaviors, and the validation of such models against experimental data which typically come from research articles. Indeed, similar challenges have been discussed recently in these fields, particularly those related to data integration, interpretation and large-scale analysis [1,6,21,35,47], highlighting several shared open issues. The framework investigated in this paper has been currently tested and optimized only for the scenario described in Section 2, but most of the design choices were taken trying to generalize its applicability.

Large-scale validation of models with respect to experimental data is an emerging topic and other integrated systems have been proposed very recently in different domains. PRISMS [1] is an integrated framework for accelerating predictive structural materials science. It includes computational modules to predict microstructural evolution and the mechanical behavior of structural metals and a set of integrated scientific “use cases” where they are linked to experiments. It also includes a collaborative repository to archive and disseminate experiments and computational models. CaRMeN [25] is a tool that automates the workflow for comparing chemical kinetic models and experiments for catalytic process applications, overcoming the issues of a manual, time-consuming and error-prone validation. It contains reactor solvers for different kinds of reactors, different models to simulate the catalytic partial oxidation of methane and related experimental data for validation. Model performance assessment is based on parity plot diagrams and the graphical interface makes it user friendly. Our work shares several goals with these projects. However, our efforts are directed more towards the management of challenges coming from the use of heterogeneous and potentially noisy data sources and their semantic interpretation to enable new analysis directions, thus obtaining requirements which are not domain specific.

4.Towards an integrated framework

Model development and automatic validation through published experimental data has several facets. In order to analyze the directions of an integrated system against this goal, the main use cases are analyzed in Section 4.1. Moreover, a first extended prototype was developed and presented in Section 4.2. This will be the basis for the formulation of new requirements and an architecture for the complete framework.

4.2.Extended prototype

The new prototype has been developed to better analyze emerging use cases and requirements, and, therefore, we followed an agile development approach. For this reason, it already includes a set of fully working features, some partially developed ones, and a set of demo features developed to better refine requirements and to design the architecture. Much effort was spent in developing the core integrated infrastructure and in the abstraction and integration layers which will allow further development and this prototype’s evolution in a full framework. The source code of the prototype is available [48]. The full framework’s architecture, which was also designed and refined starting from this prototype and includes parts that are not yet developed, is described in Section 6.

Currently, the prototype includes the following (fully or partially developed) main features:

• Batch import of experiments into ReSpecTh [43,54] and ChemKED [58] formats.1212 Folders of such experiments can be manually imported, but automatic import/update directly from these repositories is currently not supported.

• Manual input of experiments extracted by domain experts through a mixture of interactive input fields and human-readable Excel and CSV files.

• An interface to add kinetic models to the system. A model is uploaded as a folder and associated to a name and a version.

• An operational database which stores experimental data, logical paths to models and the results of simulations and analysis performed by external tools. All these data have been described through a generic meta-model with generic fields to facilitate data integration and manage different sources. For example, experiments are described in terms of “boundary conditions” and “output variables”, and the interpretation of the actual values is dynamically delayed and supported by an external domain ontology.

• Automatic recognition, validation and cleaning of experiments based on an external ontology. Currently, a simple ontology has been defined using an Excel human readable file and populated by domain experts, as discussed below.

• An interface to browse experimental data and visualize their properties and output variables, also through interactive graphs with options to compare, highlight, zoom in, etc. This interface is general enough to include different experiment types, as long as they are described in the ontology.

• Automatic creation of simulation inputs starting from (one or more) experiment(s) and a set of models to be simulated, and the parsing and storing of simulation outputs. This allows the automaticalal simulation of every interpretable1313 experiment. Currently, only the OpenSMOKE++ simulator is supported.

• Search and filter experiments based on their properties, input species, and a generic boolean condition.

• External tools supported with a modular design. Given a set of experiments obtained as a result of a search query, it is possible to run a generic analysis on them which can be defined as a standalone function receiving each experiment with associated data as an input and new data which is stored and associated to each experiment or to the entire set as an output. This “search & execute” pipeline is the prototype’s core. Function chaining is not supported yet.

• Automatically executes the Curve Matching algorithm on a set of experiments and models to evaluate the models’ performance on each experiment. An interface to the Curve Matching algorithm [7] has been implemented as a module.

• Compute a global score for each model on a set of experiments, starting from the indices defined for each model/experiment pair by the Curve Matching. This function has been implemented as a module.

• An interface to browse simulation outputs and analysis results. Once an action has been executed on a query set, it is possible to browse its results both in a global view, which shows set-level results, and in a detailed view, which shows experiment-level results. These views are currently able to show model simulation outputs and Curve Matching validation indices, including global scores and derived graphs, but are not yet general enough to show the results of third party modules.

These features already cover part of the use cases described in Section 4.1. The main features currently missing from the prototype include those related to automatically “make sense” of the results, e.g. automatically understanding the reason why a certain model does not behave correctly on some experiments, also through the dynamic integration of other data sources, the dynamic acquisition and exploration of new experiments and the integration with external systems (e.g. WMSs and model development tools). The prototype helped to highlight challenges arising in the development of use cases, which led to the new requirements discussed in Section 5 and the architecture outlined in Section 6. The latter can be considered as this prototype’s future development.

The client application was developed as a web application to minimize client-side requirements. The server offers all its services through a REST API in JSON format, allowing integration with other applications and the possibility to develop alternative clients. Examples of JSON responses are shown in Fig. 4. User authentication is supported.

Fig. 4.

Examples of JSON responses from the REST interface. On the left, validation results are returned for a set of requested models and experiments. On the right, the list of experiments satisfying a certain query is returned and each experiment includes its metadata, boundary conditions and actual data (this response is not complete in the figure).

The prototype was implemented with PostgreSQL1414 as the operational database, using its extensions to support array data and semi-structured XML and JSON data. The backend was written in Python, using the Django1515 framework as a web framework with the Django-REST1616 extension for the REST API. Python libraries used in the backend include: numPy, sciPy, and pandas for data elaboration, pint to manage physical quantities and conversions, xlrd and elementtree for data conversion. The frontend is based on the React1717 and Ant Design1818 frameworks and Plotly1919 was used as the graphing library.

At the moment, the prototype stores more than 1,200 experiments. They include those available in the ReSpecth respository2020 and manually uploaded experiments, including the collection published within [27]. The number of experiments stored is growing constantly.

In the architecture, the understanding of experimental data leverages an external ontology which is populated by domain experts and, in the current prototype, is built starting from an Excel human readable file, available at [48]. This file was designed in the structure and populated in the content for the occasion. It allows the definition of existing categories of experimental data and the properties, variables and conditions required to simulate each category, including optional fields. This information makes it possible to perform data cleaning, correctly build the simulation input for experiments and interpret their output variables. Externally defining this information in a human-readable format leads to more flexibility and gives domain experts the possibility of extending support to new experiments without changing the core system. Experiments can be stored without semantic constraints as long as they satisfy schema constraints, but their semantic interpretation, which happens dynamically based on the ontology, is then required for their usage (simulation and analysis) in the framework. This allows, for example, the storing of experiments not yet manageable by the system, which are automatically managed as soon as their information is added to the ontology. At the moment, about one third of the stored experiments are interpretable by the prototype, but this number is growing rapidly as new domain knowledge is added. The ontological information developed for the current prototype could be substituted by a more complete and complex domain ontology based on a more flexible format. However, such a complete and open ontology does not presently exist for the domain and should be generated as discussed in Section 6.1.

Fig. 5.

Prototype Screenshot/1. Manual input of an experiment in the framework.

Fig. 6.

Prototype Screenshot/2. Search experiments by complex filter conditions and select the resulting subset.

Some client-side prototype screenshots are shown in the figures. Figure 5 illustrates the interactive interface to manually upload new experiments. Figure 6 shows the search form through which it is possible to filter experiments based on a generic query and to select them. Figure 7 illustrates the interface through which a list of models can be selected and then validated on the set of experiments previously filtered using Curve Matching with the OpenSMOKE++ simulator. Figure 8 shows this validation’s global results consisting in an average index in the range [0,1] for each model and each output variable. From there, detailed results can be browsed, as shown in Fig. 9. Detailed results include experiment/model curves and the validation index for each experiment, model and output variable.

Fig. 7.

Prototype Screenshot/3. Select the models to be validated and compared against the previously selected set of experiments.

Fig. 8.

Prototype Screenshot/4. Aggregated results are shown for all the experiments and models selected. An aggregated index in the range [0,1] is computed for each output variable and each model and is also shown as a bar chart.

Fig. 9.

Prototype Screenshot/5. Detailed results can be shown upon request; in this case the index in the range [0,1] is shown for each model/experiment comparison, together with a chart reporting raw experimental outputs (dots) and related simulated outputs for each model (lines).

5.New requirements

A new set of requirements have been formulated starting from the scenario presented in Section 2 and the use cases described together with the prototype in Section 4. The goal of these requirements is to enhance the efficiency and effectiveness of data management and to enable new exploration and analysis directions in this context. They are presented and discussed below, and will also drive the design of the new architecture presented in Section 6.

6.Proposed architecture

The framework’s designed architecture is outlined in this section. Starting from the previously-discussed requirements and the new prototype’s development, this section will focus primarily on architectural requirements needed to support the use cases described in Section 4.1. A detailed architectural design, together with its complete development and testing, are out of the scope of this paper and represent ongoing work started with [49] and continued with this paper.

As previously illustrated in Section 4.2, some components and features have already been developed in the prototype. Beside these, the following architecture includes other components and features that must be considered as not yet having been developed. The development of existing features in the prototype has proved to be crucial to refine the following architecture.

Section 6.1 below describes the data model, while Section 6.2 gives an overview of the global architecture.

6.2.Framework

The main architectural components are illustrated in Fig. 10.

Fig. 10.

Sketch of the architecture for the framework.

A central server includes all data sources, core services, the input and output layer, external analysis services, workflow management and data management. The central server’s design is largely based on a service-oriented architecture (SOA). Services are meant to be as independent as possible, keeping the communication layer simple and mostly mediated by the operational database. The latter, due to its general model and lazy interpretation of data, makes it possible to retrieve and store generic attributes associated with experiments and models or other attributes. This enables, for example, assigning a generic analysis result – with a service-defined name – as an attribute to a set of experiments or to an experiment/model pair. That result can later be retrieved by the same or other services. In this sense, with the exception of the shared operational database, the proposed architecture follows a microservice architecture design [3,30], characterized by largely autonomous services and a lightweight communication layer. This has been proved to enhance interoperability, scalability, integration, and independent development with respect to traditional service-oriented architectures. These features are particularly important for the system’s analysis services, which are those meant to be continuously developed, also by third parties, to enrich framework capabilities.

Core services provide the framework’s core functions: model validation, data and results exploration, and so on. These services are designed to interface with analysis services on one side, which provide actual simulation, computational and analysis results, and with the data management and integration layer on the other side, which provides access to the data sources.

The data management and integration layer handles the integration of the different data sources described in Section 6.1 (including the operational database), data cleaning and quality management. It allows core services to perform complex, exploratory and quality-aware queries over them. This layer should be able to efficiently search, retrieve and integrate data based on the patterns of the acquisition/validation/exploration cycle. For example, retrieving all the experiments needed to validate a certain model change.

Analysis services provide a wide range of functions to compute validation indices, statistical analysis, aggregated values, and so on. Model simulation is managed as an analysis service. Each function is wrapped in a service standardizing its interface. This allows their uniform management and the storage of their outputs as attributes with an integrated schema in the operational database. Each analysis outcome is stored to be reused as needed, even as a sub-step in a more articulated task or by a different user. Analysis outcomes are stored in the operational database through a generalized attribute-based interface provided by a core service. Analysis services are designed to be developed also by third parties.

A task in the framework is defined over the core services which, in turn, can refer to the data management layer or to analysis services for specific sub-tasks. The workflow management architectural component handles functions such as resource allocation, task scheduling and parallelization. This component could be an existing workflow management system (see Section 3) integrated in the framework.

Given the variety of existing formats and standards, the input and output layer should integrate conversion tools. As discussed in Section 6.1, external scientific repositories need a syntactic wrapper for format conversion, even if semantic checks and conversions are not required at input time. The input interface enables inserting or modifying experiments and models, while the output interface allows accessing both experimental/model data and analysis outcomes. On top of the input/output layer, a REST API makes it possible to interact with the framework and export data as JSON.

The web client accesses the server through REST API and provides a user friendly way of interacting with the framework. This includes searching, starting a task, browsing the results, adding experiments or making changes in models. Moreover, it includes additional client-side functions, such as plotting, graph editing, and data editing.

7.Concluding remarks

The effective integration of large-scale scientific data analysis in the development cycle of scientific models, especially in their validation, is one of today’s challenges, given the increasing availability of scientific open data and tools to automatically assess a model’s performance. This is potentially allowing rapid and extremely significant technological advancements. Starting from the well-defined domain of kinetic modelling of combustion processes, this work takes a more general perspective, analyzing use cases and emerging requirements for an integrated framework which aims to automatically exploit large-scale scientific data analysis in the scientific development cycle, especially in the validation phase. An extended prototype was developed to better describe and refine emerging requirements and a preliminary architecture was designed, thus setting the basis for its refinement and development in future activities. This work mainly focuses on data-related requirements, since data integration, interpretation and validation are a pre-requisite for every subsequent analysis task.

Identified requirements lay the foundations for future research directions. Moreover, future work includes integration with existing third-party projects, in particular, workflow management systems and general purpose data repositories, the proposed framework’s validation in other domains and more emphasis on analysis tasks which can be enabled by data management activities discussed in this paper. Analysis tasks should not only directly reflect users’ requests, but also be automatically derived by the framework based on higher level user-defined goals. In this direction, a domain knowledge which facilitates a better interpretation of acquired data and enables automatic reasoning on it has a key role. The envisioned role for the human domain experts in this context sees them defining higher level tasks (for example, to “execute a new model on all existing experimental data necessary to validate a certain range of a variable”) and, directly or indirectly, the knowledge necessary to interpret them. Future work also includes the proposed framework’s integration with other techniques in the model development cycle. In particular, an effective integration with automatic model development techniques, which constitute a recent and very promising direction in the considered domain [15,16], could highlight critical conditions that need further refinement in these models in a totally automatic development and validation cycle.