Increasing papers’ discoverability with precise semantic labeling: The sci.AI publishing platform

1.Key objectives of the study and its significance

Over the last two decades, life sciences articles have become substantially more complex, reflecting technological evolution, particularly OMICs experimentation, increasing cooperation between multiple institutions, and involving more advanced math and statistics applied to the data. In many publications, plain unstructured text is supported by algorithms, code, and multiple files of processed and raw datasets with annotated metadata and graphs. With such enhancements in place, experimental articles, per se, might become a driving force of the Literature Based Discoveries (LBD) [10]. Recently, the whole field of “meta-analysis” has arose to describe “dry lab” studies on normalization, unification, and analysis of many similar datasets derived from different labs and projects. However, a number of experimental papers are missed, because they cannot be retrieved from the body of literature by keywords search. Needless to say that scientists are keenly interested in higher discoverability of their published research and referencing to their findings, as citation index becomes an increasingly prevalent metrics in evaluation of their work. The issue can be addressed with proper semantic labeling of the texts as the very first step in global analysis of the research reports.

The state-of-the-art section reflects the ways published papers are algorithmically processed in text mining applications. Such computations rely on preexisting, statistically supported information, while text mining of the scientific literature targets novel findings. This leads to Information Retrieval (IR) and then Information Extraction (IE) underperformance when applied to scientific literature.

The objective of the paper is to consider just one issue of many in biomedical texts processing: false terms recognition caused by ambiguity of the concepts’ names and multiple-terms spelling variants. This leads to at least two undesirable effects:

We address:

2.State of the art in the field of biomedical texts semanticization

Computational Linguistics is one of the most dynamic fields with innovations being released almost monthly. Unfortunately, there is no solid state-of-the-art solution for biomedical text labeling yet that unites all the latest advances in each subfield into a single package.

The first subfield is metatagging standards and paradigms. Semantic Web’s objects, concepts, knowledge association, and data representation utilize schema.org vocabularies and W3C RDF/XML [11]. A current limitation is the lack of a similar single schema for the life sciences. Former related initiatives here are W3C Scholarly HTML [13] and JATS4R [8]. Still both schemas do not provide a standard namespace for biomedical concepts labeling.

The second subfield is terms labeling or Named Entities Recognition.

Just as in many other areas, deep learning and neural networks (NN) methods are increasingly popular for extracting information from professional texts [1,20]. NN algorithms are rather generic and can be applied for the text analysis in unsupervised fashion (i.e., to a variety of texts without establishing prior rules generation). However, its precision and recall hardy depend on statistical data and cannot be considered as stable solution for concepts and challenges that have appeared recently. Still, NN demonstrates the highest recognition rates [6,12] among automatic methods.

Then there are methods of increasing precision by reducing concepts ambiguity by connecting the same concepts in various ontologies.

UMLS (Unified Medical Language System), in combination with MetaMap, provide graph-like links between objects from various ontologies, as widely-accepted solution for the Word Sense Disambiguation (WSD) task in the biomedical domain. Essentially, UMLS represents a metaontology of biomedical terms and concepts. UMLS is extensive and well supported by NIH, and it is in constant development. Future considerations include possibly connecting this data to the sci.AI application. Currently, there are several limitations:

SciGraph by the Neo4j [15] framework allows objects to be interconnected and can be used as a technical basis for future metaontologies.

3.Design and methodology

Resolving terms’ ambiguity and variability represents a significant challenge in text processing. Here, we investigated how these factors affect the paper’s influence. Such causality is assumed based on the logic that findings described in the paper can be reused and cited – only if the paper will be discovered by the readers first. To model paper’s influence potential mathematically, we defined Paper’s Influence as a function of a variable we called Discoverability.

The paper’s potential for influence greatly depends on how accurately search engines and aggregators solve the IR task. For further explanation, we will continue with our query scenario from above. A reader is discovering the paper about animal ‘cat’ after querying string Qi ‘genes of cat’, including term tij ‘cat’ while meaning biomedical concept ci ‘Felis catus’, corresponding to object ‘9685’ in the ontology [17]. Concept ci then can be referenced with any term (spelling variant) tij of the set Ti

If we assume that a reader will read the paper if the search engine returned it in response to the query Qi, then “discoverability” is a synonym of “retrieval”. We can then apply two major IR metrics, recall and precision:

As long as such cases could be found across biomedical terminology, when concept can have several synonyms (variable terms) or single term can refer to several concepts (ambiguous terms), probabilistic precision and recall can be calculated based on the numbers of possible outcomes when querying Qi⊇tij. For example, texts “TNF alpha”, “TNFa” and “TNF α” are variants of the object Uniprot [P01375]. This means that if search engine was queried with “TNF alpha”, an ideal result would return all documents that contain all three variants. Still, due to existence of the several variants, the there is a probability⩾0 that some of them will not be considered.

We can estimate chances of such event using a basic definition of the probability as the ratio of the number of favorable outcomes to the total number of possible outcomes. Term’s ambiguity and variability define those numbers of possible outcomes. Finally, when we know precision and recall of the paper’s retrieving while searching for the concept ci, we can answer specific questions about discoverability of the paper in some kind of progression order.

This means that precision for the specific concept does not depend on the number of variants, as long as we assume that all variants are describing the same concept in the event.

This means a probability of two independent events: A=the concept has spelling variants, and B=those variants can be found in several concepts. Therefore P (A and B) will be multiples of the probabilities above:

If operating only with the number of variants per concept, then prior probability of variants occurrence can be approximated as uniformly distributed, as long as actual frequency of terms occurrence in the papers will be retrieved in the next steps. This means that, in the first approximation, occurrence={True,False} of the term can be sufficient variable to estimate the minimum expected probabilities:

As long as the Number of the terms per concept⩾1 (each object has at least a main name) and Number of concepts per single term variant⩾1 (each term is related to at least one object) and they are in the denominator of the retrieving probabilities above – terms’ variability and ambiguity will always reduce (at least, will not increase) recall and precision, respectively, when searching for the paper.

In order to estimate influence of the existing terms’ uncertainty on the papers discoverability, we have searched for:

MeSH Categories G–Z were not analysed because they contain generic objects, such as countries’ names, which are out of scope of sci.AI semanticization for now.

Our research is ongoing and the latest results can be found on the sci.AI webpage [14].

We had not only considered synonyms that exist in the ontologies but also created a rules-based term variant generator (TVG) to cover a case when the same object, Uniprot [P01375], might be written as “TNF alpha”, “TNFa”, or “TNF α” in a paper. Next generating techniques groups were utilized:

Table 1 shows average number of original terms’ synonyms and how much variants were generated. Then we’ve searched for them in the papers. There is increase of the concept detection of 2.03 – 3 times more when searching for all variants.

Table 2 shows how much objects has terms with identical spellings, i.e. ambiguous terms. Higher overlap within the same ontology than across other ontologies makes algorithmic recognition even more challenging tasks, because algorithms have to distinguish objects within the same class.

Figure 1 shows overall influence of the variability and ambiguity of the terminology on paper’s discoverability.

Fig. 1.

Influence of the terminology variability and ambiguity on a paper retrieval when text contains proteins, chemical elements, genes, drugs, diseases.

Fig. 1.

(Continued.)

When searching by original synonyms only, average likelihood of finding papers is lower than searching by all possible variants. Retrieving higher amount of the papers can be done at the cost of their relevance. Increasing amount of variants leads to the drop of the probabilistic precision. Relevance can be guaranteed only in case of labeling terms and searching by exact ID instead of a string. As long as current literature is not labeled, exact recall and precision can’t be calculated for. We used relative changes instead, to visualize scale of the issue across most of the available literature.

Figure 2 shows the distribution of the variability across ontologies.

Fig. 2.

Number of objects that have specific amounts of original synonyms, generated variants and variants found in the papers.

Fig. 2.

(Continued.)

Fig. 2.

(Continued.)

There are original synonyms in the ontologies, generated variants and how much of them were found in the papers. It shows that there are significant chances to found term’s spelling that was never mentioned in the ontology. Thus, it reduces recall of the relevant papers.

We were going to use the MSH WSD Data Set [19] initially for the ambiguity testing purposes, but it turned out to contain generic words only. So, we performed a generic wide search across all ontologies and variants to obtain low-level detalization.

There is also a case of “artificial” ambiguity in ontologies. It is caused by intersection of alternative names and terms’ descriptions in attempt of extending variants to increase recall. “Carbon monoxide” example is provided in the Table 3.

This leads to the necessity of human validation of the same concepts identification. Such functionality exists in sci.AI to validate several ID’s from the various ontologies for the same term and relationships identification (Fig. 3).

4.Decreasing retrieving uncertainty with the precise semantic labeling feature of the sci.AI platform

Formalization and statistics above show that uncertainty is not an exception but basic feature of the biomedical text mining. This uncertainty might lead to significant deviations when interpreting academic papers with unsupervised methods only. While it might be acceptable for fiction literature mining, because the major task there is context and sentiments analysis that acts as a smoothing function – such uncertainty might contradict goals of mining STEM research communication, where we are looking for the anomalistic or novel discoveries, exact objects interactions, verification of facts, and relations between statements in various texts. This is why accepting uncertainty might have significant negative consequences on the LBD.

In order to address this issue, we implemented the sci.AI platform that has supervised labeling functionality on top of the text mining framework. After initial automatic terms recognition, no matter whether precision is 70% or 99%, users can make final verifications to level up recognition precision to 100%. From the perspective of the search and text mining algorithms, this means removing any uncertainty, which, in turn, leads to exact papers extraction in an SQL-like querying manner. Thus, assuming that author will always label terms correctly, maximum precision and recall will be achieved.

Human-made corrections will be used as training data for the next processings of a text. Such learning with human feedback provides steady path to gradient growth of text mining quality.

Current version of the sci.AI allows to upload text, then performs Named Entities Recognition (NER) task automatically. Author or annotator can validate labeling results via interface and export final structured text to the XML file (Fig. 4).

Development roadmap includes release of the next features:

The system can be embedded into the publishing process directly. Both authors and editors can create semanticized versions during submission of even publish new version of the digital paper. Key features of the current production version of sci.AI publishing engine are as follows:

sci.AI allows labeling a term with several term-2-ontologyId relationships. For example, ‘serotonin’ is the object UID=D012701 in MeSH and UID=28790 in ChEBI ontologies simultaneously. Author or reader can suggest additional UIDs too or can correct existing one. This functionality contributes to the global connectivity between terms and might support UMLS Word Sense Disambiguation (WSD) works.

We expect another possible positive effect for the future papers too. Authors might come to the same single spelling variant of the concept, like “ClNa” only and not “salt” or “NaCl”.

As of the middle of 2017, sci.AI is production ready semanticization and metadata generation tool for academic publishing. Roadmap plan is to become a full-fledged artificial intelligence (AI) platform applied to the life sciences for hypothesis generation and machine reasoning. Wide adoption of this application will extend the publisher’s role even further into research results delivery to the intended target audience.

5.Discussion

This paper is the first in our series of researchers about precise semantic labelling of life sciences texts. Our goal was to focus on dependency of the paper’s influence on two fundamental factors: ambiguity and variability of terms. In order to avoid excessive complication, we made several assumptions which may bias the results. These simplifications will be addressed in follow-up studies: