Natural language generation of biomedical argumentation for lay audiences

2.1.Analysis of conceptual model of domain

On initial review of the corpus, it was apparent that certain recurrent general concepts are used to convey a simple, mainly causal model of genetic inheritance and disease to its lay audience. For example, the text of Figure 2 can be analysed as referring to the following concepts (italicised): test (‘testing’), symptom (‘hearing loss’), test result (‘these results’), genotype (‘GJB2’), and biochemistry (‘Connexin 26’). A formal evaluation, reported in (Green 2005a), was performed to validate the inter-coder reliability of a set of eight concepts (listed in Table 1) for letters on single-factor autosomal genetic conditions. Using instances of these concepts, it is possible to manually reconstruct qualitative causal/probabilistic network graphs representing the domain content of a letter.55 The nodes of the graph represent the instances of concepts referred to in a letter (e.g. the patient's GJB2 genotype), while node types (e.g. genotype) correspond to the more abstract concepts shown in Table 1.

2.2.Analysis of argumentation

In addition to analysing the conceptual domain model underlying the letters, we analysed the argumentation schemes used in the letters. It is important to note that a challenge to identification of the argumentation schemes was that the arguments often require fine-grained discourse interpretation, including reconstruction of implicit premises and/or conclusions (Green 2010). For generality, the argumentation schemes were defined in terms of abstract properties of the conceptual domain models rather than domain-specific concepts. To support transparency, the premises of an argument were identified by function as data or warrant (Toulmin 1998), where the warrant is typically a defeasible generalisation linking data to claim.

For illustration, consider the argument partly expressed in sentences 10 and 11 of Figure 2. The implicit claim of this argument is that, in each of P’s cells, both copies (alleles) of the GJB2 gene have the mutation referred to as (MUTATION). The implicit data are that the patient has hearing loss. Note that the data were not provided explicitly since, not only would it be shared knowledge of the genetic counsellor and her client, but it is mentioned in the preceding text. The warrant is the causal path from the GJB2 genotype to hearing loss represented in the conceptual model of the domain. Sentences 10 and 11 describe the causal chain up to chemical equilibrium of the inner ear, but the path from that to hearing loss is only implicated. As this example illustrates, a writer may rely on a reader's ability to reconstruct implicit components of an argument. For someone who is not a domain expert, or skilled in processing argumentation, reconstruction of arguments could be a difficult task.

2.3.Analysis of discourse

Several aspects of discourse in the corpus were analysed. First, an RST analysis of several letters was performed. Along with published guidelines for writers of patient letters (Baker et al. 2002), the RST analysis informed the specification of the discourse grammar.66 Second, an analysis of three dimensions of arguments in four letters was done to inform argument presentation and linguistic realisation: order of components (claim, data, warrant), implicit components, and use of discourse cues associated with argument components. The results of that study are summarised in Section 5.1. The next three sections present the major components of the GenIE architecture.

3.1.Conceptual domain model

The KB represents the counsellor's beliefs about the domain and, before a letter is generated on a particular patient's case, must be updated with his beliefs about the case (findings, diagnosis, etc.). As described in Section 2.1, a corpus study was carried out to identify the set of eight recurrent general concepts in this genre (Table 1). The domain content of letters to be generated can be represented as network graphs whose nodes are instances of these eight general concepts. Arcs in the graph represent qualitative causal/probabilistic relations determined by the types of the nodes connected by the arcs. The design of domain models for GenIE is restricted to this set of node types and relations.

Restricting the models in this way has several benefits. First, it simplifies the computational process of selecting the content for arguments, since the generator does not have to filter out biomedical information that is not normally used in communication with a lay audience. Second, it guides domain knowledge acquisition since it is constrained by the set of available concepts and relations. Using this set of node types and relations, prototype KBs for a number of representative genetic conditions77 were manually implemented,88 each in little time. Third, this approach enables argumentation schemes derived from the analysis of the corpus to be specified in terms of abstract properties of a domain model, as will be shown in Section 4.

3.2.Computational representation

A KB is modelled computationally as a qualitative probabilistic network (QPN) (Wellman 1990; Druzdzel and Henrion 1993). Like a Bayesian network (Korb and Nicholson 2004), a QPN is a directed acyclic graph whose nodes represent random variables. However, in a QPN, qualitative constraints take the place of conditional probability tables. The qualitative constraints are defined in terms of the relations of qualitative influence S, additive synergy Y, and product synergy X.99 A variable A is said to have a positive qualitative influence on a variable B, written S⁺(A, B), if a higher value of A makes a higher value of B more likely. An analogous definition is given for negative influence, i.e. S⁻(A, B), if a higher value of A makes a lower value of B more likely. Each arc in a KB is described by a positive or negative influence relation. Since not all the variables are Boolean, our notation encodes threshold values of the domains explicitly. For example, positive influence is encoded as S+(<A,a>,<B,b>), i.e. when A reaches a (or higher), it is more likely that B will reach b (or higher).

Synergy relations X and Y are used to express a relationship between a set of variables and another variable. Negative product synergy, X−({A,B},<C,c>), is used to model the relation between a mutually exclusive set {A, B} of potential causes of an event <C, c> and the event, e.g. between a set of candidate diagnoses and presence of a certain symptom. Another example of its use in this domain is to model autosomal dominant inheritance; having one mutated allele of a gene fitting this inheritance pattern normally is sufficient to cause a health problem. Thus, if a child has an autosomal dominantly inherited disorder, then (since we inherit one copy of each pair of genes from each parent) the child must have inherited the mutation from one or the other of his parents. Zero product synergy, X0({A,B},<C,c>), is used to model the relation between a set of events {A, B} which are jointly necessary to cause the event <C, c>; e.g. in autosomal recessive inheritance of a disorder, the child must have inherited one copy of the mutation from each parent. Positive additive synergy, Y+({A,B},<C,c>), is used to model a situation where A enables B to result in C. Negative additive synergy, Y−({A,B},¬<C,c>}, where the notation ¬<C, c> denotes C<c, is used to model a situation in which A inhibits B from resulting in C.

Figure 3 shows part of a KB describing a patient believed to have cystic fibrosis (CF) based on his symptoms – growth failure and respiratory infections – as well as the test result showing an abnormal level of NaCl. All arcs in the figure are represented in the KB as positive influence relations (S⁺). In addition, the graph contains converging arcs, annotated Y⁺, which are represented as two additive synergy relations in the KB: Y+({TSP,BP2}, <TRP, abnormal>) and Y+({EP,PP2},<SP,true>). The first one can be glossed as: performing the test (TSP) enables the pancreas enzyme level (BP2) to be detected in the test result (TRP). The second one can be glossed as: the presence of bacteria (EP) enables the viscous lung secretions (PP2) to lead to frequent respiratory infections (SP). Note that the node types and possible relations among nodes are determined by the general conceptual model of genetic disease; the graph instantiates this model for CF; and the values of the variables, given below the graph, describe some of the expert's beliefs about a particular patient's case after the patient's test result is known.

Figure 3.

Part of KB for CF case. Concept types are from the generic model of genetic disease and inheritance. Variables describe (part of) the model of CF. States of the variables describe (part of) a particular patient's case.

Figure 4 shows another part of the same KB, focusing on the conceptual model of the probable source of the patient's inherited genetic disease. This part describes the expert's beliefs that the patient inherited two mutated alleles of the CFTR genotype, one from each parent, but that each parent carries only one copy of the mutation since neither shows symptoms of CF. Note that, for each parent, a history variable representing North European ancestry is used for representing an individual's higher risk of carrying one mutated CFTR allele in that population compared with other groups.

Figure 4.

Another part of KB for CF case. This part shows information relevant to claims about the patient's parents’ CFTR genotype.

4.1.Argumentation schemes

As a prerequisite to identification of argumentation schemes, we interpreted the conceptual model conveyed in a letter as it would be represented in a KB (as described in Section 3). Next, we analysed arguments in the letter in terms of their claim, data, and warrant.1010 In some cases, it was necessary to interpret the writer's intended message, constrained by the conceptual model of the domain, the discourse context, and knowledge presumed to be shared by writer and audience to reconstruct implicit components of an argument (Green 2010). The final step in deriving an argumentation scheme was to abstract from the terms of a specific conceptual model and represent the claim, data, and warrant in terms of variables and abstract properties (i.e. influence and synergy relations) of a KB.

To illustrate, an argumentation scheme describing the argument analysed in Section 2 is shown at the top of Table 2. The warrant of the argument is that there is a positive influence relation, or a chain of such influences, from A to B. (To simplify the presentation, a chain is denoted as S∗(<A,a>,<B,b>), i.e. S* is used to denote positive influence along a path of length one or more.) The data are that B has reached b. The claim (conclusion) is that A has reached a.1111 Of course, this is not a deductively valid conclusion. For instance, there could be an alternative explanation for B reaching b, if there were another variable C such that C reaching the value c would increase the probability that B reaches the value b.

Since, ideally, a writer would not use an argumentation scheme that he did not believe to be applicable, and to prevent the GenIE Assistant from constructing arguments that would not be constructed by the writer, an applicability constraint may be included in an argumentation scheme. For an argumentation scheme to be applicable, its applicability constraint must hold. To gloss the applicability constraint of the scheme described above, it says that there is no variable C in a negative product synergy relation X−({C,A},<B,b>) such that C has reached c. A violation of this applicability constraint is depicted in Figure 5(a). Note that the description of the applicability constraint actually implemented for warrants with a path length greater than one is not shown; it is more complicated, since one must consider alternative explanations for intermediate steps in the path from A to B.

Figure 5.

(a) C could be the explanation for B. (b) C could have failed to enable A to influence B. (c) C could have failed to jointly contribute to B. (d) C could have prevented B.

In general, the applicability constraint of an argumentation scheme was derived by consideration of qualitative constraints that could be used to encode a KB (as described in Section 3) and that represented potential exceptions to the applicability of the argumentation scheme. As will be discussed in Section 4.4, each condition of the applicability constraint can be thought of as a critical question that can be used to challenge the conclusion.1212

We now discuss the remaining argumentation schemes implemented in this version of the GenIE Assistant, followed by an example of how a typical argument can be composed from instances of several of the schemes.

A variant of the argumentation scheme described above is shown at the bottom of Table 2. For example, the following argument can be constructed from it: given the beliefs (as shown in Figure 3) that (data) the patient has had frequent respiratory infections and bacteria in his lung secretions, and that (warrant) having two mutated alleles of CFTR leads to viscous lung secretions and the presence of bacteria enables viscous lung secretions to lead to frequent respiratory infections, then (conclusion) the patient has two mutated alleles of CFTR.

Another argumentation scheme is shown in Table 3. For example, as warranted by an autosomal recessive inheritance pattern, the data that a child has two mutated alleles of a gene support the claim that each parent has at least one mutated allele of that gene.

In contrast to the above argumentation schemes, consider the argumentation scheme shown in Table 4. The warrant of the argument is identical to that given in Table 2. However, the data are that B has not reached b, and the claim (conclusion) is that A has not reached a. For example, lack of symptoms is evidence for the claim that an individual does not have the genetic condition associated with those symptoms. The applicability constraint of this scheme has three conditions, all of which must hold.

The first condition states that there is no variable C and positive additive synergy relation, Y+({C,A},<B,b>), such that C did not reach the value c; i.e. the explanation for B not reaching b could be that by not reaching c, C failed to enable A to influence B to reach b (Figure 5(b)). For example, if exposure to bacteria (C) is required to enable thickened mucous in the lungs (A) to lead to respiratory infections (B), then an argument for the absence of thickened mucous in the lungs based on the absence of respiratory infections is not applicable unless it is plausible that there was exposure to bacteria.

The second condition states that there is no variable C and zero product synergy relation, Y0({C,A},<B,b>), such that C did not reach c; i.e. the explanation could be that A and C are jointly required and the reason B failed to reach b was that C did not reach c (Figure 5(c)). For example, in an autosomal recessive inheritance pattern, a child with two inherited mutations of a gene received one mutated allele from each parent; thus, if the child does not have two inherited mutated alleles of a gene and one parent is believed to lack the mutation, then an argument for the absence of the mutation in the other parent based on the child's not having two inherited mutations is not applicable.

The third condition states that there is no variable C and negative additive synergy relation, Y−({C,A},¬<B,b>), such that C did reach c; i.e. the explanation could be that by reaching c, C prevented A from influencing B to reach b (Figure 5(d)). For example, if a patient is taking medication to treat a possible genetic condition, then failure to manifest the condition's symptoms does not support the claim that the patient does not have the condition.

The argumentation schemes shown in Tables 5 and 6 do not have warrants from the domain model as they represent commonsense reasoning patterns.

Another argumentation scheme identified in the corpus is shown in Table 7. For example, an autosomal dominant inheritance pattern fits the warrant of this scheme. Thus, the data that the child has exactly one mutated allele of a gene support the claim that either the mother or father (but not both) has at least one mutated allele of that gene. In a variant of this scheme (not shown in Table 7) found in the corpus and implemented in the GenIE Assistant, the warrant describes three mutually exclusive events.

While most of the argumentation schemes discussed so far describe arguments from effects (or lack thereof) to potential causes, the next argumentation scheme, shown in Table 8, uses as data an event that may either play a causal role in or signal risk of another event. For example, obesity is a risk factor for heart disease.

The first condition of the applicability constraint of the scheme in Table 8 is that there is no variable C and negative additive synergy relation, Y−({C,A},¬<B,b>), such that C did reach c; i.e. C may prevent A from influencing B to reach b. For example, medication may lower a risk factor for heart disease. The second condition is that there is no variable C and positive additive synergy relation, Y+({C,A},<B,b>), such that C did not reach c; i.e. C may fail to enable A to influence B to reach the value b. For example, the risk of heart disease caused by an inherited condition, familial hypercholesterolemia (FH), can be mitigated by reducing risk factors such as obesity. A related argumentation scheme (not shown) can be used to argue against the claim that B has reached (or will reach) b, given the same warrant as in Table 8, and the data that A has not reached a.

In summary, the argumentation schemes implemented in this version of the GenIE Assistant are those shown in Tables 2–8 and their variants discussed above. The schemes fall into three categories: an argument for/against a putative cause based upon the observation (or inference) of the presence/absence of its typical effect (Tables 2–4 and 7), an argument for/against the occurrence of a typical effect based upon observation (or inference) of the presence/absence of a causal or risk factor (Table 8), and common sense arguments (Tables 5 and 6).

4.2.Example of composition of argumentation schemes

Using the argumentation schemes shown in Tables 2–6, it is possible for the GenIE Assistant to compose a type of argument that is common in genetic counselling, namely, that both biological parents of a child who has an autosomal recessive disorder such as CF are carriers of the mutation responsible for the disorder. (The term carrier describes an individual with one normal and one mutated allele of a gene.) A KB representing this situation is shown in Figure 4. The argument for each parent is similar. The argument about the mother of a child with CF is diagrammed in Figure 6 in a notation that highlights the argument's structure and which argumentation schemes were used. For clarity, only data and claims are shown in the diagram; i.e. warrants are not shown.

Figure 6.

Composition of arguments.

The main claim that the mother is a carrier (has one mutated CFTR allele) is shown at the top of the diagram. The argument for that claim uses the Elimination argumentation scheme (Table 6). The argument has two premises: (1) that the mother does not have two mutated CFTR alleles and (2) that the mother has at least one mutated CFTR allele. The subargument for premise (1) uses the no effect to no cause scheme (Table 4), and is based on the evidence that the mother does not have symptoms of CF. The subargument for premise (2) uses the conjunction simplification scheme (Table 5), and is based on the claim that the both the mother and the father have at least one mutated CFTR allele. The subargument for that claim uses the effect to joint cause scheme (Table 3), and is based on the belief that the child has CF, i.e. two mutated CFTR alleles. Although not shown in the figure, arguments for the claim that the child has CF can be generated as well.

4.3.Argument generation

Given a request for all arguments for a given claim, the argument generator processes the request in two phases. The first phase, described in Figure 7, reconstructs all possible arguments for the claim. For example, given the goal to find an argument for the claim that the patient has CF (two mutated copies of the CFTR genotype), the argument generator would find that the claim of the effect to cause argumentation scheme (top of Table 2) can be unified with this goal; next that the warrant of the scheme can be unified with a chain of causal influences in the KB from this genotype to growth failure; next that the data of the scheme can be unified with the assertion in the KB that the patient has growth failure; and lastly that the applicability condition of this scheme holds, i.e. that there is no other condition believed to be applicable in this case. Next, another argument using the variant of the effect to cause scheme (bottom of Table 2) would be found.

Figure 7.

Pseudocode describing the first phase of argument generation.

In the second phase, the arguments created in the first phase are translated into equivalent RST structures, which are inserted into the discourse structure for the whole document, as follows. The data and claim of an argument are transformed into the satellite and nucleus, respectively, of an evidence relation. The warrant of an argument is transformed into the satellite of a background relation whose nucleus is an evidence relation representing the data and claim of the argument. The RST structure for two arguments for the claim that the patient has CF is shown in Figure 8.

Figure 8.

RST structure of arguments that the patient has CF. Left and right branches of background and evidence are satellite and nucleus, respectively.

4.4.Interactive argumentation

Interactive GenIE (Green 2008), a prototype interactive argument generation system, was implemented to enable a user to take the initiative in exploration of the space of arguments and counterarguments for a given claim. Interactive GenIE consists of a query-driven user interface, the same type of KB used by the GenIE Assistant, and an argument generator. Note that requests for arguments for a claim come from the user interface rather than from the discourse grammar as in the GenIE Assistant. As it was beyond the scope of this project to develop a natural language dialogue interface, communication between user and system is via an artificial language.

Argument generation by interactive GenIE differs from that by the GenIE Assistant in two respects. First, interactive GenIE does not search for any arguments in support of the data of an argument until so requested by the user. Second, in interactive GenIE, an argument need not satisfy the applicability constraint of the argumentation scheme from which the argument was constructed. Instead, each condition of the applicability constraint is treated as a critical question that the user may ask to challenge the acceptability of the argument.

The user may request pro or con arguments, i.e. arguments respectively for or against a claim C. In response, the system returns the arguments including data D, warrant W, and set S of critical questions.1313 The user may then request pro or con arguments for any of these components,1414 and so on. Figure 9 shows a dialogue with Interactive GenIE (glossed in natural language for purposes of illustration) on a case of osteogenesis imperfecta (OI) that is believed by the healthcare provider to be more likely due to a new mutation (mosaicism) than to be an autosomal dominant inherited mutation.1515 Transparency of arguments is increased in Interactive GenIE compared with the GenIE Assistant by enabling the user to request arguments that may not be presented in a patient letter, and by making applicability constraints visible as critical questions that the user may pose.1616

Figure 9.

Example of dialogue with Interactive GenIE (glossed).

5.1.Pragmatic features of argument presentation

First, we summarise the results of a study of pragmatic features of argument presentation in the corpus whose purpose was to inform the design of these components. Four letters in the corpus were analysed with respect to three dimensions relevant to argument transparency: order of argument components, enthymemes1717 (arguments with implicit components), and discourse cues marking argument components. One striking observation is the prevalence of enthymemes in the letters, reducing the number of full arguments that could be studied.

In arguments containing both explicit data and claim, the data are always given before the claim, and if present the warrant is never given between the data and claim.1818 Warrants appear both before and after the data and/or claim. Subarguments (for the data) are given before the claim of the main argument.

As for when components are omitted, the data of an argument is not given explicitly when (1) it is given in a previous section of the letter, (2) it is in the common ground,1919 or (3) it is supported by (i.e. is the claim of) a subargument. Contrary to the view that warrants are often implicit, warrants are often explicit in these letters. Furthermore, the claim is often omitted. It is given explicitly when (1) it is a weaker claim (e.g. the preliminary diagnosis), or (2) when it is not likely to be negatively valued by the audience (e.g. the claim that neither parent is likely to carry the mutation responsible for their child's condition). Conveying negatively valued claims implicitly is consistent with the guideline in this genre to use indirection to mitigate the emotional impact of negatively valued information on the client (Baker et al. 2002), which we shall refer to later as the stylistic goal of conveying empathy.

Finally, there were few discourse cues marking data, claim, or warrant. In the four letters studied, there were two uses of cues (‘therefore’, ‘then’) preceding a claim, and one use of a cue (‘since’) preceding the data of an argument.

5.2.Discourse grammar

A discourse grammar was implemented based upon an RST analysis of several letters in the corpus and professional guidelines for writing patient letters (Baker et al. 2002). Since the goal of implementing the GenIE Assistant was to investigate argument generation, the discourse grammar was implemented to cover the topic sections in this genre that make the most extensive use of argumentation. The discourse grammar is a set of rules2020 (implemented in Prolog) that construct a partial DPlan representing the RST structure and content of a document, not including the arguments. Rules of the grammar generate trees of RST relations. Leaves of a tree consist of content, in non-linguistic form, extracted by discourse grammar rules from the KB. For example, Figure 10 shows the structure generated by the grammar for the first section of a patient letter. In the figure, event propositions are glossed in italics.

Figure 10.

RST structure of first section of letter on patient who has CF. Left and right branches are satellite and nucleus respectively. Argument in Figure 8 replaces shaded node.

Several discourse grammar rules are shown in simplified form in Figure 11. Rule 1 describes the four main sections of a letter. Rule 2 describes the four subsections of the first section, and the structuring of these four subsections as elements of an RST Narration relation. Rule 3 describes the extraction of the patient's symptoms from the KB to create the left-most subtree in Figure 10, describing the reason for the patient's referral to the clinic.

Figure 11.

Some discourse grammar rules (simplified).

For each leaf in the DPlan that would be the claim of an argument, the argument generator is invoked (by a function call from the right-hand side of a grammar rule). The resulting RST structure, a list of all arguments for that claim, is inserted into the RST structure of the document instead of just the claim. For instance in Figure 10, the shaded leaf shows where the argument for that claim, shown in Figure 8, would be added to the DPlan. In this way, genre-specific discourse planning, implemented by the discourse grammar, is distinguished from argument generation. However, the content surrounding the arguments as well as the arguments are represented uniformly in RST to facilitate argument presentation and linguistic realisation.

5.3.Argument presenter

In the GenIE Assistant's architecture, the argument generator's responsibility for generating an argument is distinguished from the argument presenter's responsibility for making decisions about how an argument is to be presented. More specifically, its responsibility is to aggregate or delete components of arguments in the DPlan for the sake of discourse coherence and argument transparency in the generated text.2121 Unfortunately, coherence and transparency goals may conflict. To maximise transparency, each argument's data, warrant, and claim would be given explicitly. However, a commonly used NLG strategy for improving coherence of generated text is to make it more concise through aggregation and deletion (Reiter and Dale 2000).

The purpose of the study described in Section 5.1 was to use the corpus to gain insight into presentation strategies including when to omit information. However, it was found that many of the arguments avoid making explicit claims of negatively valued information. That is, in the corpus, in addition to coherence goals, empathy may take precedence over transparency. Since empathy is not a stylistic goal for the GenIE Assistant, it is not clear when the corpus should be used to inform the decision to delete components of arguments from the DPlan. Therefore, the argument presenter was implemented using hand-crafted heuristic rules that were refined by trial and error. The remainder of this section describes the six main heuristics implemented in this version of the system. As will be seen in Section 6, however, additional heuristics are needed.

One heuristic aggregates two conjoined propositions in the claim of an argument when they share the same logical predicate-argument structure. For example, the claim glossed as: the mother has one mutated copy of CFTR and the father has one mutated copy of CFTR will be aggregated into a structure glossed as: the mother and the father each has one mutated copy of CFTR.

Another heuristic removes a claim P derived by the conjunction simplification argumentation scheme (Table 5) if it would immediately follow the data (P and Q) of that scheme. For example, if the data of an argument is the proposition glossed as the mother has at least one mutated allele of CFTR and the father has at least one mutated allele of CFTR, the claim glossed as the mother has at least one mutated allele of CFTR would be pruned since it repeats part of the data of the argument.

In one of the test cases generated by the GenIE Assistant, a letter on FH, three consecutive arguments for the claim C that the patient could have FH are given. The data given in the first, second, and third argument are, respectively, D₀ and D₁ (The patient had a myocardial infarction and is a smoker), D₀ and D₂ (The patient had a myocardial infarction and is obese), and D₀ and D₃ (The patient had a myocardial infarction and has a low level of physical activity). In other words, some of the data, D₀, were repeated in each argument. Thus, a pruning heuristic is used to avoid repeating D₀ in the second and third consecutive arguments for C. In addition, the warrants of the three arguments have some overlapping content, which is handled by the following heuristic.

This heuristic is used to avoid repeating part of the same warrant in two or more consecutive arguments for the same claim C. Suppose that the warrant of the first argument for C is that G can cause P and P can cause S₁ (e.g. CF can cause abnormal CFTR protein. Abnormal CFTR protein can cause a viscous lung secretion.), and the warrant of the second argument for C is that G can cause P and P can cause S₂ (e.g. CF can cause abnormal CFTR protein. Abnormal CFTR protein can cause an abnormal pancreas enzyme level.). In the consecutive arguments for C following the first argument, the duplicated first part of the warrant, G can cause P, is pruned and marked for realisation using an adverbial such as also. The beginning of the warrant of the second argument in the preceding example would then be realised as Also, abnormal CFTR protein can cause an abnormal pancreas enzyme level (e.g. see paragraph one in Figure 12).

Figure 12.

Letter on CF case generated by GenIE Assistant.

A different heuristic is used to avoid repeating the same warrant in two consecutive, similar arguments referring to different individuals. Suppose that the warrant of each of the two arguments is that G can cause S (e.g. One changed copy of FGFR3 can cause Achondroplasia symptoms). Now suppose that the data and claim of the first argument are, respectively, D₁ (e.g. The patient's mother does not have any symptoms) and C₁ (e.g. The patient's mother is not a carrier) referring to individual I₁. Suppose that the data and claim of the second argument are, respectively, D₂ (e.g. The patient's father does not have any symptoms) and C₂ (e.g. The patient's father is not a carrier), where D₂ and C₂ are identical to D₁ and C₁, respectively, except that D₂ and C₂ refer to individual I₂. In the second argument, the warrant is pruned from the DPlan, and marked for realisation using an adverbial such as ‘similarly’. In this example, the second argument would then be realised as Similarly, the patient's father does not have any symptoms. Thus he is not a carrier (e.g. see paragraph three in Figure 13).

Figure 13.

Letter on achondroplasia case generated by GenIE Assistant.

While the above heuristic recognises two consecutive, analogous atomic arguments about two different individuals, in some cases, the DPlan contains two consecutive, analogous complex argument structures about two different individuals. A variation of the above heuristic is used in such cases to reduce the second argument structure to the ‘similarly’ adverbial, the data of the second argument, and the claim of the second argument (e.g. see paragraph three in Figure 12).

In summary, the above heuristics act upon elements of individual arguments or sets of arguments. In contrast, previous NLG research has focused on identifying semantic and/or syntactic structures that license aggregation (Shaw 1998; Dalianis 1999; Hielkema 2005; Harbusch and Kempen 2009).

5.4.Linguistic realisation

The linguistic realisation component transforms the DPlan, a forest of trees of RST relations and event propositions, into English text. An event proposition is described in terms of its modality and semantics, which is described by an action predicate and semantic case roles such as agent and beneficiary. The first phase of linguistic realisation maps the DPlan to a list of protosentences. A protosentence specifies one or more event propositions to be expressed in the same sentence in the final output. A protosentence that includes more than one event proposition specifies the rhetorical relation(s), e.g. Purpose, relating the events. The second phase of linguistic realisation maps each protosentence to a sentence specification. In this phase, the main NLG tasks performed are lexical choice and referring expression generation. The third phase of realisation maps the sentence specification to the input specification for SimpleNLG, an off-the-shelf surface realisation component (Gatt and Reiter 2009). Finally, SimpleNLG is used for morphological-syntactic processing.

In the rest of this section, only the aspects of linguistic realisation specific to argumentation will be described. During the first phase, clausal order of components of an argument is determined. Following the ordering observed in the corpus (as noted in Section 5.1), data (including subarguments) precede the claim. Although no pattern was observed in favour of providing the warrant before or after data and claim, it was decided to provide the warrant before data and claim since, as represented by its role of satellite in a Background relation, it provides background on the connection between data and claim.

In addition to determining clausal order, the first phase of realisation may add features to a protosentence for lexicalisation of discourse cues in the second phase of realisation. Informed by the use of such cues in the corpus, the only argument-specific information added is a feature subsequently realised as a sentence premodifier marking the claim of an argument. (As described in Section 5.3, features marking pruned components of arguments that are subsequently realised with adverbials ‘also’ and ‘similarly’ are determined by the argument presenter.) The feature is lexicalised as ‘thus’ for the claim of a subargument, and ‘therefore’ for a top-level argument. An example can be seen in the third paragraph of the generated letter shown in Figure 12. This paragraph contains three subarguments, each of whose claims is prefixed with ‘thus’, followed by the main claim, Therefore she could have one changed copy of the CFTR gene.

6.1.Goals and design of evaluation

A small controlled study of letters generated by the GenIE Assistant compared with letters written by a genetic counsellor on each of two medical cases was performed to determine if the system was generating argumentation ‘about as good’ as that written by the counsellor, and if not, to suggest areas for improvement in the Assistant. As a simple way of indirectly assessing argumentation, the study determined the number of edits made by domain experts to letters produced by the GenIE Assistant compared with letters written by a genetic counsellor. The participants performing the editing task were university graduate students, at least 18 years old, majoring in biology-related fields, who had taken college-level genetics courses,2222 and who were screened for English writing fluency. Participants were compensated for their time.

The evaluation was performed using letters on two fictitious patients with substantially different conditions – CF and achondroplasia, which together cover all three mechanisms of inheritance of single-gene autosomal genetic disorders. Information for the cases was obtained from a genetics reference book (Nussbaum, McInnes, and Willard 2001). Seventeen participants were randomly assigned to one of four conditions representing the four combinations of authorship (genetic counsellor or GenIE Assistant) and genetic condition (CF or achondroplasia).2323

It was decided to evaluate GenIE's argumentation with an editing task for a number of reasons. First, if we had not presented participants with the output of end-to-end generation, but instead had rendered the generated letters (or just the arguments) into English text ourselves, we might have inadvertently affected the judgment of the participants. Second, the editing task did not require the participants to have had training in argumentation. Third, the editing task has some ecological validity since, if the GenIE Assistant were deployed, it is expected that its output would be edited by a genetic counsellor to ensure accuracy as well as readability. Editing was used by Sripada, Reiter, and Hawizy (2005) to evaluate an operational NLG system that produces draft weather reports which are edited by human weather forecasters before being released to readers.2424 Finally, the results of the editing task could provide insight into areas of needed improvement, e.g. in argument presentation and linguistic realisation.

6.2.Materials and procedure

To generate a text for evaluation purposes, first, it is necessary to implement a KB2525 representing the relevant genetic conditions2626 and update it with the experts’ presumed beliefs about the patient's case. No changes in the argument generator are required to handle different genetic conditions or patient cases.2727 The argument presenter's heuristics should be applicable to other genetic conditions. The linguistic realisation component was implemented to enable the formal evaluation to be performed; therefore parts of it, such as lexicalisation rules, have only been implemented to cover test cases.

Each of the two human-authored letters used in the study was created by asking a professional genetics counsellor to write a patient letter given the same findings, diagnosis, etc. given to the GenIE Assistant for each letter. To control for length, subtopics in the human-authored letters not covered in the three sections of GenIE's discourse grammar were removed by the experimenter. The two letters that were generated by GenIE2828 and used in the study are shown in Figures 12 and 13. The letter generated on CF contains all of the same types of arguments, except one,2929 as a letter in the corpus on an analogous case of a patient with an autosomal recessive disorder (hearing loss due to GJB2 mutation) inherited from parents who were carriers. The letter generated on achondroplasia contains exactly the same types of arguments with the same argument content as a letter on a similar case of achondroplasia in the corpus.

Each participant was given a copy of a letter, printed double-spaced on paper, and was asked to read it and manually edit the paper copy to make the information more understandable to people without a background in medicine or genetics. Participants were not informed that the letter may have been generated by computer.

6.3.Results and discussion

To score the editing task, each participant's edits were classified and counted by the first author of this paper as follows:

All categories except reordering are mutually exclusive; e.g. a string of text can be counted as both rewording and reordering, but not as both rewording and aggregation. The experimenter's judgment as to the participant's intention was used to decide in cases where more than one classification might apply by the above criteria. Note that it would be quite challenging to assess argument content, presentation and realisation independently of each other and independently of the rest of the letter. The above scoring method conflates these factors. For example, the counted edits may include edits to text in the letter that is not part of any argument. Argument content and presentation is conflated since no distinction is made, e.g. between deletion of duplicate argument content and deletion of non-duplicate argument content. However, given the formative goals of the evaluation, it was decided to adopt a simple approach to scoring.

The ‘bottom-line’ results of the evaluation are not surprising, as shown in Table 9. For each letter, the mean total number of edits was higher for the GenIE-produced letter than its human-authored counterpart.3030 This is not surprising since the experiment compared the output of a human writer with the output of a proof-of-concept NLG system. However, it is important to look at the means for each category of edits to see what the experiment suggests about argument content as well as what areas of improvement might have the greatest impact on the results.

The categories that are most relevant to assessing GenIE's argument content are adds and deletes. In terms of adds, subjects actually made fewer additions to GenIE-produced letters than to human-authored letters. On review of the delete edits made to the GenIE-generated letters, it was observed that the majority were clauses that repeated data or claims given elsewhere in the letter. (No delete edits were made to the human-authored letters.) It would be straightforward to implement additional argument presenter heuristics to prune such clauses.

On review of the agg/disagg edits, it was observed that almost half involved combining two clauses by making one a relative clause of the other and that most did not involve argumentation. Those that did involve argumentation concerned combining arguments about the patient's mother and father, i.e. instead of discussing each of the parents individually. Finally, analysis of rewording edits suggests that overall text quality could be substantially improved by modifications to the linguistic realisation of referring expressions.

The results suggest that, although improvement is needed to eliminate redundancy and polish the expression in natural language, the argument content produced by the Assistant is comparable to that produced by the genetic counsellor.