Mining natural language answers from the web

Article type: Research Article

Authors: Neumann, Günter | Xu, Feiyu

Affiliations: Language Technology Lab, DFKI, D-66123 Saarbrücken, Germany. E-mail: {neumann, feiyu}@dfki.de

Note: [] Corresponding author: Dr. Günter Neumann, Language Technology Lab, DFKI, D-66123 Saarbrücken, Germany. Tel.: +49 681 302 5298; Fax: +49 681 302 5338; E-mail: [email protected]

Abstract: We present a novel method for mining textual answers in Web pages using semi-structured NL questions and Google for initial document retrieval. We exploit the redundancy on the Web by weighting all identified named entities (NEs) found in the relevant document set based on their occurrences and distributions. The ranked NEs are used as our primary anchors for document indexing, paragraph selection, and answer identification. The latter is dependent on two factors: the overlap of terms at different levels (e.g., tokens and named entities) between queries and sentences, and the relevance of identified NEs corresponding to the expected answer type. The set of answer candidates is further subdivided into ranked equivalent classes from which the final answer is selected. The system has been evaluated using question-answer pairs extracted from a popular German quiz book.

Journal: Web Intelligence and Agent Systems: An international journal, vol. 2, no. 2, pp. 123-135, 2004