Concordance-based entity-oriented search

Article type: Research Article

Authors: Bautin, Mikhail | Skiena, Steven

Affiliations: Department of Computer Science, Stony Brook University, Stony Brook, NY 11794-4400, USA. E-mail: {mbautin,skiena}@cs.sunysb.edu

Note: [] Corresponding author.

Abstract: We consider the problem of finding relevant named entities in response to a search query over a given text corpus. Entity search can readily be used to augment conventional web search engines for a variety of applications. We use entity concordance documents to generate lists of relevant entities for arbitrary text queries. To assess the significance of entity search, we analyzed the AOL dataset of 36 million web search queries with respect to two different sets of entities: namely (a) 2.3 million distinct entities extracted from a news text corpus and (b) 2.9 million Wikipedia article titles. The results clearly indicate that search engines should be aware of entities, for under various criteria of matching between 18–39% of all web search queries can be recognized as specifically searching for entities, while 73–87% of all queries contain entities. Our entity search engine creates a concordance document for each entity, consisting of all the sentences in the corpus containing that entity. We then index and search these documents using open-source search software. This gives a ranked list of entities as the result of search. Visit http://www.textmap.com for a demonstration of our entity search engine over a large news corpus. In the case where the query is a named entity, we evaluate the performance of our system by comparing the results of our search engine to the list of entities that have highest statistical juxtaposition scores with the queried entity. Juxtaposition score is a measure of how strongly two entities are related in terms of a probabilistic upper bound. The results show excellent performance, particularly over well-characterized classes of entities such as people.

Keywords: News and blog analysis, natural language processing, entity search, web search, text processing

DOI: 10.3233/WIA-2009-0170

Journal: Web Intelligence and Agent Systems: An International Journal, vol. 7, no. 4, pp. 303-319, 2009