World Wide Web site summarization

Article type: Research Article

Authors: Zhang, Yongzheng | Zincir-Heywood, Nur | Milios, Evangelos

Affiliations: Faculty of Computer Science, Dalhousie University, Halifax, NS, Canada B3H 1W5. Tel.: +1 902 494 7111; Fax: +1 902 492 1517; E-mail: {yongzhen, zincir, eem}@cs.dal.ca

Abstract: Summaries of Web sites help Web users get an idea of the site contents without having to spend time browsing the sites. Currently, manually constructed summaries of Web sites by volunteer experts are available, such as the DMOZ Open Directory Project. This research is directed towards automating the Web site summarization task. To achieve this objective, an approach which applies machine learning and natural language processing techniques is developed to summarize a Web site automatically. The information content of the automatically generated summaries is compared, via a formal evaluation process involving human subjects, to DMOZ summaries, home page browsing and time-limited site browsing, for a number of academic and commercial Web sites. Statistical evaluation of the scores of the answers to a list of questions about the sites demonstrates that the automatically generated summaries convey the same information to the reader as DMOZ summaries do, and more information than the two browsing options.

Journal: Web Intelligence and Agent Systems: An international journal, vol. 2, no. 1, pp. 39-53, 2004