Analysis and study on text representation to improve the accuracy of the normalized compression distance

Article type: Research Article

Authors: Granados, Ana

Affiliations: Department of Computer Science, Escuela Politécnica Superior, Universidad Autónoma de Madrid, Madrid, Spain. E-mail: [email protected]

Abstract: This thesis takes a small step towards better understanding both the nature of texts and the nature of compression distances. Broadly speaking, the way in which this is done is exploring the effects that several distortion techniques have on one of the most successful distances in the family of compression distances, the Normalized Compression Distance (NCD). The experimental results show that changing the representation of texts applying one of the explored distortion techniques can be beneficial both in NCD-based document clustering and in NCD-based document search.

Keywords: Algorithmic information theory, normalized compression distance, data compression, information filtering, word removal, text representation, document clustering, document retrieval

DOI: 10.3233/AIC-2012-0529

Journal: AI Communications, vol. 25, no. 4, pp. 381-384, 2012