Mining consumer product data via latent semantic indexing^☆

Article type: Research Article

Authors: Jiang, Jingqian^a | Berry, Michael W.^{a; *} | Donato, June M.^b | Ostrouchov, George^c | Grady, Nancy W.^d

Affiliations: [a] Department of Computer Science, University of Tennessee, 107 Ayres Hall, Knoxville, TN 37996-1301, USA | [b] Data Exploration and User Environment Technologies Group, Computer Science and Mathematics Division, Oak Ridge National Laboratory, P.O. Box 2008, Building 6010, Oak Ridge, TN 37831-6414, USA | [c] Statistics Group, Computer Science and Mathematics Division, Oak Ridge National Laboratory, P.O. Box 2008, Building 6012, Oak Ridge, TN 37831-6367, USA | [d] Data Exploration and User Environment Technologies Group, Computer Science and Mathematics Division, Oak Ridge National Laboratory, P.O. Box 2008, Building 6012, Oak Ridge, TN 37831-6367, USA

Correspondence: [*] Corresponding author. E-mail addresses: [email protected] (J. Jiang), [email protected] (M.W. Berry), [email protected] (J.M. Donato), [email protected] (G. Ostrouchov), [email protected] (N.W. Grady)

Note: [☆] This research was sponsored by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by Lockheed Martin Energy Research Corp. for the US Department of Energy under Contract No. DE-AC05-96OR22464.

Abstract: One important focus of data mining research is in the development of algorithms for extracting valuable information from large databases in order to facilitate business decisions. This study explores a new technique for data mining – latent semantic indexing (LSI). LSI is an efficient information retrieval method for textual documents. By determining the singular value decomposition (SVD) of a large sparse term-by-document matrix, LSI constructs an approximate vector space model which represents important associative relationships between terms and documents that are not evident in individual documents. This paper explores the applicability of the LSI model to numerical databases, namely consumer product data. By properly choosing attributes of data records as terms or documents, a term-by-document frequency matrix is built from which a distribution-based indexing scheme is employed to construct a correlated distribution matrix (CDM). An LSI-like vector space model is then used to detect useful or hidden patterns in the numerical data. The extracted information can then be validated using statistical hypotheses testing or resampling. LSI is an automatic yet intelligent indexing method. Its application to numerical data introduces a promising way to discover knowledge in important commercial application areas such as retail and consumer banking.

Keywords: Consumer product data, Correlated distribution matrix (CDM), Latent semantic indexing (LSI), Data mining, Demographic variables, Purchase behaviors

DOI: 10.3233/IDA-1999-3505

Journal: Intelligent Data Analysis, vol. 3, no. 5, pp. 377-398, 1999