Affiliations: Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan, #1 Roosevelt Rd. Sec. 4, Taipei 106, Taiwan. E-mail: [email protected], [email protected]; [email protected]
Correspondence:
[*]
Correspondence author: C.-Y. Chen is now with the Graduate School of Biotechnology and Bioinformatics, Yuan Ze University, Chung-Li 320, Taiwan. Tel.: +886 3 463 8800 ext 2185, Fax: +886 3 463 8850; E-mail: [email protected].
Abstract: This paper presents a novel mechanism to analyze and summarize the statistical correlations among the attributes of a data cube. To perform the analysis and summarization, this paper proposes a new measure of statistical significance. The main reason for proposing the new measure of statistical significance is to have an essential closure property, which is exploited in the summarization stage of the data mining process. In addition to the closure property, the proposed measure of statistical significance has two other important properties. First, the proposed measure of statistical significance is more conservative than the well-known chi-square test in classical statistics and, therefore, inherits its statistical robustness. This paper does not simply employ the chi-square test due to lack of the desired closure property, which may lead to a precision problem in the summarization process. The second additional property is that, though the proposed measure of statistical significance is more conservative than the chi-square test, for most cases, the proposed measure yields a value that is almost equal to the z test, a conventional measurement of statistical significance based on the normal distribution. Based on the closure property addressed above, this paper develops an algorithm to summarize the results from performing statistical analysis in the data cube. Though the proposed measure of statistical significance avoids the precision problem due to having the closure property, its conservative nature may lead to a recall rate problem in the data mining process. On the other hand, if the chi-square test, which does not have the closure property, was employed, then the summarization process may suffer a precision problem. In this paper, we also show a possible application in bioinformatics. We applied the proposed mechanism on a microarray dataset, in order to identify groups of genes with similar expression patterns in subspaces of feature space.
Keywords: correlation, data cube, statistical test, segmentation, summarization
