NICFS: A novel feature selection method applied to lexicon based sentiment analysis

Article type: Research Article

Authors: Mehta, Poornima | Chandra, Satish^*

Affiliations: Department of CSE, Jaypee Institute of Information Technology, Noida, India

Correspondence: [*] Corresponding author: Satish Chandra, Department of CSE, Jaypee Institute of Information Technology, Noida, UP, India. Tel.: +91 120 2594253; E-mail: [email protected].

Abstract: The technique used for feature selection immensely affects the performance of classification in case of high dimensional datasets. The bag-of-word model is often used for sentiment classification using machine learning. The set of unique words in a text based dataset constitute the feature vector which has a high dimension. In this work, a new feature selection method has been proposed which chooses features that are highly relevant to the class using the Information Gain score and least redundant with respect to the selected feature set using the CHI square statistic. The two scores were normalized to make them comparable. The proposed method was applied to lexicon based sentiment analysis using the lexicon SentiWordNet. Previously, IG and mRMR have been shown to be the best filter feature selection methods for sentiment term selection. The performance of our proposed method in terms of classification accuracy in sentiment analysis is significantly higher than IG and mRMR methods. Experiments were performed on three datasets in different domains. Also, the feature subset obtained by removing words with zero polarity score in SentiWordNet led to a significant reduction in feature vector size with no effect on performance.

Keywords: Feature selection, sentiment analysis, sentiment lexicon, Information Gain (IG), Maximum Relevance and Minimum Redundancy (mRMR), Normalized IG and CHI feature selection (NICFS)

DOI: 10.3233/IDT-190361

Journal: Intelligent Decision Technologies, vol. 13, no. 1, pp. 41-48, 2019