Emotion recognition from multichannel EEG signals using K-nearest neighbor classification

1.Introduction

Human emotion includes not only the psychological reaction of a human being to the external world or self-stimulation but also the physiological reaction to these psychological reactions. Human emotion is a combination of human thinking, feeling and behavior.

The role of emotion is ubiquitous in people’s daily lives and work. Analyzing and estimating emotions has become an important interdisciplinary research topic in the fields of psychology, neuroscience, computer science, cognitive science and artificial intelligence.

In recent decades, researchers in multiple fields have proposed various methods for emotion recognition. They can be divided into three main methods. One is based on the study of non-physiological signals such as facial expressions [1] and speech [2]. The advantage of emotion recognition using non-physiological signals is that it is easy to perform and does not require special equipment. The disadvantage is that people can disguise their true emotional states by disguising their facial expressions and phonetic intonations. Thus, the reliability of emotion recognition is not ensured. Moreover, the non-physiological signal recognition method cannot be used for people with disabilities or special diseases. The second approach is the study of physiological signals such as electroencephalogram (EEG) [3], electromyography (EMG) [4], electrocardiogram (ECG) [5], skin resistance (SR) [6], heart rate, pulse rate [6] and so on. These physiological indicators are intrinsic manifestations that are independent of the individuals’ control. Therefore, they are more appropriate and effective for emotion recognition. Unlike other physiological signals, EEG is a noninvasive technique with good temporal resolution and acceptable spatial resolution. Therefore, EEG may play a major role in detecting emotions directly in the brain at higher spatial and temporal resolutions [7]. The third method is emotion recognition based on multimodal fusion. For example, Busso et al. identify four emotions (sadness, anger, happiness and neutrality) by blending facial expressions and speech. Liu et al. fuse EEG signals and EMG signals with the highest classification accuracy of 91.01% [8]. Koelstra and Patras fuse facial expression and electroencephalogram (EEG) signals in the valence and arousal dimensions for emotion classification and regression. And the valence is the pleasure degree of the individual, and the value of valence is changed from small to large, indicating that the emotion is changed from negative to positive. Arousal indicates the degree of activation of the emotion, and the change of arousal from small to large indicates the change of emotion from calm to excitement.

2.Related research

Although different methods are used in emotion recognition with EEG, the ultimate goals of the research are the same. One of the goals is to find suitable features for emotion recognition through different analytic methods and use an optimized classification model to classify and improve the accuracy of emotion classification. Another goal is to identify the most relevant frequency bands and brain regions for emotion recognition activities and provide a solid physiological basis for EEG-based emotion recognition research. In the study of EEG emotion recognition, the EEG time-domain, frequency-domain and time-frequency-domain features are widely used. Zhang et al. [9] use three methods to select the best channels to classify the four emotions (joy, fear, sadness and relaxation) using SVM classifier, and the best classification accuracy over 19 channels is 59.13% ± 11.00%. Mert and Akan explore the advanced features of EMD and its multivariable extensions (MEMD) in emotion recognition, the multi-channel IMFs extracted by MEMD are analyzed by a variety of time-domain and frequency-domain features such as power ratio, power spectral density, entropy, Hjorth parameters and correlation. The classification accuracies in valence and arousal dimensions are 72.87 ± 4.68% and 75.00 ± 7.48%, respectively [10]. Atkinson and Campos improve the accuracy of emotion classification by combining feature selection method based on the mutual information with the kernel classifier. The accuracy rates of emotion classification using an SVM classifier in the valence and arousal dimensions are as follows: two classification (73.14%, 73.06%), three classification (62.33%, 60.70%) and five classification (45.32%, 46.69%) [11]. Li and Lu propose a frequency band search method to select the best frequency band and use linear SVM to classify two emotions (happiness and sadness) in EEG signals. The mean accuracy of the three experiments is 93.5%, and the gamma frequency band is considered suitable for EEG-based emotion classification [12]. Thammasan et al. use DBNs to improve emotion recognition in music listening, the results of their subjects’ dependence show that DBNs can improve the classification performance of fractional dimension (FD), power spectral density (PSD) and discrete wavelet transform (DWT) in the valence dimension, and improve the classification performance of FD and DWT in arousal dimension [13]. Li et al. propose a preprocessing method for encapsulating multi-channel neurophysiological signals into grid-like frames by wavelet and scale transformation. They further devise a hybrid deep learning model that combines “Convolutional Neural Networks (CNN)” and “Recursive Neural Networks (RNNs)” to extract task-related features, the classification results of the two types of emotions on the valence and the arousal dimensions are 74.12% and 72.06% [14]. Cheng and Liu use the discrete wavelet transform to decompose EMG signals and extract the maximum and minimum wavelet coefficients as the features. Then, an artificial neural network is used to classify the five emotions. The correct classification rate is 82.29% [15]. Kyamakya et al. propose an EEG-based emotion recognition method to detect the patient’s emotional states for Ambient Assisted Living (AAL) and decompose EEG signals into five frequency bands using discrete wavelet transform techniques and extract the wavelet energy, modified energy, wavelet entropy and statistical features of each frequency band as the inputs of the classifier. The average classification accuracy of the four types of emotions using the SVM classifier is 83.87% [16].

Although excellent temporal resolution and acceptable spatial resolution make EEG a suitable choice for emotion recognition systems, a few EEG channels have poor spatial resolution which has a significant impact on the accuracy of emotion recognition. To overcome this problem, multi-channel EEG signals are often used. In channel selection, many studies choose channels empirically. Mert and Akan select EEG signals from 18 channels (eight pairs of left and right symmetric channels and two central channels in the frontal lobe) for emotion recognition in the valence and arousal dimensions [10]. Nattapong et al. [13] select the 12-channel EEG signals near the frontal lobe for identifying emotion states. These channels are thought to play a major role in emotion recognition [17]. The EEG channels that are selected in these papers mainly correspond to specific scalp positions with specific functions. Some studies suggest exploring the correspondence between emotional states and the entire brain [18]. However, the traditional research methods usually only explore EEG emotions in either specific scalp positions or the whole brain area. The influence of the number of EEG channels on EEG emotion recognition has not been systematically studied.

Based on the DEAP data set, this paper discussed the influence of the 10-channel, 14-channel and 18-channel EEG signals based on experience selection and the 32-channel EEG signals of the whole brain on emotion recognition accuracy. The EEG signals were divided into multiple time windows. Each time window was transformed into several frequency bands by discrete wavelet transform (DWT). We extracted entropy and energy as features from each frequency band and the KNN classifier was used to identify the emotional states. In addition, the classification effects using different frequency bands were discussed.

The goal of this paper is to explore the influence of the emotion recognition accuracy of EEG signals in different frequency bands (gamma, beta, alpha and theta) and different number of channels (10-channel, 14-channel, 18-channel and 32-channel), to provide better frequency bands and channels reference for emotion recognition based on EEG.

3.Preliminaries

In this section, we systematically describe the model of emotion, the classification method and the selection of time windows.

4.Materials and methods

4.1Data acquisition

Figure 1.

Emotional state: a Valence, b Arousal, c Dominance and d Liking [22, 26].

In this paper, we used data from the DEAP dataset that have been post-processed in MATLAB. In this dataset, 32 subjects viewed 40 clips of video that could stimulate various emotions. The length of each video is 60 s. Each subject provided a personal rating in the valence-arousal-dominance-liking four dimensions, ranging from 1 to 9, 1 is the smallest, and 9 is the largest. Emotional states in the four dimensions are shown in Fig. 1. emotional state changes with the increase of personal ratings from left to right, for example, Fig. 1a shows the change of valence (degree of pleasure) from small to large (from negative to positive), and Fig. 1b shows the change of arousal (degree of activation) from small to large (from calm to excited). This dataset recorded seven physiological modalities of the participants, including 32-channel EEG signals and other peripheral physiological signals of 8 channels. During the pre-processing phase, a preparation time of 3 s was added to each video, thereby changing the total time of each video to 63 s. EOG interference was removed from the original signal and a bandpass filter was also applied. The sampling frequency of the original signal was reduced from 512 Hz to 128 Hz. In practice, we removed the first 3 seconds of all data and analyzed the remaining 60 s of data. This paper analyzed the emotion in the valence and arousal dimensions. If an individual’s score is greater than 4.5, the level of arousal/valence is classified as high, whereas if the individual’s score is less than 4.5, the level of arousal/valence is classified as low [22, 23].

5.Results and discussion

5.1Different channel combinations of full frequency bands

We compared the emotion recognition accuracies of 10 channels, 14 channels,18 channels and 32 channels of EEG signals in the valence and arousal dimensions. As shown in Table 2, we find that the valence and arousal dimensions of all channel combinations have similar classification accuracies. In both emotion dimensions, the classification accuracy of the emotional state increases with the increase of the number of channels. The highest classification accuracies of 87.03% (valence) and 87.90% (arousal) were obtained when using 32-channel EEG signals. Paired sample T-tests for different channel combinations are shown in Table 3, and all differences are significant (p< 0.05 or p< 0.001), regardless of valence or arousal.

Table 2

All frequency bands classification of each channel combination

Emotion dimensions	No. channels
	10	14	18	32
Valence	82.48% ± 0.53%	84.53% ± 0.66%	85.74% ± 0.53%	87.03% ± 0.88%
Arousal	83.27% ± 0.80%	85.26% ± 0.56%	86.46% ± 0.41%	87.90% ± 0.62%

Table 3

Difference verification between different channels in full band (1 = valence, 2 = arousal)

	10	14	18	32
10		1**	1**	1**
14	2**		1*	1**
18	2**	2*		1**
32	2**	2**	2**

**, p< 0.001; *, p< 0.05.

Figure 2.

Accuracies of different EEG frequency bands and different channel combinations in the (a) valence and (b) arousal dimensions.

Table 4

Accuracy of different EEG frequency bands and different channel combinations

Emotion dimensions	Frequency bands	No. channels
		10	14	18	32
Valence	Gamma	89.54% ± 0.81%	92.28% ± 0.62%	93.72% ± 0.48%	95.70% ± 0.62%
	Beta	87.64% ± 0.49%	90.21% ± 0.30%	92.23% ± 0.41%	94.44% ± 0.38%
	Alpha	73.91% ± 0.92%	77.14% ± 0.70%	79.08% ± 0.73%	81.99% ± 0.61%
	Theta	69.67% ± 0.64%	71.93% ± 0.62%	74.01% ± 0.72%	76.81% ± 0.65%
Arousal	Gamma	89.81% ± 0.46%	92.24% ± 0.33%	93.69% ± 0.30%	95.69% ± 0.21%
	Beta	88.17% ± 0.60%	90.67% ± 0.30%	92.59% ± 0.31%	94.98% ± 0.34%
	Alpha	75.31% ± 0.42%	78.19% ± 0.46%	80.41% ± 0.42%	83.47% ± 0.61%
	Theta	71.52% ± 0.81%	74.16% ± 0.82%	76.05% ± 0.80%	78.92% ± 0.67%

Table 5

Difference verification between different channel combinations of each frequency band (1 = valence, 2 = arousal)

Gamma	10	14	18	32
10		1**	1**	1**
14	2**		1*	1**
18	2**	2*		1**
32	2**	2**	2**

Beta	10	14	18	32
10		1**	1**	1**
14	2**		1*	1**
18	2**	2*		1**
32	2**	2**	2**

Alpha	10	14	18	32
10		1**	1**	1**
14	2**		1*	1**
18	2**	2*		1**
32	2**	2**	2**

Theta	10	14	18	32
10		1**	1**	1**
14	2**		1*	1**
18	2**	2*		1**
32	2**	2**	2**

**, p< 0.001; *, p< 0.05.

Table 6

Difference verification between different frequency bands of each channel combination (1 = valence, 2 = arousal)

10	Gamma	Beta	Alpha	Theta
Gamma		1*	1**	1**
Beta	2**		1**	1**
Alpha	2**	2**		1**
Theta	2**	2**	2**

14	Gamma	Beta	Alpha	Theta
Gamma		1**	1**	1**
Beta	2**		1**	1**
Alpha	2**	2**		1**
Theta	2**	2**	2**

18	Gamma	Beta	Alpha	Theta
Gamma		1**	1**	1**
Beta	2**		1**	1**
Alpha	2**	2**		1**
Theta	2**	2**	2**

32	Gamma	Beta	Alpha	Theta
Gamma		1*	1**	1**
Beta	2**		1**	1**
Alpha	2**	2**		1**
Theta	2**	2**	2**

**, p< 0.001; *, p< 0.05.

6.Conclusions

In the emotional classification of EEG, the methods of EEG data preprocessing, the features of EEG, the feature selection method used (if present), the number and location of channels, the classification of EEG data and the choice of classifier affect the correct rate of EEG emotional classification, It is very difficult for us to simply evaluate the effect of a certain factor on the accuracy of EEG classification in different papers unless the treatment of EEG data is exactly the same among other factors. In general, each paper contains a comparison of the effect of one or more of the above factors on the accuracy of EEG classification. We can discuss the impact of a single factors or a combination of multiple factors on the correct rate of affective classification in a paper, for example: reference [25] aims to study various feature extraction techniques for EEG signals, extract bandwidth power and power spectral density (PSD) features of four frequency bands as features, which are trained and tested on SVM with 40-fold cross-validation (for video selection) and 32-fold cross-validation (for person selection), respectively. The results show that the better classification accuracy is obtained by using the bandwidth power of the four frequency bands and 32-fold cross-validation. Different methods of using dataset have an impact on the classification accuracy. The reference [22] compares the window differences between 2 s and 4 s of EEG, the differences of different frequency bands, and the differences between SVM and KNN classifiers, the results show that 4 s window, gamma band, KNN classifier has a good effect on EEG emotion classification. The reference [11] evaluates the accuracy rate of affective classification of EEG signals using different feature selection methods (minimum redundancy maximum relevance mRMR and genetic algorithm (GA) under different SVM parameters, the results show that mRMR results in a higher classification accuracy with fewer features, and SVM has better classification results using RBF kernels and γ= 0.05.

This paper explores the influence of 10-channel, 14-channel, 18-channel and 32-channel EEG signals on classification accuracy. A normalization method was used to preprocess data from the DEAP dataset. Data from windows of 4 s are divided into four frequency bands (gamma, beta, alpha and theta) using db4 DWT 4-time decomposition. Then, the entropy and energy of each band are calculated as the input features of a KNN classifier. The results show that the classification accuracy of the gamma frequency band is the highest, regardless of the valence or arousal dimension. Based on the gamma frequency band, the classification accuracies of valence of 10, 14, 18, and 32 channels reached 89.54%, 92.28%, 93.72% and 95.70%, respectively, and the classification accuracies of arousal were 89.81%, 92.24%, 93.69% and 95.69%. This shows that the gamma frequency band is more relevant to the emotional state in the valence and arousal dimensions compared to the low frequency band. In addition, it also shows that increasing the number of EEG channels can improve the classification accuracy of emotional states. Our research provides a reference for the selection of EEG channels for emotion recognition.