Abstract: Nowadays, with the remarkable advancements in detection instruments and artificial intelligence, there has been extensive utilization of human mental state monitoring in various domains. Few studies have explored how nonlinear analysis methods can detect cognitive workload despite the complex nature of EEG signals and advancements in signal processing techniques. In addition, the fuzziness of human mental conditions makes the need to use fuzzy engineering tools tangible in this field. Therefore, this investigation aimed to develop a decision support algorithm to improve previous efforts for the classification of task EEG and resting through machine learning algorithms. Various nonlinear features were calculated from all 19 EEG channels: Hurst exponent, Lempel–Ziv complexity, detrended fluctuation analysis, Higuchi fractal dimension, Katz fractal dimension, permutation entropy, singular value decomposition entropy, Petrosian fractal dimension, sample entropy, and Lyapunov exponent. During the classification step, a newly developed EPC-FC (Expert per Class Fuzzy Classifier) is introduced, utilizing an ensemble framework with specialized sub-classifiers for identifying a particular condition. By training sub-classifiers with the negative correlation learning (NCL) approach, the EPC-FC is designed to be exceptionally adaptable. Additionally, the separation of sub-classifiers within each class provides versatility and clarity to the system’s design. The proposed approach based on fuzzy systems and nonlinear analyses was applied to EEG data for mental workload recognition, which provides an excellent accuracy of 98.50% and an F1-score of 98.56% which is much higher than previous findings in this field. Also, the obtained results indicate that utilizing the proposed EPC-FC classifier maintains a consistently high accuracy exceeding 90% across various levels of SNRs. The obtained results proved the high potential of nonlinear analysis to detect cognitive states of the brain, which is consistent with the nonlinear and fuzzy nature of EEG data. Other nonlinear approaches should be considered for future studies to improve the current results.
