Audio-textual multi-label demographic recognition of Arabic speakers using deep learning

Article type: Research Article

Authors: Al-Azani, Sadam^{a; *} | Almeshari, Ridha^{a; b} | El-Alfy, El-Sayed^{a; b; c}

Affiliations: [a] SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, Saudi Arabia | [b] Information and Computer Science Department, King Fahd University of Petroleum & Minerals, Dhahran, Saudi Arabia | [c] Computer Engineering Department and Interdisciplinary Research Center of Intelligent Secure Systems (IRC-ISS), King Fahd University of Petroleum and Minerals, Saudi Arabia

Correspondence: [*] Corresponding author Sadam Al-Azani, SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran 31261, Saudi Arabia. Email: [email protected].

Abstract: Speaker demographic recognition and segmentation analytics play a key role in offering personalized experiences across different automated industries and businesses. This paper aims at developing a multi-label demographic recognition system for Arabic speakers from audio and associated textual modalities. The system can detect age groups, genders, and dialects, but it can be easily extended to incorporate more demographic traits. The proposed method is based on deep learning for feature learning and recognition. Representations of audio modality are learned through 3D spectrogram and AlexNet CNN-based architecture. An AraBERT transformer is employed for learning representations of the textual modality. Additionally, a method is provided for fusing audio and textual representations. The effectiveness of the proposed method is evaluated using the Saudi Audio Dataset for Arabic (SADA), which is a recently published database containing audio recordings of TV shows in different Arabic dialects. The experimental findings show that when using models with standalone modalities for multi-label demographic classification, textual modality using AraBERT performed better than the audio modality represented using 3D spectrogram along with AlexNet CNN-based architecture. Furthermore, when combining both modalities, audio and textual, significant improvement has been attained for all demographic traits.

Keywords: Demographic, 3D spectrogram, AraBERT, multi-label classification, Arabic LLMs, multimodal deep learning

DOI: 10.3233/JIFS-219389

Journal: Journal of Intelligent & Fuzzy Systems, vol. Pre-press, no. Pre-press, pp. 1-12, 2024