Visual recognition of speech consonants using facial movement features

Issue title: Informatics in Control, Automation and Robotics

Guest editors: Joaquim Filipe

Article type: Research Article

Authors: Yau, Wai Chee | Kumar, Dinesh Kant^{; *} | Arjunan, Sridhar Poosapadi

Affiliations: School of Electrical and Computer Engineering, RMIT University, GPO Box 2476V Melbourne, Victoria 3001, Australia

Correspondence: [*] Corresponding author. Tel.: +61399251954; E-mail: [email protected].

Abstract: This paper presents a visual speech recognition technique using facial movement video. The acoustic signals of consonants are often confusing in noisy environments. To overcome this shortcoming, the focus of this paper is identifying consonants using visual information. This paper investigates the feasibility of using facial movements to identify phonemes. The proposed approach adopts a visual speech model based on the viseme model of the Moving Picture Experts Group 4 (MPEG-4) standard. It is a movement-based system, and the facial movements are segmented from the video using an accumulative image subtraction method that results in a 2-D grayscale motion history image (MHI). The MHI is classified using a combination of the discrete stationary wavelet transform (SWT) and image moments (Hu moments, geometric moments and Zernike moments). Feedforward multilayer perceptron (MLP) neural networks with backpropagation (BPN) learning algorithm are used to classify the features to investigate the performance of the three moment features. The experimental results indicate that Zernike moments have better representation ability and provide rotational invariant property for the proposed application. The results also demonstrate that the proposed technique can identify consonants reliably using the viseme model of MPEG-4 standard with a recognition rate of 85%.

DOI: 10.3233/ICA-2007-14105

Journal: Integrated Computer-Aided Engineering, vol. 14, no. 1, pp. 49-61, 2007