A statistical simulation technique to develop and evaluate conversational agents

Article type: Research Article

Authors: Griol, David^; | Carbo, Javier | Molina, José M.

Affiliations: Applied Artificial Intelligence Group (GIAA), Department of Computer Science, Carlos III University of Madrid, Avda. de la Universidad, 30, 28911, Leganés, Spain. E-mails: {david.griol, javier.carbo, josemanuel.molina}@uc3m.es

Note: [] Corresponding author. E-mail: [email protected]

Abstract: In this paper, we present a technique for developing user simulators which are able to interact and evaluate conversational agents. Our technique is based on a statistical model that is automatically learned from a dialog corpus. This model is used by the user simulator to provide the next answer taking into account the complete history of the interaction. The main objective of our proposal is not only to evaluate the conversational agent, but also to improve this agent by employing the simulated dialogs to learn a better dialog model. We have applied this technique to design and evaluate a conversational agent which provides academic information in a multi-agent system. The results of the evaluation show that the proposed user simulation methodology can be used not only to evaluate conversational agents but also to explore new enhanced dialog strategies, thereby allowing the conversational agent to reduce the time needed to complete the dialogs and automatically detect new valid paths to achieve each of the required objectives defined for the task.

Keywords: Conversational agents, user simulation, statistical methodologies, multiagent systems

DOI: 10.3233/AIC-130573

Journal: AI Communications, vol. 26, no. 4, pp. 355-371, 2013