Note: [*] The author is currently a student at University of Ontario Institute of Technology and this work was done while he was an intern at SRI International.
Abstract: We present a new collaborative visual storytelling platform, Aesop, for direction and animation. Our system operates in two main modes, common sense grounding (annotation) and conversation. The Aesop system senses the human state and input using a natural language parser and human gesture monitoring for natural interactions. The interface consists of a 3D animation software and a web controller to interact with the internal state of the system. For knowledge representation, we formulate novel knowledge graphs which enable spatio-temporal event representation. Aesop thus enables 3D spatial and temporal reasoning which are both essential for storytelling. Finally, the system utilizes a dialog manager to track the conversation state and manage goals. Aesop provides a rich platform that enables research in language, gestures, vision, and planning in the context of storytelling.
Keywords: Visual storytelling, common sense grounding, natural language, knowledge representation, dialogue, language and vision, conversational AI, communication
