Course Description

The main objective of this course is to give an overview on Conversational Virtual Characters, their potential and use, and on the technologies involved in their creation. We will focus on the existing models that have been developed to endow these characters with communicative and emotional capabilities. We will present computational models of multimodal behaviors such as gesture, facial expressions and gaze. We will introduce some representation languages that are used to control ECAs at a high level (communicative functions) and low level (behaviors). ECAs are mainly used in real-time interactive applications. Systems architecture needs to allow for real-time as well as interactivity. The course will emphasize on the communicative qualities an ECA ought to have to be able to dialog with users/other agents. Being in an interaction with users and/or other virtual characters, they will be at turn speaker or listener. The course will present the background information and computational models in these different research areas.

Course Overview

We will first provide a definition of Embodied Conversational Agent ECA where we explicit their potential roles in HCI. In this course we will concentrate on behavior modeling of ECAs. In particular we will offer an overview of the theoretical models of nonverbal communication on which they are based and present taxonomies for facial expressions, gesture and gaze have been elaborated.

We will review how lexicon of nonverbal behaviors is built. We will concentrate on communicative hand gestures and facial expressions. Gestures can be described by the shape of its components (hand, wrist, elbow), the path the wrist follow and their temporal structure. A coding of facial expressions has been proposed to describe any visible action on the face. We will present also how these behaviors are represented in a computational form.

Emotions are often expressed through facial expressions. Studies have looked at the pattern these expressions have, which muscle contractions are involved. Different theories have been introduced that view the expression of emotions either as an expression at its apex, as a continuum of expressions in space or even as a sequence of signals. We will introduce the different computational models of facial expressions of emotions depending on the theoretical model they are based on.

Controlling the animation of a humanoid agent is really tedious and time consuming; especially it requires really good animation skills. Moreover having to specify manually the behaviors of these agents in real-time interactive applications is not feasible. We will introduce some representation languages that are used to control ECAs at a high level (communicative functions) and low level (behaviors).

Background Reading

J. Cassell, J. Sullivan, S. Prevost, E. Churchill (Editors) “Embodied Conversational Characters”, MIT Press, 2000.

Neff, M., Kipp, M., Albrecht, I. and Seidel, H.-P. (2008) Gesture Modeling and Animation Based on a Probabilistic Recreation of Speaker Style. In: ACM Transactions on Graphics 27 (1), ACM Press, pp. 1-24.

Niewiadomski, R., Hyniewska, S., Pelachaud, C., Modeling emotional expressions as sequences of Behaviors, International conference on Intelligent virtual agents IVA'09, Amsterdam, Sept 2009.

H. Prendinger, M. Ishizuka (Editors), “Life-like Characters. Tools, Affective Functions and Applications”, Springer, 2004.

Presenter

Professor Catherine Pelachaud

Presenter Biography

Catherine Pelachaud is Director of Research at CNRS in the laboratory LTCI, TELECOM ParisTech. She received her PhD in Computer Graphics at the University of Pennsylvania, Philadelphia, USA in 1991. She participated to the elaboration of the first embodied conversation agent system, GestureJack, with Justine Cassell, Norman Badler and Mark Steedman when being a post-doctorate at the University of Pennsylvania. Her research interest includes representation language for agent, embodied conversational agent, nonverbal communication (face, gaze, and gesture), expressive behaviors and multimodal interfaces. She has been involved and is still involved in several European projects related to multimodal communication (EAGLES, IST-ISLE), to believable embodied conversational agents (IST-MagiCster, FP5 PF-STAR), emotion (FP5 NoE Humaine, FP6 IP CALLAS, FP7 STREP SEMAINE) and social behaviors (FP7 NoE SSPNet).