Using embodied conversational agents in video games to investigate emotional facial expressions

Michael Lankes, Regina Bernhaupt

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)


The perception of emotional facial expressions has been diligently studied in psychology for the last 40. years. The stimuli mainly used in these studies were photos or short film clips. Today, video games and their development frameworks allow researchers to design, model, and animate embodied conversational agents (ECAs). This research paper introduces an experimental setting to observe and measure the perception of facial expression performed by embodied conversational agents. The experimental setup is based on the research efforts of the psychologist Harald G. Wallbott and uses the CryENGINE to visualize the ECAs in various contextual settings. A new paradigm, coined " Interaction-Paradigm" , is integrated in the experimental setting, which is derived from the video-games domain. It utilizes the benefits of video games as a research tool: the setup grants participants an interactive experience of a given emotional situation, which allows the investigation of the perception process in a more realistic setting. In contrast to the traditional approaches in emotion theory research, the usage of video games ensures that situational aspects are not presented by simply showing subjects a piece of paper containing descriptions of a situation.Based on an extensive state of the art on using video games and games development frameworks in research, this article shows how to set up experiments using video game technology. The aim of the study is to establish a comparative experimental framework to analyze subjects' judgment on emotional stimuli in different context dimensions. The findings show that the importance of information channels changes, when people decide for the emotional quality of a situation. While in still images the majority of people bases the decision on the facial expression, rather than based on the text description, in interactive scenarios only 11.8% of the participants relied on the facial expressions, with the majority using the contextual description. We present the results of the experiment in detail and summarize how video games and games development frameworks can support research in the area of ECAs and emotion perception.

Original languageEnglish
Pages (from-to)29-37
Number of pages9
JournalEntertainment Computing
Issue number1
Publication statusPublished - 2011


  • Embodied conversational agents
  • Emotion theory
  • Games
  • Research method


Dive into the research topics of 'Using embodied conversational agents in video games to investigate emotional facial expressions'. Together they form a unique fingerprint.

Cite this