Framework and Assessment of Conversational Virtual Humans as Role-players in Simulated Social Encounters with People

As technology advances, people increasingly converse with computer conversational agents serving in a personal assistant role such as Siri or Cortona. Beyond the personal assistant role, a virtual humans may substitute for a person in another role during a social encounter in a simulated environment. From an experiential or pedantic training perspective, the resulting simulated social encounter may provide a forum by which real people may gain social skills. This article reviews advances in technology and science of virtual humans in faceto-face social encounters with people. An assessment framework is proposed based on a modification of the Schroeder, Adesope, & Gilbert framework [1]. Conversational attributes include verbal content and expression and nonverbal behavior such as head position or inclination, micro and macro facial or body emotional expression, eye gaze and direction, and other facial and other bodily gestures. Using that framework, ten instantiations of virtual humans are analyzed with respect to simulating a person in a bi-directional conversation. Findings and discussion address the question: can a virtual human truly be a social partner with a person for the purpose of social skills training? Within a social skills learning setting, analysis raises concern about the impact of textual data concurrently displayed with verbal message content and expression, and nonverbal behavior. Suggested future advanced technology research includes virtual human eye tracking technology in order to a better understand the cognitive science and behavioral responses associated with periods of engagement and boredom during interactions with a person.