Exploring Un-Intentional Body Gestures for Affective System Design

Imagine the following situation: a student is attempting a tutorial through an intelligent tutoring system (ITS). During the learning session, he starts scratching on his head. What might be the reason for this action? Is he anxious, or having a problem with his hair? Now imagine how effective the intelligent tutoring system (ITS) could be if it could correctly realize students’ mental state and consequently adopt a suitable instructive strategy to address the situation. Human affect-sensitive system such as envisioned here, capable of interpreting its users’ affect and promising applications proposed by Picard (2000) and others (Brave & Nass, 2002) are source of inspiration for growing interest in researching affective systems. However, reliable recognition of affect needs to address uncertainty and context dependency when mapping affect from human behavioral cues. Uncertainty comes from the fact that affective interpretations vary from person to person and likewise being context dependent these interpretations vary with situation quite often. Here, we discuss our approach to address both of these issues. As such no precise and generally agreed definition of affect or emotion exists. Recently, (Minsky, 2006) describes emotional state as not different from the process such as thinking. Human affect may consist of emotional and/or mental state of a person. Beside verbal expression there are non-verbal means of expressing affect by humans. They include visual cues that may inform about the human affective state. Gestures from face, hand and body are part of human body language (Sebe & Lew, 2003), and may communicate affect in various situations. We consider human body gestures for affect interpretation, and use them for designing affective systems. In this chapter, we report an extension of our earlier work (Abbasi et al., 2007) that explores the presence of body gestures that we found as common among a group of students attending a class lecture. Most of these gestures involve hand movements around the face and are unintentional in nature. We map these gestures to affective states reported by students. We propose using this information for designing an intelligent tutoring system or an affective class barometer. To address the uncertainty in subjective interpretations, we propose using a probabilistic approach. These interpretations are dependent on situational context as they occur in a