Spoken language and multimodal applications for electronic realities

We use the term ‘electronic reality’ (ER) to encompass a broad class of concepts that mix real-world metaphors and computer interfaces. In our definition, ‘electronic reality’ includes notions such as virtual reality, augmented reality, computer interactions with physical devices, interfaces that enhance 2D media such as paper or maps, and social interfaces where computer avatars engage humans in various forms of dialogue. One reason for bringing real-world metaphors to computer interfaces is that people already know how to navigate and interact with the world around them. Every day, people interact with each other, with pets, and sometimes with physical objects by using a combination of expressivemodalities, such as spoken words, lone of voice, pointing and gesturing, facial expressions, and body language. In contrast, when people typically interact with computers or appliances, interactions are unimodal, with a single method of communication such as the click of a mouse or a set of keystrokes serving to express intent. In this article, we describe our efforts to apply multimodal and spoken language interfaces to a number of ER applications, with the goal of creating an even more ‘realistic’ or natural experience for the end user.