Experiences in Using a Smartphone as a Virtual Reality Interaction Device

Input devices such as Nintendo Wiimotes are often used to select and manipulate virtual objects. While simple to use and easily available, these devices have some limitations. When used with common large displays such as televisions, they support only indirect manipulation. These devices also require the user to learn and remember which buttons map to which functions. We propose overcoming these limitations by using a smartphone as an interaction device. Smartphones, like Wiimotes, are readily available and easy to operate. Unlike the Wiimote, the smartphone has a touchscreen that can display the selected object, allowing the user to directly manipulate the object. Further, the touchscreen can customize the interface and provide buttons with clearly labeled functions specific to the object. We report on the lessons learned in integrating and using a smartphone as the interaction device for two applications. The first is a mixed reality game focused on general object selection and pose manipulation. We used this game in a pilot study evaluating usability. The second is an adaptation of an existing virtual reality application. This application demonstrated the ease of adaptation as well as improvements from using a smartphone.