An integrated approach to motion and sound

Until recently, sound has been given little attention in computer graphics and related domains of computer animation and virtual environments, although sounds which are properly synchronized to motion provide a great deal of information about events in the environment. Sounds are often not properly synchronized because the sounds and the phenomena that caused the sounds are not considered in an integrated way. In this paper, we present an integrated approach to motion and sound as it applies to computer animation and virtual environments. The key to this approach is synchronization by mapping the motion parameters to sound parameters so that the sound changes as a result of changes in the motion. This is done by representing sounds using a technique for functional composition analogous to the ‘shade trees’ which we call timbre trees. These timbre trees are used as a part of a sound description language that is analogous to scene description languages such as RenderMan. Using this methodology, we have produced convincing sound effects for a wide variety of animated scenes including the automatic generation of background music.