The Enlightenment of the Theory Model of Audio-Visual Integration to Human-Computer Interaction Technology

The visual and auditory systems are the most important information processing systems in the brain. We usually sense external stimuli in the way of audio-visual integration. In this process, “visual” and “auditory” nervous form two functional systems which are relatively independent and interactive. It is of great significance for the fusion of multi-sensor data and human-computer interaction technology to understand the theoretical principles of integrating visual and auditory cues and the neural mechanism of audio-visual integration. Here, we try to explore new methods of audio-visual integration and develop new potential of human-computer interface. Previous studies have shown that the principles of integrating visual and auditory cues mainly include spatio-temporal rule, inverse-effectiveness rule and reliability rule. On this basis, researchers have gradually developed a theoretical model of audio-visual integration—causal inference model. By clarifying the mechanism of audio-visual integration, it provides a theoretical basis for the multimodal interaction technology of human-computer interaction, so as to improve the naturalness and efficiency of human-computer interaction and overcome the shortcomings of traditional human-computer interaction.