Integration of shape information from vision and touch: Optimal perception and neural correlates

Recently, Ernst and Banks (2002) showed that visual-haptic size information is integrated in a statistically optimal manner, i.e. visual and haptic size estimates are weighted according to their reliabilities. Here we investigate whether the same is true for visual-haptic shape information. We further explored the neural substrates underlying visual-haptic integration in shape processing using fMRI and examined whether neural activity elicited by multisensory integration correlates with cue weighting. For this we used ridges of elliptical objects. Subjects saw the front of the object and/or they felt the back. The elongation of the elliptical ridges on both sides of the objects could differ. Subjects’ task was to decide whether the ellipse was elongated vertically or horizontally. This way we could study the weight of vision and touch during shape discrimination. We varied the weight given to vision by degrading the visual information, using blur. The psychophysical experiments showed that visual and haptic shape information is integrated in a statistical optimal way even when the visual information is displayed via a mirror. That is, we observed a decrease in visual weight when vision was degraded and thus less reliable. Furthermore, we found an increase in discrimination performance when both modalities were presented together. We also determined neural activity with fMRI while individuals were performing the same ellipse discrimination task. When the visual reliability is reduced in the visual-haptic task, neural responses decreased in the lateral occipital cortex while increased in the anterior intraparietal cortex, a brain region strongly involved in multisensory integration