Open AccessVisual Enhancement of Relevant Speech in a ‘Cocktail Party’
Author(s) -
Niti Jaha,
Stanley Shen,
Jess R. Kerlin,
Antoine J. Shahin
Publication year - 2020
Publication title -
multisensory research
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.521
H-Index - 50
eISSN - 2213-4808
pISSN - 2213-4794
DOI - 10.1163/22134808-20191423
Subject(s) - psychology , sentence , active listening , comprehension , perception , speech recognition , audiology , speech perception , task (project management) , stimulus (psychology) , communication , intelligibility (philosophy) , cognitive psychology , linguistics , computer science , medicine , philosophy , management , epistemology , neuroscience , economics
Lip-reading improves intelligibility in noisy acoustical environments. We hypothesized that watching mouth movements benefits speech comprehension in a 'cocktail party' by strengthening the encoding of the neural representations of the visually paired speech stream. In an audiovisual (AV) task, EEG was recorded as participants watched and listened to videos of a speaker uttering a sentence while also hearing a concurrent sentence by a speaker of the opposite gender. A key manipulation was that each audio sentence had a 200-ms segment replaced by white noise. To assess comprehension, subjects were tasked with transcribing the AV-attended sentence on randomly selected trials. In the auditory-only trials, subjects listened to the same sentences and completed the same task while watching a static picture of a speaker of either gender. Subjects directed their listening to the voice of the gender of the speaker in the video. We found that the N1 auditory-evoked potential (AEP) time-locked to white noise onsets was significantly more inhibited for the AV-attended sentences than for those of the auditorily-attended (A-attended) and AV-unattended sentences. N1 inhibition to noise onsets has been shown to index restoration of phonemic representations of degraded speech. These results underscore that attention and congruency in the AV setting help streamline the complex auditory scene, partly by reinforcing the neural representations of the visually attended stream, heightening the perception of continuity and comprehension.