An EEG-EMG-based Hybrid Brain-Computer Interface for Decoding Tones in Silent and Audible Speech

Speech recognition can be widely applied to support people with language disabilities by enabling them to communicate through brain-computer interfaces (BCIs), thus improving their quality of life. Despite the essential role of tonal variations in conveying semantic meaning, there have been limited studies focusing on the neural signatures of tones and their decoding. This paper systematically investigates the neural signatures of the four tones of Mandarin. It explores the feasibility of tone decoding in both silent and audible speech using a multimodal BCI based on electroencephalography (EEG) and electromyography (EMG). The time-frequency analysis of EEG has revealed significant variations in neural activation patterns across various tones and speech modes. For example, in the silent speech condition, temporal-domain analysis shows significant tone-dependent activation in the frontal lobe (ANOVA p = 0.000, Tone1 vs Tone2: p = 0.000, Tone1 vs Tone4: p = 0.000, Tone2 vs Tone3: p = 0.000, Tone3 vs Tone4: p = 0.001) and in channel F8 (ANOVA p=0.008, Tone1 vs Tone2: p=0.014, Tone2 vs Tone3: p=0.034). Spectral analysis shows significant differences between four tones in event-related spectral perturbation (ERSP) in the central region (p = 0.000) and channel C6 (p = 0.000). EMG analysis identifies a significant tone-related difference in activation of the left buccinator muscle (p = 0.023), and ERSP from the mentalis muscle also shows a marked difference across tones in both speech conditions (p = 0.00). Overall, tone-related neural differences were more pronounced in the audible speech condition than in the silent condition. For tone classification, RLDA and SVM classifiers achieved accuracies of 71.22% and 72.43%, respectively, using EEG temporal features in both speech modes. Additionally, the RLDA classifier with temporal features achieves binary tone classification accuracies of 90.92% (audible tones) and 91.00% (silent tones). The combination of EEG and EMG yields the highest speech modes decoding accuracy of 81.33%. These findings provide a potential strategy for speech restoration in tonal languages and further validate the feasibility of a speech brain-computer interface (BCI) as a clinically effective treatment for individuals with tonal language impairment.