SilentWhisper: Enabling User-Independent Ultra-Low Volume Whispered Speech Recognition

Existing Silent Speech Interfaces (SSIs), which enable private voice input, are often hampered by user dependency—a critical barrier requiring extensive training data from each individual. To address this, we propose SilentWhisper, a system that achieves truly user-independent recognition of ultra-low volume (45-50dB) whispered speech using only standard microphones. By fine-tuning a pre-trained speech recognition model on a multi-user dataset, our method allows anyone to use the system immediately without per-user adaptation. We validate our user-independent claim through rigorous leave-one-out cross-validation with 13 participants, which confirmed a significant reduction in the average Character Error Rate (CER) from 17.0% to 7.3% (a 57.1% improvement). Crucially, this specialized fine-tuning is also robust; the model maintains performance comparable to the powerful baseline on unseen texts, thus avoiding catastrophic forgetting. Evaluated on a 1,002-word vocabulary, our system’s strong performance represents a significant step towards a practical, private conversational interface that is accessible to all.