Three Years of VoiceMOS Challenges: Lessons Learned by the UWB-NTIS-TTS Team

Automatic prediction of mean-opinion scores (MOS) promises a faster, cheaper alternative to listening tests, yet robust generalization across speakers, languages, and domains remains a significant challenge. This article presents our system designs and experimental results from three years of participation in the VoiceMOS Challenges (2022–2024), covering MOS prediction for synthesized or voice-converted speech and singing voice, including out-of-domain and cross-language conditions.We evaluate six neural architectures – wav2vec 2.0, QuartzNet, CNN-RNN, LDNet, RawNet3, and HiFi-GAN – and their ensembles. Across all tasks, we find that (i) self-supervised acoustic encoders are the most consistently reliable foundation, (ii) ensembling yields rapidly diminishing returns once complementary representations are covered, and (iii) the diversity and balance of training data outweigh architectural complexity. Notably, the indiscriminate fusion strategy that performed well in 2022 degrades under the mismatched French TTS conditions of 2023, emphasizing the importance of out-of-domain validation. Further experiments show that carefully pruned ensembles can modestly outperform the best single model while remaining within real-time constraints. We conclude with several observations to guide the development of computationally efficient, domain-robust MOS prediction systems.