English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent studies have proposed the use of Text-To-Speech (TTS) systems toautomatically synthesise speech test cases on a scale and uncover a largenumber of failures in ASR systems. However, the failures uncovered by synthetictest cases may not reflect the actual performance of an ASR system when ittranscribes human audio, which we refer to as false alarms. Given a failed testcase synthesised from TTS systems, which consists of TTS-generated audio andthe corresponding ground truth text, we feed the human audio stating the sametext to an ASR system. If human audio can be correctly transcribed, an instanceof a false alarm is detected. In this study, we investigate false alarmoccurrences in five popular ASR systems using synthetic audio generated fromfour TTS systems and human audio obtained from two commonly used datasets. Ourresults show that the least number of false alarms is identified when testingDeepspeech, and the number of false alarms is the highest when testingWav2vec2. On average, false alarm rates range from 21% to 34% in all five ASRsystems. Among the TTS systems used, Google TTS produces the least number offalse alarms (17%), and Espeak TTS produces the highest number of false alarms(32%) among the four TTS systems. Additionally, we build a false alarmestimator that flags potential false alarms, which achieves promising results:a precision of 98.3%, a recall of 96.4%, an accuracy of 98.5%, and an F1 scoreof 97.3%. Our study provides insight into the appropriate selection of TTSsystems to generate high-quality speech to test ASR systems. Additionally, afalse alarm estimator can be a way to minimise the impact of false alarms andhelp developers choose suitable test inputs when evaluating ASR systems. Thesource code used in this paper is publicly available on GitHub athttps://github.com/julianyonghao/FAinASRtest.

Synthesizing Speech Test Cases with Text-to-Speech? An Empirical Study on the False Alarms in Automated Speech Recognition Testing