English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Automatic speech recognition (ASR) systems have become popular in different applications (e.g., in virtual assistants). In order to ensure its robustness, ASR systems should be tested with diverse speech test data (i.e., with different sentences and speakers) in order to simulate different real scenarios of use. Relying on human speakers to test ASR systems is time-consuming and expensive. An alternative is to use text-to-speech (TTS) tools to synthesize audios from a set of sentences given as input. The ASR under test then receives the synthesized audios as test data, and the transcription errors are recorded for evaluation. Despite the availability of TTS tools, not all synthesized speeches have the same quality for all speakers and sentences. So before testing the ASR, it is important to evaluate the usefulness of the speakers as well as to determine which sentences are more relevant for ASR evaluation. In this paper, we propose the use of Item Response Theory (IRT) to evaluate speech synthesizers. IRT is a paradigm developed in psychometrics to estimate the cognitive ability of human respondents based on their responses to items with different levels of difficulty. In our context, an item is a sentence to be synthesized and a respondent is a speaker. In turn, each response is the transcription accuracy observed when a given sentence and speaker are adopted for testing the ASR. An IRT model identifies latent patterns of responses to estimate the difficulty of each sentence. In turn, the performance of each speaker is estimated by taking into account its responses to difficult sentences. In order to verify the viability of the proposal, a case study was developed, in which an ASR was used to transcribe voice commands (e.g., “set to vibrate mode”) for a mobile virtual assistant. IRT was applied to evaluate 62 speakers (from four TTS tools) and to characterize the difficulty of 12 different sentences. We present interesting insights about the relevance of the synthesized audios for the ASR under test by inspecting the estimated parameters of difficulties and abilities. The knowledge acquired from such analysis can be applied to optimize the task of ASR testing.

Item Response Theory to Estimate the Latent Ability of Speech Synthesizers