English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Cued Speech (CS) is an augmented lip reading complemented by hand coding, andit is very helpful to the deaf people. Automatic CS recognition can helpcommunications between the deaf people and others. Due to the asynchronousnature of lips and hand movements, fusion of them in automatic CS recognitionis a challenging problem. In this work, we propose a novel re-synchronizationprocedure for multi-modal fusion, which aligns the hand features with lipsfeature. It is realized by delaying hand position and hand shape with theiroptimal hand preceding time which is derived by investigating the temporalorganizations of hand position and hand shape movements in CS. Thisre-synchronization procedure is incorporated into a practical continuous CSrecognition system that combines convolutional neural network (CNN) withmulti-stream hidden markov model (MSHMM). A significant improvement of about4.6\% has been achieved retaining 76.6\% CS phoneme recognition correctnesscompared with the state-of-the-art architecture (72.04\%), which did not takeinto account the asynchrony of multi-modal fusion in CS. To our knowledge, thisis the first work to tackle the asynchronous multi-modal fusion in theautomatic continuous CS recognition.

Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognition