English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Lip reading, aiming to recognize spoken sentences according to the givenvideo of lip movements without relying on the audio stream, has attracted greatinterest due to its application in many scenarios. Although prior works thatexplore lip reading have obtained salient achievements, they are all trained ina non-simultaneous manner where the predictions are generated requiring accessto the full video. To breakthrough this constraint, we study the task ofsimultaneous lip reading and devise SimulLR, a simultaneous lip Readingtransducer with attention-guided adaptive memory from three aspects: (1) Toaddress the challenge of monotonic alignments while considering the syntacticstructure of the generated sentences under simultaneous setting, we build atransducer-based model and design several effective training strategiesincluding CTC pre-training, model warm-up and curriculum learning to promotethe training of the lip reading transducer. (2) To learn better spatio-temporalrepresentations for simultaneous encoder, we construct a truncated 3Dconvolution and time-restricted self-attention layer to perform theframe-to-frame interaction within a video segment containing fixed number offrames. (3) The history information is always limited due to the storage inreal-time scenarios, especially for massive video data. Therefore, we devise anovel attention-guided adaptive memory to organize semantic information ofhistory segments and enhance the visual representations with acceptablecomputation-aware latency. The experiments show that the SimulLR achieves thetranslation speedup 9.10$\times$ compared with the state-of-the-artnon-simultaneous methods, and also obtains competitive results, which indicatesthe effectiveness of our proposed methods.

SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory