English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We present VisualHints, a novel environment for multimodal reinforcementlearning (RL) involving text-based interactions along with visual hints(obtained from the environment). Real-life problems often demand that agentsinteract with the environment using both natural language information andvisual perception towards solving a goal. However, most traditional RLenvironments either solve pure vision-based tasks like Atari games orvideo-based robotic manipulation; or entirely use natural language as a mode ofinteraction, like Text-based games and dialog systems. In this work, we aim tobridge this gap and unify these two approaches in a single environment formultimodal RL. We introduce an extension of the TextWorld cooking environmentwith the addition of visual clues interspersed throughout the environment. Thegoal is to force an RL agent to use both text and visual features to predictnatural language action commands for solving the final task of cooking a meal.We enable variations and difficulties in our environment to emulate variousinteractive real-world scenarios. We present a baseline multimodal agent forsolving such problems using CNN-based feature extraction from visual hints andLSTMs for textual feature extraction. We believe that our proposedvisual-lingual environment will facilitate novel problem settings for the RLcommunity.

VisualHints: A Visual-Lingual Environment for Multimodal Reinforcement Learning