English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Captioning images is a challenging scene-understanding task that connectscomputer vision and natural language processing. While image captioning modelshave been successful in producing excellent descriptions, the field hasprimarily focused on generating a single sentence for 2D images. This paperinvestigates whether integrating depth information with RGB images can enhancethe captioning task and generate better descriptions. For this purpose, wepropose a Transformer-based encoder-decoder framework for generating amulti-sentence description of a 3D scene. The RGB image and its correspondingdepth map are provided as inputs to our framework, which combines them toproduce a better understanding of the input scene. Depth maps could be groundtruth or estimated, which makes our framework widely applicable to any RGBcaptioning dataset. We explored different fusion approaches to fuse RGB anddepth images. The experiments are performed on the NYU-v2 dataset and theStanford image paragraph captioning dataset. During our work with the NYU-v2dataset, we found inconsistent labeling that prevents the benefit of usingdepth information to enhance the captioning task. The results were even worsethan using RGB images only. As a result, we propose a cleaned version of theNYU-v2 dataset that is more consistent and informative. Our results on bothdatasets demonstrate that the proposed framework effectively benefits fromdepth information, whether it is ground truth or estimated, and generatesbetter captions. Code, pre-trained models, and the cleaned version of theNYU-v2 dataset will be made publically available.

Enhancing image captioning with depth information using a Transformer-based framework