English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We address the problem of estimating depth with multi modal audio visualdata. Inspired by the ability of animals, such as bats and dolphins, to inferdistance of objects with echolocation, some recent methods have utilized echoesfor depth estimation. We propose an end-to-end deep learning based pipelineutilizing RGB images, binaural echoes and estimated material properties ofvarious objects within a scene. We argue that the relation between image,echoes and depth, for different scene elements, is greatly influenced by theproperties of those elements, and a method designed to leverage thisinformation can lead to significantly improved depth estimation from audiovisual inputs. We propose a novel multi modal fusion technique, whichincorporates the material properties explicitly while combining audio (echoes)and visual modalities to predict the scene depth. We show empirically, withexperiments on Replica dataset, that the proposed method obtains 28%improvement in RMSE compared to the state-of-the-art audio-visual depthprediction method. To demonstrate the effectiveness of our method on largerdataset, we report competitive performance on Matterport3D, proposing to use itas a multimodal depth prediction benchmark with echoes for the first time. Wealso analyse the proposed method with exhaustive ablation experiments andqualitative results. The code and models are available athttps://krantiparida.github.io/projects/bimgdepth.html

Beyond Image to Depth: Improving Depth Prediction using Echoes