Open AccessReconstruction Lighting Positions of a Scene for Mixed Reality Systems Using Convolutional Neural Network and Shadow Ray Tracing
Author(s) -
Максим Сорокин,
Dmitry Zhdanov,
Ildar Valiev
Publication year - 2021
Publication title -
preprint/preprinty ipm im. m.v. keldyša
Language(s) - English
Resource type - Journals
eISSN - 2071-2901
pISSN - 2071-2898
DOI - 10.20948/prepr-2021-67
Subject(s) - shadow (psychology) , computer vision , computer science , artificial intelligence , convolutional neural network , ray tracing (physics) , computer graphics (images) , embedding , virtual reality , mixed reality , augmented reality , optics , psychology , physics , psychotherapist
The paper examines the causes of visual discomfort in mixed reality systems and algorithmic solutions that eliminate one of the main causes of discomfort, namely, the mismatch between the lighting conditions of objects in the real and virtual worlds. To eliminate this cause of discomfort, the algorithm is proposed, which consists in constructing groups of shadow rays from points on the boundaries of shadows to points on the boundaries of objects. Part of the rays corresponding to the real lighting conditions form caustics in area of the real light source, which makes it possible to determine the source of illumination of virtual objects for their correct embedding into the mixed reality system. Convolutional neural networks and computer vision algorithms were used to classify shadows in the image. Examples of reconstructing the coordinates of a light source from RGBD data are presented.