z-logo
Premium
Interactive Photo Editing on Smartphones via Intrinsic Decomposition
Author(s) -
Shekhar Sumit,
Reimann Max,
Mayer Maximilian,
Semmo Amir,
Pasewaldt Sebastian,
Döllner Jürgen,
Trapp Matthias
Publication year - 2021
Publication title -
computer graphics forum
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.578
H-Index - 120
eISSN - 1467-8659
pISSN - 0167-7055
DOI - 10.1111/cgf.142650
Subject(s) - specularity , computer science , computer vision , artificial intelligence , inpainting , specular reflection , pixel , decomposition , shading , rgb color model , computer graphics (images) , image (mathematics) , ecology , biology , physics , quantum mechanics
Intrinsic decomposition refers to the problem of estimating scene characteristics, such as albedo and shading, when one view or multiple views of a scene are provided. The inverse problem setting, where multiple unknowns are solved given a single known pixel‐value, is highly under‐constrained. When provided with correlating image and depth data, intrinsic scene decomposition can be facilitated using depth‐based priors, which nowadays is easy to acquire with high‐end smartphones by utilizing their depth sensors. In this work, we present a system for intrinsic decomposition of RGB‐D images on smartphones and the algorithmic as well as design choices therein. Unlike state‐of‐the‐art methods that assume only diffuse reflectance, we consider both diffuse and specular pixels. For this purpose, we present a novel specularity extraction algorithm based on a multi‐scale intensity decomposition and chroma inpainting. At this, the diffuse component is further decomposed into albedo and shading components. We use an inertial proximal algorithm for non‐convex optimization (iPiano) to ensure albedo sparsity. Our GPU‐based visual processing is implemented on iOS via the Metal API and enables interactive performance on an iPhone 11 Pro. Further, a qualitative evaluation shows that we are able to obtain high‐quality outputs. Furthermore, our proposed approach for specularity removal outperforms state‐of‐the‐art approaches for real‐world images, while our albedo and shading layer decomposition is faster than the prior work at a comparable output quality. Manifold applications such as recoloring, retexturing, relighting, appearance editing, and stylization are shown, each using the intrinsic layers obtained with our method and/or the corresponding depth data.

This content is not available in your region!

Continue researching here.

Having issues? You can contact us here