Simultaneous Inpainting and Super-resolution Using Self-learning

In applications like creating immersive walkthrough systems or digital reconstruction of invaluable artwork, both inpainting and super-resolution of the given images are the preliminary steps in order to provide better visual experience. The usual practice is to solve these problems independently in a pipelined manner. In this paper we propose a unified framework to perform simultaneous inpainting and super-resolution (SR). The main focus of this paper is inpainting, i.e. to remove objects in photographs and replace them with visually plausible backgrounds. The super-resolved version is obtained as a by-product in the process of using an additional constraint that helps in finding a better source for inpainting. The proposed approach starts with a given test image I0 having a region Ω0 to be inpainted. We obtain the coarser resolution image I−1 by blurring and downsampling I0 as done in [2]. We then construct dictionaries of image-representative low and high resolution (LR-HR) patch pairs from the known regions in the test image I0 and its coarser resolution I−1. The inpainting of the missing pixels is performed using exemplars found by comparing patch details at a finer resolution. Here, self-learning [4] is used to obtain the finer resolution patches by making use of the constructed dictionaries. The obtained finer resolution patches represent the super-resolved patches in the missing regions. Advantage of our approach when compared to other exemplar based inpainting techniques are (1) additional constraint in the form of finer resolution matching results in better inpainting and (2) inpainting is obtained not only in the given spatial resolution but also at higher resolution leading to super-resolution inpainting. The proposed approach is summarized in table 1 and one result is shown in figure 2.