Dynamic Shape and Appearance Modeling Via Moving and Deforming Layers

We propose a model of the shape, motion and appearance of a sequence of images that captures occlusions, scene deformations, arbitrary viewpoint variations and changes in irradiance. This model is based on a collection of overlapping layers that can move and deform, each supporting an intensity function that can change over time. We discuss the generality and limitations of this model in relations to existing ones such as traditional optical flow or motion segmentation, layers, deformable templates and deformotion. We then illustrate how this model can be used for inference of shape, motion, deformation and appearance of the scene from a collection of images. The layering structure allows for automatic inpainting of partially occluded regions. We illustrate the model on synthetic and real sequences where existing schemes fail; we implement our gradient-based infinite-dimensional optimization using level set methods.