Subpixel Semantic Flow

Dense semantic correspondence is usually cast as a variational optimization problem. Current methods generally focus on obtaining more discriminative features (to improve the data/correspondence term), and adopt a message-passing algorithm for inference, which is generally a variant of loopy belief propagation. One drawback of such optimization is that the flow vectors are constrained to be discrete variables resulting in pixel resolution, “blocky” flow fields. The main hinderance to formulating the problem in continuous space, and hence solving the problem at subpixel resolution, is the use of histogram based descriptors such as SIFT, HOG, etc. Such sparse feature descriptors are distinctive but linearize poorly. In this paper, we revisit a classic dense descriptor, namely Geometric Blur, which is, in contrast, extracted from a linear filter (spatially varying Gaussian) response that can be linearized and therefore interpolated at subpixel values. In addition to the data and smoothness terms used in variational models, we also add a term promoting bidirectional flow consistency. As there is no longer a finite set of values a flow vector can take, we use gradient descent based continuous optimization. We present promising results encouraging the use of gradient based continuous optimization in establishing dense semantic correspondences. The proposed subpixel approach is applicable to any dense descriptor, including the mid-level convolutional layer outputs of convolutional neural networks (CNNs).