Flow‐based frame interpolation networks combined with occlusion‐aware mask estimation

Frame interpolation is one of the most challenging tasks in the video processing field. Recent advances have demonstrated that the deep learning‐based frame interpolation methods are promising. However, the experiments show that most existing deep learning‐based methods have the same problem as traditional methods. When these algorithms handle severe occlusions, they will produce distortions, especially around the motion boundaries. To better synthesise the image of the motion areas, the authors design a mask‐guided frame synthesis model, which consists of multiple components, based on deep convolutional neural networks. The proposed model first estimates the asymmetric bi‐directional optical flows from the intermediate frame to the input frames. Then it estimates the occlusion‐aware masks, which can compensate for the optical flow inaccuracy based on optical flows and correlation information. Finally, the warped frames are adaptively fused under the guidance of the masks to generate a high‐quality intermediate frame. Furthermore, to generate more realistic video frames, they train the network model with the pixel‐based loss and the feature‐based loss in a step‐by‐step way. In the experiment, they analyse the proposed model and compare it with the high‐performance methods, both qualitative and quantitative results show that their method performs better.