Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

As the deep learning revolution marches on, self-supervised learning hasgarnered increasing attention in recent years thanks to its remarkablerepresentation learning ability and the low dependence on labeled data. Amongthese varied self-supervised techniques, masked modeling has emerged as adistinctive approach that involves predicting parts of the original data thatare proportionally masked during training. This paradigm enables deep models tolearn robust representations and has demonstrated exceptional performance inthe context of computer vision, natural language processing, and othermodalities. In this survey, we present a comprehensive review of the maskedmodeling framework and its methodology. We elaborate on the details oftechniques within masked modeling, including diverse masking strategies,recovering targets, network architectures, and more. Then, we systematicallyinvestigate its wide-ranging applications across domains. Furthermore, we alsoexplore the commonalities and differences between masked modeling methods indifferent fields. Toward the end of this paper, we conclude by discussing thelimitations of current techniques and point out several potential avenues foradvancing masked modeling research. A paper list project with this survey isavailable at \url{https://github.com/Lupin1998/Awesome-MIM}.