Inconsistency-Based Data-Centric Active Open-Set Annotation

Active learning is a commonly used approach that reduces the labeling effortrequired to train deep neural networks. However, the effectiveness of currentactive learning methods is limited by their closed-world assumptions, whichassume that all data in the unlabeled pool comes from a set of predefined knownclasses. This assumption is often not valid in practical situations, as theremay be unknown classes in the unlabeled data, leading to the active open-setannotation problem. The presence of unknown classes in the data cansignificantly impact the performance of existing active learning methods due tothe uncertainty they introduce. To address this issue, we propose a noveldata-centric active learning method called NEAT that actively annotatesopen-set data. NEAT is designed to label known classes data from a pool of bothknown and unknown classes unlabeled data. It utilizes the clusterability oflabels to identify the known classes from the unlabeled pool and selectsinformative samples from those classes based on a consistency criterion thatmeasures inconsistencies between model predictions and local featuredistribution. Unlike the recently proposed learning-centric method for the sameproblem, NEAT is much more computationally efficient and is a data-centricactive open-set annotation method. Our experiments demonstrate that NEATachieves significantly better performance than state-of-the-art active learningmethods for active open-set annotation.