Depth integrated Multi-Task Prototypical Learning with Self refinement for Unsupervised Domain Adaptation

Unsupervised Domain Adaptation (UDA) serves as a potential alternative for improving cross-domain segmentation tasks. Recent UDA approaches have identified class-wise prototypes and leveraged them to guide the segmentation process in the target domain. However, these methods overlook additional information from other auxiliary tasks, such as depth, which can potentially improve overall segmentation performance. This paper proposes a Depth-Aware Prototypical Learning for Semantic Segmentation (DA-ProSS) pipeline, which includes a novel multitask prototype learning framework that comprises task-specific and task-integrated classifiers. The task-specific classifiers capture the semantic and depth features, and the task-integrated classifier captures the hidden semantic features from depth prediction. This framework ensures that the prototypes from respective heads learn better class-representative features using semantic information and depth cues. Additionally, this pipeline integrates a Self-Refinement Learning (SRL) algorithm that generates cross-domain pseudo-labels, which are leveraged to generate refined targets for further self-supervised training. Results indicate that prototypes generated through the depth-encoded semantic task could understand the underlying semantics of the object. The proposed DA-ProSS pipeline with SRL helps the model generalize better to achieve a mIoU of 57.3% that outperforms the previous state-of-the-art methods in the SYNTHIA-to-Cityscapes benchmark dataset.