MeTA: Multi-source Test Time Adaptation

Test time adaptation is the process of adapting, in an unsupervised manner, apre-trained source model to each incoming batch of the test data (i.e., withoutrequiring a substantial portion of the test data to be available, as intraditional domain adaptation) and without access to the source data. Since itworks with each batch of test data, it is well-suited for dynamic environmentswhere decisions need to be made as the data is streaming in. Current test timeadaptation methods are primarily focused on a single source model. We proposethe first completely unsupervised Multi-source Test Time Adaptation (MeTA)framework that handles multiple source models and optimally combines them toadapt to the test data. MeTA has two distinguishing features. First, itefficiently obtains the optimal combination weights to combine the sourcemodels to adapt to the test data distribution. Second, it identifies which ofthe source model parameters to update so that only the model which is mostcorrelated to the target data is adapted, leaving the less correlated onesuntouched; this mitigates the issue of "forgetting" the source model parametersby focusing only on the source model that exhibits the strongest correlationwith the test batch distribution. Experiments on diverse datasets demonstratethat the combination of multiple source models does at least as well as thebest source (with hindsight knowledge), and performance does not degrade as thetest data distribution changes over time (robust to forgetting).