Improving Generalization Capability of Deep Learning-Based Nuclei Instance Segmentation by Non-deterministic Train Time and Deterministic Test Time Stain Normalization

With the advent of digital pathology and microscopic systems that can scanand save whole slide histological images automatically, there is a growingtrend to use computerized methods to analyze acquired images. Among differenthistopathological image analysis tasks, nuclei instance segmentation plays afundamental role in a wide range of clinical and research applications. Whilemany semi- and fully-automatic computerized methods have been proposed fornuclei instance segmentation, deep learning (DL)-based approaches have beenshown to deliver the best performances. However, the performance of suchapproaches usually degrades when tested on unseen datasets. In this work, we propose a novel method to improve the generalizationcapability of a DL-based automatic segmentation approach. Besides utilizing oneof the state-of-the-art DL-based models as a baseline, our method incorporatesnon-deterministic train time and deterministic test time stain normalization,and ensembling to boost the segmentation performance. We trained the model withone single training set and evaluated its segmentation performance on seventest datasets. Our results show that the proposed method provides up to 4.9%,5.4%, and 5.9% better average performance in segmenting nuclei based on Dicescore, aggregated Jaccard index, and panoptic quality score, respectively,compared to the baseline segmentation model.