Multi‐layer random walker image segmentation for overlapped cervical cells using probabilistic deep learning methods

A method for overlapping cell image segmentation is presented with a focus on multi‐layer image processing in a three‐phase scheme. In the first phase, a convolutional neural network is developed to provide a coarse cell segmentation with multiple output layers to identify cell cytoplasm, locations of cell nuclei, and the background, all as probabilistic image maps for the layer outputs. In the second phase, the probabilistic image maps from the convolutional neural network are used to identify locations of cell nuclei and cell cytoplasm. Then, multi‐layer random walker image segmentation is used with cell nuclei as hard initial seeds and the cytoplasm estimates as soft seeds in a diffusion graph‐based segmentation of the cells. With rough cell segmentation from both the trained convolutional neural network and the multi‐layer random walker graph‐based technique, a third phase combines and refines the cell segmentation using the Hungarian algorithm to optimise the assignment of individual pixel locations for the final cell segmentation. We evaluate the proposed method on cervical cell images generated from the International Symposium on Biomedical Imaging 2014 dataset with results that give a Dice similarity coefficient of 97.2% (compared to 93.2% for competitors) when trained on the generated dataset.