Computerized delineation of nuclei in liquid‐based Pap smears stained with immunohistochemical biomarkers

Background Infection with high‐risk human papillomaviruses (hrHPVs) is a frequent cause of cervical intraepithelial neoplasias and carcinomas. The recently developed p16/Ki67 dual stain of cytologic preparations possesses superior specificity over current HPV testing for detecting moderate‐ and high‐grade dysplasias and can potentially be applied in routine cytology screening. Image analysis can possibly improve the efficiency of evaluating Papanicolaou (Pap) smears, if dual‐stained nuclei are accurately localized and reliably distinguished from the background of other cells. Methods Here, we describe a technique comprising color deconvolution, radial symmetry detector, and a superpixel‐based segmentation for computerized delineation of nuclei in Pap smears stained with p16/Ki67. Results The performance of the method was determined by the precision and recall in 99 images ( n  = 19,323 cells) and reached 0.952 and 0.958, respectively. The accuracy of delineation, assessed by the Jaccard index ( n  = 1,080 cells), was 0.794. In single cells the precision and recall was higher than in clumps ( P  = 0.005). Conclusions In summary, the new technique delineates large and small nuclei irrespectively of coloration with a significantly better performance than a method solely involving the radial symmetry detector. Therefore, it is suited to automatically define nuclear areas for quantification of nuclear biomarkers in smears. © 2014 International Clinical Cytometry Society