Comparison of a segmentation algorithm to six expert imagiologists in detecting pulmonary contours on x-ray CT images

Quantitative evaluation of the performance of segmentation algorithms on medical images is crucial before their clinical use can be considered. We have quantitatively compared the contours obtained by a pulmonary segmentation algorithm to contours manually-drawn by six expert imaiologists on the same set of images, since the ground truth is unknown. Two types of variability (inter-observer and intra-observer) should be taken into account in the performance evaluation of segmentation algorithms and several methods to do it have been proposed. This paper describes the quantitative evaluation of the performance of our segmentation algorithm using several figures of merit, exploratory and multivariate data analysis and non parametric tests, based on the assessment of the inter-observer variability of six expert imagiologists from three different hospitals and the intra-observer variability of two expert imagiologists from the same hospital. As an overall result of this comparison we were able to claim that the consistency and accuracy of our pulmonary segmentation algorithm is adequate for most of the quantitative requirements mentioned by the imagiologists. We also believe that the methodology used to evaluate the performance of our algorithm is general enough to be applicable to many other segmentation problems on medical images.