An analysis of two ground truth estimation methods

An estimation of the so called Ground Truth (GT), i.e. the actual lesion region, can minimize readers' sub- jectivity if multiple readers' markings are combined. Two methods perform this estimate by considering the spatial location of voxels: Thresholded Probability-Map (TPM) and Simultaneous Truth and Performance Level Estimation (STAPLE). An analysis of these two methods has already been performed. The purpose of this study, however, is gaining a new insight into the method outcomes by comparing the estimated regions. A subset of the publicly available Lung Image Database Consortium archive was used, selecting pulmonary nodules documented by all four radiologists. The TPM estimator was computed by assigning to each voxel a value equal to average number of readers that included such voxel in their markings and then applying a threshold of 0.5. Our STAPLE implementation is loosely based on a version from ITK, to which we added the graph cut post-processing. The pair-wise similarities between the estimated ground truths were analyzed by computing the respective Jaccard coefficients. Then, the sign test of the differences between the volumes of TPM and STAPLE was performed. A total of 35 nodules documented on 26 scans by all four radiologists were available. The spatial agreement had a one-sided 90% Confidence Interval of (0.92, 1.00). The sign test of the differences had a p-value less than 0.001. We found that (a) the differences in their volume estimates are statistically significant, (b) the spatial disagreement between the two estimators is almost completely due to the exclusion of voxels marked by exactly two readers, (c) STAPLE tends to weight more, in its GT estimate, readers marking broader regions.