Predictive and discriminative localization of pathology using high resolution class activation maps with CNNs

Purpose Existing class activation mapping (CAM) techniques extract the feature maps only from a single layer of the convolutional neural net (CNN), generally from the final layer and then interpolate to upsample to the original image resolution to locate the discriminative regions. Consequently these provide a coarse localization that may not be able to capture subtle abnormalities in medical images. To alleviate this, our work proposes a technique called high resolution class activation mapping (HR-CAMs) that can provide enhanced visual explainability to the CNN models. Methods HR-CAMs fuse feature maps by training a network using the input from multiple layers of a trained CNN, thus gaining information from every layer that can localize abnormalities with greater details in original image resolution. The technique is validated qualitatively and quantitatively on a simulated dataset of 8,000 images followed by applications on multiple image analysis tasks that include (1) skin lesion classification (ISIC open dataset—25,331 cases) and (2) predicting bone fractures (MURA open dataset—40,561 images) (3) predicting Parkinson’s disease (PD) from neuromelanin sensitive MRI (small cohort-80 subjects). Results We demonstrate that our model creates clinically interpretable subject specific high resolution discriminative localizations when compared to widely used CAMs and Gradient-CAMs. Conclusion HR-CAMs provide finer delineation of abnormalities thus facilitating superior explainability to CNNs as has been demonstrated from its rigorous validation.