Hybrid ResNet-ViT Transfer Learning Approach for Brain Stroke Classification on Computed Tomography Images

This study investigates the utilization of a hybrid Convolutional Neural Network (CNN) and Vision Transformer (ViT) model, employing transfer learning methods, to enhance brain stroke detection and classification of CT images. The objective is to integrate ResNet-101’s local feature extraction capabilities with ViT’s global context comprehension to develop a resilient model for precise detection and categorization of brain strokes using non-contrast-enhanced brain computed tomography (NCCT) data. Data from two hospitals in Sri Lanka comprising 11,300 images were retrospectively collected. ViT and ResNet-101 architectures were modified for multi-class classification of brain normal, ischemic, and hemorrhagic conditions, and further differentiated ischemic acute, subacute, and chronic conditions in two-step ways including two models of the proposed architecture. We developed two models, incorporating the ResNet-101 component with MC dropout layer and a fully connected layer by removing the final two layers and the ViT component modifying multi-layer perceptron, incorporating three classes by adding a fully connected layer. The proposed model 01 training, and testing accuracy were 99.69%, 99.16% whereas model 02 achieved 97.31%, and 95.33% respectively. The hybrid models offer a robust method for impartial stroke diagnosis, potentially enabling tailored treatment approaches based on stroke type and severity. Further validation of the proposed approach on larger and more diverse datasets in clinical settings is required.