Eff-SAM: SAM-based Efficient Method for Brain Tumor Segmentation in Multimodal 3D MRI Scans

Accurate brain tumor segmentation in MRI scans is crucial for effective diagnosis and treatment planning, as even minor segmentation errors can lead to significant clinical concerns. However, this task is challenged by the complex anatomy of the brain, variable tumor shapes, and low contrast between tumor sub-regions. Moreover, existing methods often lack generalizability across diverse segmentation tasks and imaging modalities. Large Vision Model (LVM) like the Segment Anything Model (SAM) represents a foundational advancement in tumor segmentation. While SAM has shown impressive performance on natural images, its effectiveness in medical imaging, especially brain tumor segmentation, is limited due to domain differences and indistinct boundaries in MRI scans. To address these challenges, we propose Eff-SAM, a 3D brain tumor segmentation framework that adapts SAM for medical applications through Parameter-Efficient Fine-Tuning (PEFT) technique. The framework incorporates PEFT with adapters into SAM’s encoder to fine-tune the model effectively and optimize its performance for medical imaging while maintaining computational efficiency. Additionally, a Cross-Sliced Attention (CSA) mechanism captures semantic relationships, improves tumor localization, and is introduced into the encoder. Robust data preprocessing further enhances the model’s generalization across datasets. Eff-SAM demonstrates state-of-the-art performance and outperforms benchmark methods on the BraTS 2020 and BraTS 2021 datasets. It achieves Dice scores of 0.884 for Whole Tumor (WT), 0.853 for Tumor Core (TC), and 0.818 for Enhancing Tumor (ET) on the BraTS 2020 dataset, and 0.880 (WT), 0.861 (TC), and 0.821 (ET) on the BraTS 2021 dataset. This work highlights the potential of integrating vision models like SAM with lightweight, domain-specific modules to deliver accurate and efficient brain tumor segmentation offering a clinically valuable tool, especially in scenarios with limited data and complex tumor morphology.