Flexible Deep Learning Frameworks for Heterogeneous 3D Brain Magnetic Resonance Imaging
Deep learning has become widely used in medical imaging to improve diagnostic accuracy, but real-world clinical data is messy: imaging protocols, modalities, and scanner settings all vary across institutions. Radiologists handle this variability naturally, but most AI systems are built around fixed input formats, which limits how well they adapt, generalize, and get reused across different clinical setups. This thesis proposes deep learning frameworks specifically designed to handle that heterogeneity in 3D medical imaging, working from the idea that models built to tolerate this kind of variation can learn more effectively and generalize more broadly, while also exploring ways to make Transformer-based models more efficient.
The first step is a coordinate-based embedding that boosts Transformer model performance. Building on that, the thesis introduces the Multiplane Vision Transformer (MPViT), which can work directly with MRI images in different orientations without having to resample everything into a single plane first, avoiding the information loss that resampling causes.
The thesis then introduces the Adaptive Vision Transformer (AdaViT), a framework built to generalize across varying sets of input modalities. AdaViT outperforms conventional approaches in adaptability and performance across zero-shot testing, few-shot fine-tuning, and backward transfer, making it a strong foundation for handling heterogeneous MRI inputs. Building further on this, the thesis presents the Variable-input Vision Transformer (VIViT), designed for self-supervised pretraining and segmentation fine-tuning across studies with different input contrasts. Together, AdaViT and VIViT make better use of available data during pretraining and transfer what they’ve learned to new tasks even when input requirements shift, outperforming conventional methods in both adaptability and accuracy. The thesis also introduces SegResMamba, an efficient hybrid segmentation model that balances global context modeling against computational cost.
These methods are evaluated mainly on neuroimaging tasks, including brain infarct segmentation, brain tumor segmentation, and hemorrhage classification, with promising results throughout. Together, they lay groundwork for AI systems that can scale and generalize across the genuinely heterogeneous nature of MRI data, moving a step closer toward foundation models in medical imaging that learn well from diverse data and transfer reliably to real clinical settings.