Tiny Diffusion, Big Brain: Lightweight Dual-control Stable Diffusion for Brain MRI Synthesis

Serena Pei, School of Engineering, MIT, Cambridge, MA, USA

ABSTRACT

We present a lightweight pipeline using Stable Diffusion v1.5 [7] for generating anatomically accurate brain MRI images depicting tumors. Using a public dataset of 1,426 glioma MRI slices from 233 patients [5,2] we condition image generation on both descriptive text prompts (text input) and visually transformed grayscale MRI slices (visual input). We explore three visual transforms: Gaussian-blurring, checkerboard-masked, and edge-mapped. Inspired by ControlNet [9], our method supports dual conditioning during both training and inference but avoids duplicating the U-Net architecture—significantly reducing memory overhead. This enables training on standard GPUs such as a single 15GB T4 in Google Colab. To assess image realism on synthesized images, we use both qualitative inspection and Fréchet Inception Distance (FID). This model is an important step towards building more flexible, privacy-preserving methods for creating high-quality medical images in low-data, low-memory settings— with potential applications in rare disease research and AI-driven healthcare.

Keywords

Stable Diffusion, ControlNet, healthcare, medical imaging.