Cancer is a complex, multifactorial disease. No single measurement -- whether a blood test, a biopsy, or a scan -- fully captures the range of biological processes driving tumor growth, spread, and response to treatment. Modern medicine is generating massive amounts of data across multiple domains simultaneously, from genomics and proteomics to digital pathology images, radiology scans, and electronic health records (EHRs).
Most current AI approaches in oncology analyze only one of these data types at a time, leaving most of this information unused. This creates a significant opportunity: combining complementary data types through multimodal data fusion could reveal biological signals that no single modality captures alone, potentially transforming cancer diagnosis, prognosis, and treatment selection.
The need is especially acute for therapies like immunotherapy, where response rates vary dramatically and depend on the complex cellular environment surrounding the tumor, known as the tumor microenvironment (TME). Understanding the TME requires molecular, imaging, and clinical data in combination -- no single test can characterize it fully.
This review, published in Nature Machine Intelligence, synthesizes the state of multimodal deep learning in oncology, covering data fusion strategies, current challenges including missing data and interpretability, and the path toward clinically deployable multimodal biomarker models.
Electronic health records (EHRs) contain structured data (medications, lab values, vital signs, diagnosis codes) and unstructured data (clinical notes, physician narratives). Natural language processing (NLP) techniques, including transformer-based models like BERT, can extract meaningful clinical features from free text, while recurrent neural networks process the time-ordered nature of structured EHR entries.
Molecular data -- including genomics, transcriptomics, proteomics, and epigenomics -- provides the deepest biological resolution into tumor characteristics. Large consortia like The Cancer Genome Atlas (TCGA) have assembled petabyte-scale multi-omics datasets across cancer types. However, molecular data alone discards spatial and morphological information about how cells are organized in tissue.
Digital pathology images capture tissue architecture at the cellular level through whole slide imaging. AI analysis of these images, a field called pathomics, can reveal spatial patterns in the tumor microenvironment that correlate with molecular subtypes and prognosis, including information about immune cell infiltration, gland structure, and tumor heterogeneity.
Radiographic images (CT, MRI, PET scans) provide non-invasive, three-dimensional views of tumor morphology, size, location, and surrounding structures. A field called radiogenomics has emerged to link image features directly to underlying molecular properties, demonstrating that tumor appearance on imaging can serve as a proxy for specific gene mutations.
Early fusion concatenates feature vectors from all data modalities into a single combined input that is fed into one model. This approach is simple and allows the model to learn interactions across modalities from the start, but it requires all modalities to be represented in the same way and struggles when some data types are missing for certain patients.
Late fusion trains a separate model for each modality independently, then combines the individual predictions using averaging, weighted voting, or another aggregation method. This approach allows specialized model architectures for each data type and more gracefully handles missing modalities, but it cannot learn synergistic interactions that only emerge when modalities are analyzed together.
Intermediate (joint) fusion sits between these extremes: it develops shared feature representations that capture interactions between modalities during training, rather than at the input or output stage. The training loss propagates back to update each modality's feature encoder, enabling complex cross-modal patterns to be learned. This approach is generally the most powerful but requires the most careful design to avoid overfitting.
There is currently no clear winner among these three strategies. The optimal choice depends on the available data, the specific clinical task, and the degree of missing data across patients. Many researchers combine elements of these approaches, and dedicated attention mechanisms and cross-modal transformers are emerging as particularly promising architectures for flexible multimodal integration.
Multiple studies demonstrate that fusing data modalities consistently improves predictive performance over single-modality models. One study on lung cancer combined RNA sequencing, microRNA, whole slide images, copy number variation, and DNA methylation data, achieving better survival prediction than any individual modality alone.
A pan-cancer prognosis study by Cheerla and Gevaert used an intermediate fusion strategy to integrate histopathology images, gene expression data, and clinical variables across 14 cancer types. Each modality was compressed into a compact feature vector by an unsupervised encoder, and the joint representation handled missing data by design.
In the realm of radiogenomics, studies have demonstrated that MRI image features can predict molecular subtypes in glioblastoma and breast cancer, and can predict treatment response in ways that complement but do not duplicate what molecular testing reveals. This suggests imaging captures biological signal that genomics alone misses.
For survival and prognosis specifically, combining molecular data with imaging typically adds 0.05 to 0.15 points of AUC improvement over the best single modality, which -- while seemingly modest -- can translate to meaningful differences in patient stratification when applied at scale across thousands of patients.
The biggest practical barrier to multimodal fusion is missing data. In real clinical practice, patients rarely have all data modalities available -- some may have MRI but not genomics, others may have pathology but no clinical notes. Restricting analysis to only fully-observed patients severely limits sample size and introduces selection bias. Imputation methods ranging from basic mean substitution to advanced generative adversarial networks (GANs) can fill gaps, but remain imperfect.
A related problem is the mismatch between data depth and cohort size. Modern multimodal datasets can contain thousands of features per patient, but most clinical cohorts have only hundreds or low thousands of patients. This disparity makes models highly prone to overfitting. It is also exacerbated by the fact that available cohorts tend to oversample patients from higher socioeconomic backgrounds, limiting generalizability to diverse populations.
Perhaps the most important barrier to clinical adoption is model interpretability. Deep learning models are often described as black boxes -- they produce predictions without explaining which features drove them. Clinicians are unlikely to trust or adopt models whose reasoning they cannot inspect. Techniques like Grad-CAM for imaging (highlighting relevant image regions) and SHAP values for molecular data (ranking feature importance) are advancing this area, but multimodal interpretability remains in its early stages.
Data standardization is also essential. Even within a single modality, staining protocols, scanner brands, and sequencing platforms vary across institutions, making it difficult to train models that generalize broadly. Initiatives such as TCGA, The Cancer Imaging Archive (TCIA), and the OMOP framework for EHRs are working toward FAIR (findable, accessible, interoperable, and reusable) data standards, but much work remains.
The authors argue that multimodal data fusion is the only viable path to true precision oncology. Cancer is inherently multifactorial, and single-modality biomarkers -- however sophisticated -- capture only one slice of a patient's biological reality. Achieving individualized diagnosis, prognosis, and treatment selection requires integrating information across multiple scales simultaneously.
The field is expected to advance significantly through community-driven data sharing initiatives and competition platforms such as DREAM and Kaggle, which provide standardized multimodal datasets, encourage code sharing, enable reproducibility, and allow unbiased benchmarking of competing algorithms. These platforms have already produced landmark models in genomics and imaging independently.
An important near-term opportunity is using NLP on clinical notes to automatically extract treatment response data from patient records at scale. Most clinical outcome data is locked in narrative text rather than structured fields, and NLP-powered extraction could enable the large longitudinal outcome datasets needed to train and validate multimodal prognosis models.
Ultimately, the review calls for rigorous prospective clinical validation of multimodal models in large independent cohorts, with clear regulatory frameworks for AI-driven clinical decision support. The FDA has begun publishing guidance on AI as a medical device, and the field will need to meet these standards before multimodal biomarker tools reach routine clinical practice for cancer patients.