Understanding cancer requires reading multiple kinds of information simultaneously -- the structure of tissue, which genes are active, which proteins are present, and how the patient has responded to treatment. Traditionally, these data types were analyzed separately, missing crucial connections between them. Multimodal data analysis integrates genomic, proteomic, imaging, and clinical data to build a more complete picture of cancer biology.
Spatial transcriptomics (ST) is a technology that maps gene expression -- which genes are active and how much -- directly onto the physical locations within a tissue sample. Rather than averaging gene activity across an entire tumor, ST reveals which cells in which specific areas are behaving differently. This spatial context is critical because the behavior of a cancer cell is shaped by its neighbors, the immune cells surrounding it, and the local tissue environment.
Deep learning (DL), a branch of artificial intelligence using layered neural networks, has demonstrated exceptional ability to identify patterns in complex data -- whether images, sequences, or numbers. Convolutional neural networks (CNNs) in particular excel at interpreting medical images, identifying tumors, segmenting tissue regions, and predicting treatment outcomes from visual data.
This chapter reviews how combining DL with multimodal ST is creating a new paradigm in cancer research -- one where gene expression patterns, tissue images, and molecular profiles are analyzed together, enabling more precise diagnosis, personalized treatment planning, and a deeper understanding of why cancers behave the way they do.
Spatial transcriptomics preserves the location of each cell's gene activity within a tissue sample, allowing researchers to ask not just what genes are active but where. The technology relies on high-throughput sequencing methods applied directly to tissue sections, producing a gene expression map that aligns with the physical structure of the tumor and surrounding tissue.
The main technical approaches include in situ hybridization methods such as MERFISH and seqFISH, which use fluorescent probes to detect RNA molecules at their exact locations within the tissue with very high spatial resolution. Spatially resolved transcriptomics platforms like 10x Genomics' Visium use spatial barcodes printed on a slide to capture RNA from the tissue placed on top, mapping transcripts to defined coordinates.
Laser capture microdissection (LCM) is a complementary technique in which a laser is used to physically cut out specific small regions of tissue for separate RNA extraction and sequencing. LCM offers high precision for analyzing rare or small cell populations that would be diluted by bulk approaches.
What makes ST especially powerful in cancer is its ability to reveal tumor heterogeneity -- the fact that different regions of the same tumor can have very different gene activity profiles. It also maps tumor microenvironment (TME) interactions, showing where immune cells, stromal cells, and cancer cells coexist and how they influence each other spatially.
Graph neural networks (GNNs) are particularly suited for ST data because they naturally represent spatial relationships. In ST, tissue spots or cells can be modeled as nodes in a graph, with edges encoding proximity. GNNs analyze these networks to detect cellular interaction patterns associated with cancer progression. The STGNNks method combines GNNs with denoising autoencoders and clustering to identify tissue regions with distinct transcriptional profiles and track tumor progression through spatial trajectory analysis.
Convolutional neural networks (CNNs) excel at analyzing histopathological images and can be trained to predict which genes are highly active just from tissue images -- connecting visual morphology to molecular biology. One method achieved 9-fold improvement in spatial resolution of gene expression maps using a CNN-based superresolution technique, revealing fine-grained expression patterns previously invisible at standard resolution.
Transformer models, which use a self-attention mechanism to relate distant parts of an input simultaneously, are being applied to whole tissue sections to capture long-range gene expression dependencies. A contrastive learning framework using transformers successfully inferred molecular subtypes and clinical outcomes of breast cancer from unannotated whole-slide images, establishing connections between tissue appearance and underlying genetics.
Hybrid approaches combine CNNs, transformers, and GNNs to capture local texture, global context, and spatial relationships together. The ErwaNet model constructs heterogeneous graphs to model local tissue interactions while using attention mechanisms to integrate global information, offering an efficient and accurate approach to predicting gene expression from standard tissue images.
Tumor heterogeneity -- the coexistence of genetically distinct cell populations within a single tumor -- is a major driver of treatment resistance and disease progression. ST combined with single-cell sequencing can map these subclonal populations spatially, revealing how aggressive subpopulations cluster and interact. In esophageal cancer, AI-driven multi-omics ST analysis identified previously unknown cell types and their spatial relationships, clarifying how intratumor diversity shapes disease behavior.
The tumor microenvironment (TME) consists of immune cells, fibroblasts, blood vessels, and signaling molecules surrounding cancer cells. Spatial profiling reveals how these components are arranged and communicate. In hepatocellular carcinoma, ST identified the spatial distribution of tumor-infiltrating lymphocytes and specialized immune cells called stimulatory dendritic cells and macrophages, which function as biomarkers for predicting response to immunotherapy.
Understanding therapy response requires knowing where in the tumor there are regions of hypoxia (oxygen deprivation), necrosis, and abnormal vasculature -- factors that create treatment-resistant zones. ST can map these features spatially, identifying pockets of resistance that would be invisible with standard analysis. In breast cancer, integrating genomic, transcriptomic, and imaging data allowed identification of molecular biomarkers that predict response to specific drug treatments, advancing truly personalized treatment selection.
In cancer diagnosis, integrating histopathological images with genomic data has enabled more precise tumor subtype classification. The FDTrans model combined tissue images with genomic profiles to classify lung cancer subtypes with high accuracy, demonstrating that multimodal fusion outperforms either data type alone. A modality-correlation embedding model for breast tumor diagnosis using both mammography and ultrasound images achieved 91.67% sensitivity and 95.83% specificity -- substantially better than single-modality approaches.
For personalized treatment planning, deep learning models processing multimodal data can predict how a specific patient's tumor will respond to chemotherapy or radiotherapy before treatment begins. A hybrid deep learning method combining pathology images with electronic medical record data significantly outperformed single-modality methods for breast cancer patient stratification, enabling more precise assignment of patients to the treatments most likely to benefit them.
For lung cancer staging, integrating microbial data with transcriptomics using combinatorial fusion analysis achieved an accuracy of 80.9% -- one of the highest reported for this task. This illustrates how multimodal integration can uncover unexpected data sources that carry diagnostic information, expanding the information available for clinical decisions beyond conventional imaging and pathology.
Despite impressive results, several technical and practical challenges limit the clinical adoption of DL-based multimodal ST. ST data is inherently sparse and noisy -- many gene measurements in a tissue spot may be missing or unreliable due to low RNA capture efficiency. Addressing this requires robust imputation methods and careful preprocessing, and current approaches vary widely across research groups.
A major barrier is the lack of standardization across imaging platforms, sequencing technologies, and data analysis pipelines. Results from one institution using one sequencing platform may not translate to another institution using different equipment. Batch effects -- systematic technical differences between experiments -- can mimic or obscure real biological signals, requiring careful correction that is not yet standardized.
The interpretability of deep learning models remains a significant concern for clinical use. A model that predicts cancer recurrence from a tissue image but cannot explain which features drove the prediction is difficult for clinicians to trust and impossible to audit for errors or biases. Techniques like attention visualization and SHAP values are being incorporated to improve explainability, but the field has not yet converged on standards for model transparency in clinical AI.
Finally, data availability and sharing bottlenecks slow progress. Annotated multimodal datasets -- which require expert pathologists, genomics sequencing, and clinical outcomes for the same patients -- are expensive to create and rarely shared openly. Most published models are trained on hundreds of patients rather than the tens of thousands that would support robust generalization.
The authors identify several research priorities for advancing the field. Improving sequencing depth and preprocessing robustness will reduce data sparsity and noise. Developing standardized workflows and data-sharing repositories -- analogous to The Cancer Genome Atlas but with spatial and multimodal components -- will allow models to be trained on larger, more diverse datasets and validated externally across institutions.
Integrating DL-based ST with experimental validation will be critical to confirm that patterns identified computationally reflect real biological phenomena rather than statistical artifacts. AI predictions about spatial interactions between immune cells and tumor cells, for example, should be verified with targeted functional assays before being used to guide clinical decisions.
The long-term vision is a system where a patient's tissue sample is analyzed holistically -- with gene expression, protein levels, morphology, and clinical history all processed together by a unified AI model that produces actionable diagnostic and treatment recommendations. Such a system would represent the clinical realization of precision oncology: treating each patient's cancer based on its unique molecular and spatial architecture rather than its histological category alone.