Two complementary data types Histopathology slide images encode tumor morphology, immune infiltration, and tissue architecture at cellular resolution. Multi-omics data (gene expression, copy number variation, DNA methylation) captures molecular alterations driving tumor biology. Neither data source alone fully characterizes prognosis - the combination should be more informative than either alone.
The fusion challenge Combining image and omics data is non-trivial because they have different formats, dimensionalities, and noise characteristics. Zhao and colleagues propose a decision-level fusion strategy where separate deep learning networks first process each modality independently, then a fusion network combines their outputs. This avoids the curse of dimensionality that plagues feature-level concatenation approaches.
Key results The Fusion Net achieved a C-index of 0.67 for NSCLC overall survival prediction compared to 0.61 for the best single-modality model, representing a clinically meaningful 10% improvement. A similar gain was observed in the BRCA breast cancer cohort, suggesting the approach generalizes across cancer types.
Multi-omics inputs Three omics layers were included for each patient: (1) mRNA gene expression from RNA-seq (log-normalized FPKM values for approximately 20,000 genes), (2) copy number variation (CNV) segments summarized as gene-level amplification/deletion scores, and (3) DNA methylation from Infinium 450K arrays summarized as mean beta values per gene promoter. These three layers capture transcriptional activity, chromosomal structural changes, and epigenetic regulation respectively.
Histology slide images Whole-slide H&E images from TCGA were processed by tiling into 256x256 pixel patches at 20x magnification. Approximately 500-2000 patches per slide were extracted from tissue regions. A pretrained ResNet50 (initialized from ImageNet weights) extracted 2048-dimensional feature vectors from each patch, which served as input nodes in the graph construction.
Dataset characteristics TCGA-NSCLC provided 802 patients with survival outcomes and both omics and image data available. TCGA-BRCA provided 679 patients. The authors restricted analysis to patients with complete data across all modalities, which reduces sample size but ensures the comparison between fusion and single-modality approaches is on equivalent patients.
Why graph convolution for slides Patches extracted from a whole slide have spatial relationships - patches near each other share tissue context. Standard pooling of independent patch features ignores this spatial structure. A graph convolutional network (GCN) represents each patch as a node with edges connecting spatially adjacent patches, enabling information to propagate between neighboring regions when computing slide-level representations.
Graph construction For each slide, patches were embedded in a spatial graph where edges connected patches within a defined radius (representing physical proximity on the slide). The ResNet50 features at each node were updated through two rounds of graph convolution, allowing each patch's representation to incorporate context from its tissue neighborhood before being aggregated into a slide-level feature vector via mean pooling.
Survival prediction from images alone The GCN pathway alone achieved a C-index of 0.59 for NSCLC and 0.62 for BRCA, establishing a baseline for image-only prognosis. These performance levels are consistent with published computational pathology models on TCGA data, validating the implementation before fusion.
Dimensionality reduction for omics The raw multi-omics feature vectors are extremely high dimensional (approximately 60,000 features across three layers for 20,000 genes each). Before feeding into the MLP, each omics layer was independently compressed using principal component analysis to 256 dimensions, retaining approximately 80% of variance. The three 256-dimension vectors were then concatenated into a 768-dimensional integrated omics feature vector.
MLP structure The omics MLP used three fully connected layers with 512, 256, and 128 neurons, with batch normalization and dropout (rate 0.3) after each layer to prevent overfitting. The 128-dimensional output vector served as the omics representation entering the fusion network. Survival prediction from omics alone achieved C-index 0.61 for NSCLC and 0.63 for BRCA.
Attention weighting across omics layers An attention mechanism was applied across the three omics compression outputs to allow the network to learn which omics layer was most informative for prognosis in each cancer type. In NSCLC, gene expression received the highest attention weight, while in BRCA, DNA methylation received relatively more weight, reflecting known biology-driven differences in the molecular drivers of these cancers.
Fusion architecture The GCN image output vector and MLP omics output vector are concatenated and passed through a final two-layer fusion network that learns the optimal combination for survival prediction. Crucially, this decision-level approach preserves the separately-learned modality-specific representations rather than forcing premature feature mixing, which would introduce information loss and make interpretation harder.
C-index results The Fusion Net achieved C-index 0.67 for NSCLC (versus 0.61 omics alone, 0.59 image alone) and 0.70 for BRCA (versus 0.63 omics alone, 0.62 image alone). Kaplan-Meier curves for predicted high-risk versus low-risk groups showed more distinct separation with the fusion model than with either single modality, with log-rank p-values more significant in the fusion setting.
Feature contribution analysis Gradient-based feature importance analysis showed that fusion model predictions weighted both modalities, with the relative contribution of images varying by patient subgroup. Patients with ambiguous molecular subtype assignments showed higher image feature contributions, suggesting pathology images can provide prognostic information complementary to molecular classification in histologically heterogeneous cases.
TCGA-specific limitations TCGA data was collected under controlled research conditions with high-quality specimens and uniform processing. Performance on routinely collected clinical samples with variable pre-analytic quality, missing omics data, or older scanner technologies may be lower. Prospective clinical validation studies with realistic data quality are needed.
Missing modality handling The current framework requires complete data across all four modalities (three omics layers plus image). In clinical practice, not all patients will have RNA-seq, CNV, and methylation data available. Future versions should implement robust missing modality handling through imputation or modality-specific confidence weighting, enabling partial-data predictions.
Toward single-cell and spatial omics fusion The multi-omics inputs currently use bulk sequencing data, averaging molecular signals across heterogeneous cell populations. Replacing bulk transcriptomics with single-cell or spatial transcriptomics data, and pairing this with matched spatial pathology analysis, would allow cell-type-specific molecular features to be co-registered with their tissue location - a much richer information fusion than is currently possible.