Clear cell renal cell carcinoma (ccRCC) is the most common type of kidney cancer, responsible for more than 175,000 deaths worldwide each year. Despite its severity, there is no clearly defined set of prognostic biomarkers used in clinical routine. Current prognosis relies almost entirely on TNM staging and histopathological grading, which have remained the sole determinants of patient outlook for decades. Existing clinical tools such as the UCLA Integrated Staging System (UISS) and the IMDC risk model incorporate clinical parameters like performance status, hemoglobin, and calcium levels, but they can be cumbersome to use and only incorporate a subset of available patient information.
Clinical management of ccRCC spans multiple disciplines including urology, radiology, oncology, and pathology, each generating highly complex medical data. A single patient's record may include CT/MRI scans, whole slide images (WSIs) of tissue, and genomic data from sequencing. The authors argue that artificial intelligence is uniquely suited to extract meaningful patterns from these diverse data streams simultaneously, mimicking the interdisciplinary decision-making that happens at tumor boards but at a speed and scale that manual review cannot match.
The study's goal was to develop and evaluate a multimodal deep learning model (MMDLM) that combines histopathological images, radiological scans, and genomic information to predict survival outcomes in ccRCC patients. This approach represents one of the first efforts to fuse three distinct medical data modalities for prognosis prediction in kidney cancer, rather than relying on a single data source.
The study used two patient cohorts of ccRCC cases. The primary cohort came from The Cancer Genome Atlas (TCGA), specifically the KIRC (Kidney Renal Clear Cell Carcinoma) dataset, comprising 230 patients for whom diagnostic H&E-stained whole slide images, radiological scans, and clinical data were all available. Histopathological slides and CT/MRI scans were downloaded through the GDC portal and the Cancer Imaging Archive, while clinicopathological data, disease-specific survival (DSS) information, and the ten most frequent mutations and copy number alterations were gathered from cBioPortal.
The second cohort was an external test set of 18 patients from the University Medical Center Mainz, diagnosed between 2011 and 2015. Although small, this cohort was deliberately kept limited to ensure high quality across radiologic, pathologic, and clinical follow-up data. The cohort included both non-metastatic and metastatic ccRCC patients, with tumor staging and grading carried out according to ISUP guidelines. Use of patient data was approved by the ethical committee of the medical association of Rhineland-Palatinate.
From the TCGA cohort, the team generated 58,829 tiles at level 5 (approximately 10x magnification) and 17,514 tiles at level 10 (approximately 5x magnification) from 230 WSIs. The radiological data consisted of approximately 199 CT scans and 31 MRI scans, from which 690 images (coronal, sagittal, and transversal planes) were extracted showing the maximum tumor diameter. For CT, nephrogenic or late systemic arterial phase images were preferred; for MRI, T1-weighted sequences were used when possible.
The multimodal deep learning model (MMDLM) was built from multiple parallel branches, one for each data modality. Each imaging modality (histopathology at level 5, histopathology at level 10, and radiology) was processed by its own 18-layer residual neural network (ResNet18). Genomic data was handled through a separate dense (fully connected) layer. The ResNet18 architecture was chosen as a deliberate compromise between model depth and computational time, providing enough capacity to learn meaningful features without excessive training overhead.
After each branch produced its output, the representations were combined through an attention layer that weighted every modality's contribution according to its importance for the prediction task. This multimodal fusion approach is critical because it allows the model to learn which data sources matter most for a given patient rather than treating all inputs equally. The fused representation then passed through a final fully connected network, leading to either a concordance index (C-index) calculation for survival ranking or a binary classification for predicting 5-year disease-specific survival status (5YSS).
Training followed a staged protocol. Unimodal training was performed first by muting all other inputs and initializing ResNet weights from pretrained ImageNet weights. Multimodal training then used the pretrained weights from the unimodal phase. The model trained for 200 to 400 epochs using Cox loss (based on the DeepSurv framework) for survival ranking and cross-entropy loss for binary classification. Optimization used stochastic gradient descent with a learning rate of 0.004, momentum of 0.9, and batch size of 32.
A customized data loader generated random combinations of one histopathologic image at level 5, one at level 10, and one radiologic image per patient during training. Genomic data was limited to the ten most frequent mutations to avoid making the model overly complex. Validation used the Cartesian product of fixed image combinations for reproducibility. The team used 6-fold cross-validation for C-index prediction and 12-fold cross-validation for binary classification on the TCGA cohort.
The researchers first established baselines for each imaging modality alone. Unimodal training on radiological data (CT/MRI) yielded a mean C-index of 0.7074 (max 0.7590). Training on histopathological tiles alone achieved a mean C-index of 0.7169 at level 10 (5x magnification) and 0.7424 at level 5 (10x magnification). Level 5, the higher magnification, consistently outperformed level 10, suggesting that finer cellular detail carries stronger prognostic signal in ccRCC.
When the MMDLM combined histopathological and radiological inputs through multimodal fusion, the mean C-index increased to 0.7791 with a maximum of 0.8123. This improvement was statistically significant compared to both radiology-only training (p = 0.0207) and histopathology-only training (p = 0.0140). The multimodal model was the only approach that significantly outperformed all independent clinical prognostic factors, including histopathological grading (C-index 0.7010), T-Stage (0.7470), N-Stage (0.5140), and M-Stage (0.6850).
For the clinically actionable 5-year survival status (5YSS) binary classification task, the MMDLM achieved a mean accuracy of 83.43% with a maximum of 100% across 12-fold cross-validation. The AUC of the ROC reached 0.916 (max 1.0) and the AUC of the precision-recall curve was 0.944 (max 1.0). Stratifying patients according to the MMDLM's prediction into "Alive" versus "Dead" groups produced highly significant differences in Kaplan-Meier survival curves, confirming the model's ability to separate low-risk from high-risk patients.
To assess whether the MMDLM's predictions carried independent prognostic value beyond known clinical risk factors, the authors performed multivariable Cox regression analysis. Among all variables tested (T-Stage, N-Stage, M-Stage, histopathological grading, and MMDLM prediction), only T-Stage and the MMDLM prediction emerged as independent, statistically significant prognostic factors. The MMDLM displayed the highest hazard ratio of nearly 4, meaning patients classified as high-risk by the model had roughly four times the mortality risk compared to those classified as low-risk, after controlling for all other factors.
A somewhat surprising finding emerged when the researchers tested whether adding genomic data could further improve the image-based prognosis prediction. They compared the MMDLM's performance with and without training on the top ten mutations and copy number alterations (CNAs) found in the cohort (including VHL, the most commonly mutated gene in ccRCC). The addition of genomic information did not improve performance. Neither individual mutations nor the combined mutational profile showed a statistically significant difference in patient survival.
The authors interpret this result in the context of ccRCC biology. Unlike some cancers where specific mutations carry strong prognostic weight, ccRCC is highly dependent on mutations (particularly VHL) that are extremely common across the disease. Because these mutations are nearly ubiquitous in the cohort, they provide little discriminative power for distinguishing patients with different prognoses. The imaging modalities, by contrast, capture downstream phenotypic consequences of the full genomic landscape, including tissue architecture, vascularity, and necrosis patterns that collectively encode more prognostic information than the top mutations alone.
To test generalizability beyond the TCGA training data, the MMDLM was evaluated on the 18-patient Mainz cohort, which represented 9.3% of the training set for C-index calculation and 17.6% for binary classification. On this external test set the mean C-index reached 0.799 (max 0.8662), accuracy averaged 79.17% (max 94.44%), AUC of the ROC was 0.905 (max 1.0), and AUC of the precision-recall curve was 0.956 (max 1.0). None of these metrics were significantly different from those achieved during cross-validation on the TCGA cohort, indicating that the model generalized well despite differences in scanning equipment and institutional protocols.
To increase model transparency, the authors employed two visualization strategies. First, they used a sliding-window approach to generate classification markup on unimodal histopathology WSIs, producing spatial maps that showed which tile regions were classified as "alive" versus "deceased" and the prediction certainty within the majority class. This allowed pathologists to see how the model's predictions distributed across the tissue.
Second, the team established class activation maps (CAMs) using the cross-validation fold with the highest C-index prediction, covering 17,550 image combinations. A descriptive screening of representative CAMs revealed that the model focused on specific histopathologic features such as tumor vasculature, hemorrhage, and necrosis for high-risk predictions. For low-risk predictions, the model highlighted clear cell morphology and papillary tumor appearance. On the radiology side, tumor volume emerged as a key feature driving the model's risk assessment.
The authors highlight three major improvements their work offers over previous studies. First, their model integrates comprehensive histopathologic and radiologic imaging together with whole exome sequencing data, mirroring the interdisciplinary decision-making process at clinical tumor boards. Second, the target variable is patient prognosis rather than diagnosis of a tumor entity, addressing the urgent need for reliable prognostic biomarkers in ccRCC. Third, the use of visualization techniques (CAMs and sliding-window markup) adds interpretability that is essential for clinical adoption. The multimodal fusion approach consistently outperformed unimodal training, aligning with reports from non-medical fields like autonomous driving where multi-sensor fusion yields accuracy gains of up to 27.7%.
The study also contextualizes its results against related work. A concurrent study by Ning et al. used convolutional neural networks (CNNs) for feature extraction on radiologic and pathologic data combined with genomic data for ccRCC prognosis prediction, achieving similar results. However, the authors note that the Ning et al. study did not clarify how image features were selected and lacked a true external test set, whereas the present study addressed both of these limitations with the Mainz cohort and the CAM-based feature analysis.
Several limitations are acknowledged. The external validation cohort of 18 patients is small, and additional larger studies are needed to confirm generalizability. The study lacks a direct head-to-head comparison with established clinical tools such as IMDC and UISS scores, which incorporate clinical parameters (performance status, calcium levels) not included in the MMDLM. The genomic component was limited to the top ten mutations, which may not capture the full complexity of ccRCC's epigenetic and metabolic drivers. Furthermore, the CAM analysis was descriptive rather than systematic, and a comprehensive pathological evaluation of the features driving model predictions remains for future work.
Despite these limitations, the authors conclude that multimodal deep learning can meaningfully contribute to survival prediction in ccRCC and potentially improve clinical management. The ability to stratify patients into low- and high-risk groups based on routinely available clinical data could guide decisions about intensified treatment or surveillance strategies, bringing AI-driven prognosis tools closer to clinical practice.