Following surgery to remove the prostate (radical prostatectomy), roughly 70% of patients are cured, but about 30% experience biochemical recurrence (BCR) -- a rise in PSA blood levels indicating the cancer has returned. BCR is the earliest warning sign that the tumor was not fully eliminated by surgery.
Current methods for predicting who will recur rely on clinico-pathological variables such as Gleason score, tumor stage (TNM), and pre-surgery PSA levels. While these factors provide meaningful risk estimates, they lack the precision needed to reliably distinguish individual patients who will recur from those who will not, especially in intermediate-grade disease.
If a patient's tumor could be identified before or shortly after surgery as likely to recur, doctors could offer adjuvant treatments earlier -- such as radiation or hormonal therapy -- potentially preventing the cancer from returning. Conversely, correctly identifying patients who are truly cured avoids unnecessary treatment with its associated side effects and costs.
The falling cost of RNA sequencing (RNA-seq) technology -- which measures the activity levels of thousands of genes simultaneously in a tumor sample -- has opened a new path for more precise prognosis. Gene expression patterns in the tumor may capture biological aggressiveness that the microscope and PSA tests cannot detect.
A central challenge in cancer genomics is that high-quality RNA-seq datasets with long patient follow-up are small and rare -- no single institution typically has hundreds of carefully followed patients with complete molecular data. This study addressed this by combining three independent public datasets: the TCGA-PRAD cohort (Cancer Genome Atlas), GSE54460, and a cohort from the Vancouver Prostate Cancer Centre (VPCC).
After applying strict quality filters -- removing samples with RNA degradation, low tumor cell content, or follow-up shorter than 60 months (to avoid misclassifying future recurrences as non-recurrences) -- the combined dataset comprised 171 patients, of whom 98 experienced BCR. This is a modest but carefully curated cohort.
A key technical challenge was batch effects -- systematic differences between datasets caused by different laboratories, sequencing machines, and sample preparation protocols. The authors used a normalization method based on housekeeping genes (genes known to be stably expressed in all tissues regardless of cancer status) to remove these technical biases while preserving genuine biological differences.
All three datasets were reprocessed through the same bioinformatics pipeline from raw sequencing data, ensuring consistent read quality filtering, genome alignment, and gene counting. This uniformity is essential for machine learning analysis -- inconsistent preprocessing across datasets would introduce noise that could confound the model.
The study tested 14 different machine learning classification algorithms, including linear discriminant analysis, support vector machines, gradient boosting, naive Bayes, k-nearest neighbors, and random forest, among others. Each was evaluated for its ability to predict BCR using gene expression features, with performance measured by the balanced error rate (BER) -- a metric that accounts for class imbalance (more BCR patients than non-BCR).
Feature selection was performed using information gain ranking, an entropy-based method that scores each gene by how much knowledge of its expression level reduces uncertainty about whether a patient will have BCR. The top 400 genes were retained, then the number was progressively reduced while monitoring performance to find the minimum gene set needed.
Model validation used a rigorous subsampling approach: the dataset was randomly split into 2/3 training and 1/3 test sets, repeated 1,000 times. This avoids overfitting and provides a reliable estimate of how the model would perform on genuinely new patients. The BER and other metrics were averaged across all 1,000 splits.
Random forest performed best among all tested algorithms. Random forest builds hundreds of decision trees, each trained on a random subset of patients and genes, then combines their votes for the final prediction. This ensemble approach is robust to noisy features and tends to generalize well on small biological datasets.
The analysis converged on a three-gene signature -- JUN, HES4, and PPDPF -- as the minimum gene set achieving near-optimal predictive performance. Adding more genes beyond these three yielded minimal improvement in accuracy, confirming that these three genes together capture most of the prognostically relevant information in the dataset.
JUN is a well-established oncogene and transcription factor. JUN proteins combine with FOS proteins to form the AP-1 complex, which binds DNA and activates genes involved in cell growth, proliferation, and survival in response to growth factors and cytokines. Dysregulated JUN activity is linked to tumor invasion and metastasis in multiple cancer types.
HES4 is connected to the NOTCH3 receptor and the PI3K-AKT signaling pathway -- one of the most commonly activated survival pathways in cancer. HES4 has previously been described as a biomarker of prostate cancer aggressiveness and has been linked to tumor progression in colorectal cancer and osteosarcoma.
PPDPF (Pancreatic Progenitor Cell Differentiation and Proliferation Factor) is expressed during organ development and has been found differentially expressed in several cancers including pancreatic and others, but had not previously been linked to prostate cancer. Its inclusion in this signature represents a novel finding that warrants further mechanistic investigation.
The three-gene model achieved an overall accuracy of 74.2% with a balanced error rate of 27% and an AUC of 0.761 -- better than the clinical model using Gleason score, tumor stage, and PSA, which achieved 69.2% accuracy with a BER of 32%.
Interestingly, combining the three-gene signature with clinical variables (a merged model) did not substantially improve performance over the gene model alone (BER 27.6% vs. 27%). This suggests that the gene expression signature captures most of the clinically relevant prognostic information already captured by conventional variables, plus additional independent signal.
When the analysis was repeated on each cohort individually (training on two datasets, testing on the third), performance was lower and more variable than when all three datasets were combined. This directly demonstrates the key methodological point: combining small datasets produces a more reliable and generalizable model than any single cohort alone.
The performance metrics (AUC 0.76, accuracy 74%) are competitive with published BCR prediction models trained on much larger single cohorts, validating the approach of pooling heterogeneous datasets with appropriate normalization rather than waiting for single institutions to accumulate large patient numbers.
This study provides a practical blueprint for using machine learning to extract reliable prognostic signatures from the combined analysis of multiple small RNA-seq datasets -- a realistic and scalable alternative to waiting for single large cohorts to accumulate.
The key methodological contributions are: (1) harmonizing multi-institution datasets through uniform reprocessing and housekeeping-gene normalization; (2) testing a broad panel of classifiers to objectively identify the best model; and (3) using subsampling-based validation to generate unbiased performance estimates on small datasets.
The three-gene signature (JUN, HES4, PPDPF) is short enough to be potentially translated into a practical clinical test -- short gene panels are far cheaper, faster, and more reproducible to measure than multi-hundred-gene arrays, which is essential for routine clinical deployment.
Future work should validate this signature prospectively in larger independent cohorts with longer follow-up, and investigate whether the signature retains predictive value when measured from simpler sample types such as needle biopsies rather than radical prostatectomy specimens. Mechanistic studies into how JUN, HES4, and PPDPF co-regulate cancer aggressiveness would also help clarify the biology underlying this predictive signal.