Prostate cancer (PCa) affects approximately 1.1 million men annually and causes around 307,000 deaths worldwide each year. It is the most common cancer exclusive to males, making accurate diagnosis a major global health priority.
The standard diagnostic approach relies on the Gleason grading system, which rates tumor aggressiveness on a scale of 1 to 5 by examining the shape and differentiation of prostate gland tissue. Pathologists combine a primary and secondary Gleason score to determine overall cancer grade and staging from T1 to T4.
Traditional pathology requires a trained specialist to visually examine tissue slides under a microscope -- a process that is slow, subjective, and prone to inter-observer variability. Only a tiny fraction of cells on any given slide may be cancerous, making manual detection error-prone.
AI systems have already shown promise in related areas: DeepMind (Google) dramatically reduced breast cancer misdiagnosis rates, and companies like Novartis are partnering with AI firms to automate cancer pathology. This study explores how AI can fill a similar role for prostate cancer.
The study used a radiomics approach to extract quantitative features from Hematoxylin and Eosin (H&E)-stained histopathology images. H&E staining is the gold standard for tissue slides, highlighting nuclei (blue) and cytoplasm (pink) to reveal cellular architecture.
Images came from two datasets: a private set from Severance Hospital in Korea (500 patches from 20 patients) and a public set from Radboud University Medical Center (Netherlands) involving 40 patients. Tissue was scanned at 40x and 10x magnification respectively, then cropped into regions of interest (ROIs) for analysis.
A technique called stain deconvolution was used to separate the Hematoxylin and Eosin channels into two distinct image inputs -- the basis of the paper's dual-channel approach. Each channel captures different tissue information: Hematoxylin highlights nuclei while Eosin highlights connective tissue and cytoplasm.
From each channel, six first-order statistic (FOS) features were extracted per image patch: energy, entropy, skewness, kurtosis, variance, and uniformity. These mathematical descriptors capture pixel intensity distributions that correspond to tissue texture differences between benign and malignant tissue.
Gamma correction was applied to adjust image brightness and contrast before feature extraction. Patches were generated at 64x64 pixel size to increase the number of samples per class, which is important for training robust machine learning models.
Two feature selection steps were used before classification. First, Recursive Feature Elimination (RFE) -- a method that repeatedly trains a model and removes the weakest features -- identified the top 5 of 6 extracted features. A gradient boosting classifier was used as the baseline model for this process.
Second, a one-way ANOVA (analysis of variance) test confirmed statistical significance of the selected features by computing p-values and effect sizes (eta squared). All five retained features showed large effect sizes (eta squared greater than 0.14), confirming meaningful differences between benign and malignant tissue.
Five classification algorithms were evaluated: Support Vector Machine (SVM), Logistic Regression (LR), Bagging Tree, Boosting Tree, and the novel Dual-Channel Bidirectional Long Short-Term Memory (DC-BiLSTM) network. The first four are traditional machine learning methods; the last is a deep learning approach designed specifically for this dual-channel framework.
The DC-BiLSTM network used two parallel LSTM branches -- one per staining channel -- each with two BiLSTM layers of 64 nodes. Bidirectional LSTMs process input sequences both forwards and backwards, capturing contextual patterns that unidirectional models miss. The outputs of both channels were concatenated before a final classification layer.
For distinguishing benign from malignant tissue on the internal test set, the DC-BiLSTM model achieved 98.6% accuracy, 98.2% precision, 98.9% recall, and a 98.6% F1-score -- the best result of all five models. SVM and LR both reached 96.1% accuracy, while the Bagging Tree scored 95.6%.
For low-grade versus high-grade cancer classification (Grade 3 vs. Grade 5), DC-BiLSTM again led with 93.6% accuracy and 96.3% precision. This task is inherently harder because Grade 3 and Grade 5 tissues are more similar to each other than benign and malignant are -- and low-grade tissue can resemble benign tissue in texture.
On the external public dataset (from the Netherlands), the Boosting Tree classifier outperformed DC-BiLSTM, achieving 93.5% accuracy. This crossover in performance occurred because DC-BiLSTM was fine-tuned to the Korean private dataset, while tree-based methods generalized better to unseen data with different imaging protocols.
AUC (Area Under the ROC Curve) values of 1.00, 0.98, and 0.95 were recorded for the best performing models across the three classification tasks, confirming strong diagnostic discrimination between tissue types. Box plot analyses confirmed that the texture feature distributions were consistent between the internal and external datasets.
The AI system excelled at distinguishing malignant from benign tissue because their textures differ substantially -- malignant tissue contains irregular gland masses and sheets of cells, while benign tissue has small, uniformly shaped glands with defined stroma between them.
A key limitation was that some low-grade malignant samples (Grade 3) have texture patterns similar to benign tissue, causing occasional misclassification. This is a known challenge in prostate pathology and is one reason expert pathologists can also disagree on borderline cases.
The external dataset used a different magnification factor (10x vs 40x) and different spatial distributions, which explains why DC-BiLSTM performed slightly less well on it. This highlights the general challenge of domain shift -- models trained on one imaging setup may not transfer perfectly to another.
Compared to the team's previous study using second-order GLCM features and SVM, the current approach with FOS features and DC-BiLSTM showed substantial gains (e.g., SVM accuracy improved from ~84% to 96.1%). This demonstrates that combining dual-channel feature engineering with deep learning offers measurable performance improvements.
The study demonstrated that AI-based tissue analysis using dual-channel FOS radiomic features can classify prostate cancer with high accuracy, offering a potential tool to assist pathologists in routine diagnosis and reduce workload.
The two-step feature selection process (RFE followed by ANOVA) was validated as an effective strategy for identifying the most informative texture features without overfitting the model to noise in the data.
All five models -- SVM, LR, Bagging Tree, Boosting Tree, and DC-BiLSTM -- achieved high recall rates for benign vs. malignant classification, which is clinically important: missing a malignancy is a more dangerous error than a false positive.
Future work should validate these models across larger, more diverse histopathology datasets from multiple cancer types and imaging centers. The authors recommend incorporating additional preprocessing techniques such as image normalization to further improve cross-dataset generalizability.