Non-Muscle Invasive Bladder Cancer and the Recurrence Challenge. About 70% of bladder cancers are non-muscle invasive (NMIBC) at the time of initial diagnosis. While NMIBC generally has a favorable prognosis, intravesical recurrence after transurethral resection of the bladder tumor (TURBT) occurs in 30-50% of patients -- a major clinical challenge.
Bacillus Calmette-Guerin (BCG) therapy is recommended for high- and intermediate-risk patients to reduce recurrence, but BCG carries significant side effects including hematuria, fever, and pain. Determining which patients truly need BCG requires accurate recurrence risk prediction.
Limitations of Current Risk Classification. Existing risk systems from the American Urological Association, European Association of Urology, and others classify patients using clinical and pathological factors such as tumor number, size, recurrence history, invasion depth, CIS presence, and grade. Despite these systems, many patients still relapse within 2 years of initial TURBT, indicating that conventional classifications are incomplete and a new prediction approach is needed.
Why Nuclear Features Specifically. TURBT specimens pose a unique challenge: unlike surgical resection specimens, TURBT tissue is fragmented and often thermally damaged (cautery effect), making structural atypia and invasive morphology difficult to assess reliably. The researchers therefore deliberately focused only on nuclear atypia -- the morphology of individual cell nuclei -- which is not affected by sampling conditions in the same way.
125 patients diagnosed with NMIBC at Tokyo Medical University Hospital between 2012 and 2019 were included. All had undergone TURBT with standard follow-up (cystoscopy and urinary cytology every 3 months for 2 years). 45 patients relapsed within 2 years and 80 did not. Data from 95 patients were used for training and 30 for testing.
Image Processing Pipeline. Hematoxylin and eosin-stained slides were scanned at 20x magnification. Regions of interest (ROIs) with viable tumor cells and no artifacts were manually selected by a pathologist -- an average of 7 ROIs per case, totaling 877 ROI images of 2048 x 2048 pixels (1 mm2 each). The software Ilastik automatically extracted nuclei from each ROI, and YOLOv3 (a deep learning object detection system) was used to separate touching nuclei, enabling precise individual nucleus measurement. A total of 1,008,502 nuclei were segmented across all cases.
CellProfiler then measured 79 quantitative morphological features for each nucleus, divided into 27 shape-related features (size, contour length, major and minor axis length, roundness, solidity, eccentricity, orientation) and 52 intranuclear texture features (angular moment, homogeneity, entropy, and others based on Haralick gray-level co-occurrence matrix). A cell feature level co-occurrence matrix (CFLCM) method captured nuclear heterogeneity and pleomorphism across each ROI, generating 960 features per ROI for machine learning input.
Two Machine Learning Algorithms Compared. Support Vector Machine (SVM) and Random Forest (RF) were selected as the machine learning algorithms because they perform well with relatively small sample sizes. Both are classification methods well suited to high-dimensional feature sets like the 960 nuclear features generated per ROI.
The primary outcome was binary: recurrence within 2 years versus no recurrence within 2 years. For each case (patient), predictions from all ROI images of that patient were aggregated by averaging recurrence probabilities, providing a single case-level prediction rather than relying on any single tissue region.
Avoiding Overfitting. With only 125 patients, overfitting was a real concern. The team addressed this by using a held-out test set (30 patients, approximately 25% of the total), and for the RF model, out-of-bag error estimation provided an additional internal validation of model performance without requiring additional data.
SVM Model Performance. The SVM model achieved 100% accuracy on the training set and 83.8% accuracy on ROI-level classification in the test set. When ROI-level predictions were aggregated to case-level predictions (averaging across all ROIs per patient), the accuracy improved to 90%. Only 3 of 30 test patients were misclassified.
The RF model similarly achieved 100% training accuracy, with 74.9% ROI-level test accuracy and 86.7% case-level accuracy. The out-of-bag error rate of 15.6% confirmed the validity of the RF model. There were 4 misclassifications, 3 of which were the same cases misclassified by SVM.
The Misclassifications Were Revealing. Almost all errors in both models were cases where a patient with no recurrence within 2 years was predicted as recurring. Notably, 3 of these 4 misclassified 'non-recurrence' patients actually experienced recurrence at 3 years or 41 months after TURBT -- suggesting the AI may have correctly identified high-risk nuclear features in patients who went on to recur later, rather than making true errors.
The nuclear morphological features most important for recurrence prediction in the SVM model included measures of nuclear shape, size, and chromatin texture. The consistency of results between SVM and RF -- two methodologically distinct algorithms -- supports the conclusion that nuclear morphological features genuinely carry predictive information about recurrence risk.
A New Dimension for Risk Assessment. This is the first study to use AI on digital pathology images specifically to predict prognosis in NMIBC. The approach works from the initial TURBT specimen alone, requires no additional tissue, and provides risk information that is entirely orthogonal to current clinical risk classification systems -- based on morphology rather than clinical history or tumor characteristics.
The model could function as an auxiliary tool alongside current AUA or EAU risk classifications. Conventional classification systems struggle most with the intermediate-risk category, where tumor heterogeneity makes treatment decisions difficult. Nuclear feature analysis could help stratify patients within this ambiguous group to better guide BCG treatment decisions.
Practical Advantages of the Nuclear Feature Approach. By relying only on cell nuclei rather than tissue architecture, the method avoids the sampling artifacts inherent to TURBT -- charred tissue edges, fragmented specimens, and crushed cells -- that make structural assessment unreliable. This makes the approach applicable to the very specimens that are routinely collected in clinical practice.
Future directions include expanding the method to predict progression to muscle-invasive disease, predicting response to intravesical chemotherapy or BCG, and assessing long-term prognosis. Limitations include the small sample size, manual ROI selection (which introduces pathologist variability), the exclusion of BCG maintenance patients, and the need for external validation at other institutions.
Nuclear Atypia as a Standalone Predictor. This study demonstrates that quantitative nuclear morphological information extracted from routine H&E-stained TURBT specimens is sufficient to predict recurrence within 2 years with 90% accuracy. The result is achieved without any clinical information about tumor stage, grade, or treatment history -- purely from morphological features invisible to the human eye.
The concordance between SVM and RF models strengthens confidence that the predictive signal is real and robust across different algorithmic approaches. The method is scalable, objective, and independent of pathologist experience, offering a new standardized tool for NMIBC management.
Broader Implications for AI Pathology. The researchers expect this nuclear feature extraction methodology to be applicable beyond NMIBC to other cancers and diseases where quantitative pathology can add prognostic value. The work represents a proof of concept for AI-driven nuclear atypia analysis as a clinical decision support tool in urological oncology.