High recurrence, high surveillance burden, high stakes. Non-muscle invasive bladder cancer (NMIBC) is one of the most expensive cancers per patient in oncology, driven primarily by its notoriously high recurrence rate and the lifelong cystoscopic surveillance it requires. Up to 70% of patients experience recurrence after initial treatment, and approximately 10-20% progress to muscle-invasive bladder cancer -- a development that dramatically worsens prognosis and typically requires radical bladder removal.
The standard treatment for intermediate- and high-risk NMIBC is intravesical BCG immunotherapy administered directly into the bladder. However, up to 40% of patients do not respond to BCG. For these BCG-unresponsive patients, and those who progress to muscle-invasive disease, aggressive treatment in the form of radical cystectomy becomes necessary -- a procedure carrying substantial morbidity and mortality. Early identification of which patients are at high risk of recurrence or progression could enable earlier treatment changes and better patient counseling.
Current risk stratification tools, including the widely used EORTC (European Organisation for Research and Treatment of Cancer) nomogram, are built on traditional statistical methods and clinical variables. Artificial intelligence approaches, which can integrate diverse data types and model complex non-linear relationships, have attracted substantial research interest as potentially superior alternatives. This systematic review applied the APPRAISE-AI quality assessment tool to critically evaluate the current state of that evidence.
A quantitative quality scorecard designed for AI studies. Standard quality assessment tools for medical research were designed for clinical trials and observational studies, not for AI models. They do not evaluate AI-specific methodological concerns such as data leakage (using test data during model training), hyperparameter tuning transparency, or model reproducibility. APPRAISE-AI was developed to fill this gap by providing a domain-specific scoring framework that evaluates both methodological conduct and reporting quality in AI studies.
The tool assesses studies across six domains: data quality, clinical relevance, methodological conduct, robustness of results, reporting quality, and reproducibility. Each domain contains specific items that are scored, with weights applied based on their clinical importance. Overall scores are categorized as: very low quality (0-19), low quality (20-39), moderate quality (40-59), high quality (60-79), and very high quality (80-100). This quantitative scoring enables direct comparison of study quality across a field.
For this systematic review, MEDLINE, EMBASE, Web of Science, and Scopus were searched from inception to February 2024, identifying 7,102 studies. After deduplication and screening, 15 studies met inclusion criteria -- five focused on recurrence prediction, four on progression prediction, and six addressing both outcomes. All included studies developed AI models to predict NMIBC outcomes from pathologically confirmed NMIBC cohorts, and only studies using AI for clinical decision support (not purely diagnostic imaging studies) were included.
A field still in early development. All 15 included studies were retrospective, and most were single-institution designs conducted at academic medical centers. The median cohort size was only 125 patients (IQR 93-309) -- considered small even for traditional regression-based prediction modeling, let alone deep learning algorithms that typically require larger datasets. Median follow-up was 71 months (IQR 32-93). Geographically, 60% of studies were from Europe, with the remainder from Asia and one from Africa, limiting representation of diverse healthcare settings and patient populations.
The most commonly used AI model types were neural networks, used in 11 of 15 studies (73%), including shallow neural networks, neuro-fuzzy models, deep belief networks, DeepSurv (a deep learning survival model), and convolutional neural networks. Three studies used support vector machines, and two incorporated radiomic features from imaging. Most models (10 of 15) were trained solely on clinicopathological features -- standard clinical variables like stage, grade, age, and tumor size. Six studies incorporated gene expression data and two used radiomic features.
Outcome definitions were strikingly inconsistent. Despite the review focusing on just two outcomes (recurrence and progression), 14 distinct definitions were identified across 15 studies. Recurrence definitions ranged from 'relapse of equivalent or lower stage' to 'any stage within two years' to 'Ta, T1, or CIS.' Progression was similarly variably defined -- some studies used T2 or higher stage, others used any stage or grade increase, and others combined staging changes with lymph node or metastatic disease. This heterogeneity makes comparing performance across studies unreliable.
High reporting scores, poor methodological quality. The median APPRAISE-AI overall score was 37 -- placing the typical study in the low quality category. Only one study across two decades of AI research in NMIBC was classified as high quality. Study quality improved significantly over time (regression coefficient 0.65, p = 0.03), with all studies from 2000 to 2010 rated as low quality except one moderate quality study, and three of seven studies from 2010 to 2022 rated low quality.
The two highest-scoring APPRAISE-AI domains were clinical relevance and reporting quality, while the three lowest were methodological conduct, robustness of results, and reproducibility. This pattern is informative: clinical relevance and reporting quality encompass elements that clinicians are familiar with and journals routinely require -- background sections, patient characteristics, limitations, and disclosures. Methodological conduct, robustness, and reproducibility require AI-specific practices that the medical community has not yet standardized in its reporting expectations.
Specific items with low scores (below 40% of maximum possible) included sample size calculation (reported in zero studies), hyperparameter tuning description, clinical utility assessment using decision curve analysis, bias assessment across demographic subgroups, error analysis, and model transparency. Only one study made its data and code publicly available in a repository that would allow independent replication of the findings. Only three studies described how missing data were handled.
AI wins most comparisons, but quality matters. Seven of 15 studies (47%) compared AI models against non-AI alternatives, including logistic regression, Cox regression, EORTC nomograms, and clinical expert judgment. In most comparisons, AI models outperformed non-AI approaches across accuracy, c-index, sensitivity, and specificity. AI c-index values had a median of 0.76 for both recurrence and progression prediction -- compared to the EORTC nomogram's documented AUC of approximately 0.60-0.65 for similar tasks.
However, the margin of AI benefit varied critically with study quality. Low-quality studies showed a median absolute performance difference of 10 percentage points in favor of AI. Moderate-quality studies showed a 22-point difference. The single high-quality study showed only a 4-point difference. This pattern suggests that poor methodology -- particularly data leakage and absence of external validation -- inflates AI performance estimates in low-quality studies. Studies with data leakage (training and testing on the same dataset without correction) reported median accuracy of 86% compared to 83% for studies without this concern, a gap attributable to overfitting.
Two studies found that non-AI approaches were superior for some performance metrics. One found that consultant urologists outperformed an artificial neural network for certain accuracy metrics, and another found Cox regression superior on specificity measures. These exceptions reinforce the conclusion that AI does not inherently outperform simpler approaches when the comparison is made fairly. The apparent advantage of AI narrows substantially in the most rigorous studies.
Systematic problems requiring systematic solutions. Data leakage -- using test data during model training without isolation -- is a fundamental methodological error that produces artificially optimistic performance estimates. Several studies trained and tested on identical datasets without applying correction methods, making their reported accuracy figures unreliable. Small sample sizes (median 125 patients) compound the problem, as models trained on small cohorts are inherently unstable and their performance varies substantially depending on which patients happen to be in the training versus test set.
Algorithmic bias -- differential performance across demographic subgroups including sex, race, and socioeconomic status -- was rarely addressed. Only two of 15 studies conducted any form of subgroup bias assessment. AI models trained on non-representative datasets may perform well on average while systematically failing specific patient groups, perpetuating rather than correcting existing healthcare disparities. This gap is particularly concerning given that several studies found AI models in other domains disproportionately affect marginalized patients.
Reproducibility represents the most severe gap. Only one study made its dataset and code publicly available, enabling independent replication. A technology that requires independent verification before clinical adoption cannot achieve that verification when its methods and data are inaccessible. The tumor staging and grading variability inherent to pathological assessment -- with interobserver kappa scores ranging from 0.003 to 0.70 for grading -- further complicates reproducibility, since the input labels on which AI models are trained are themselves inconsistent across institutions and pathologists.
Six areas requiring coordinated improvement. The review provides specific, actionable recommendations across six domains. For dataset quality: training cohorts should include all NMIBC stages, grades, and tumor histories including recurrent tumors; should not be restricted to academic institutions; and should reflect standard of care including repeat TURBT and intravesical BCG. Sample size calculations should be reported before model development, not ignored entirely as was the case in all 15 reviewed studies.
For outcome definitions: the International Bladder Cancer Group's standardized definitions should be adopted -- recurrence as any stage or grade relapse, and progression as T1 disease development or muscle invasion. Using these universal definitions would allow meaningful cross-study comparisons and meta-analysis. For methodology: testing cohorts must be isolated before any preprocessing steps including normalization and imputation to prevent data leakage; hyperparameter tuning procedures must be described transparently; and overfitting must be addressed through bootstrapping, cross-validation, or external validation in independent cohorts.
For evaluation and reproducibility: AI models must be compared against established clinical prediction tools such as the EAU NMIBC risk calculator, and performance must include not just accuracy and AUC but also calibration plots and decision curve analysis. Models and de-identified datasets should be shared in public repositories. Journals should recruit reviewers with AI expertise. The only study achieving high quality in this review -- Jobczyk et al., which provided a publicly accessible web application and a de-identified dataset -- demonstrates that these standards are achievable and should serve as a benchmark for the field.