Radiation therapy for prostate cancer has become increasingly automated, with AI systems now able to draw organ boundaries on CT scans and generate full treatment plans. These tools save time and reduce variability, but they also introduce new challenges around safety and quality assurance.
Most published studies evaluate AI performance against known correct answers in controlled research settings. In real clinical practice, however, there is no ready-made ground truth to compare against -- radiologists modify AI outputs based on clinical judgment, and the original AI output is not formally rated against a gold standard every time.
This creates a gap: once an AI model is deployed, how do you know if it is performing well on new patients? How do you catch the cases where the AI made a subtle error that could affect radiation dose to healthy tissue? This study was designed to fill that gap by developing practical, automated monitoring tools.
Stereotactic body radiation therapy (SBRT) delivers very high doses of radiation to the prostate in just five treatment sessions, compared to conventional radiotherapy which may require up to 40 sessions. This efficiency comes with a strict requirement for precision -- errors in either tumor targeting or organ boundary drawing can result in serious harm to nearby structures.
Two AI tools were studied: a Deep Learning Segmentation (DLS) model that automatically draws the boundaries of the prostate, bladder, and rectum on CT scans, and a Deep Learning Planning (DLP) model that automatically generates the radiation treatment plan from those boundaries. Both models were commercially available and already in clinical use at the study institution.
Because SBRT requires especially stringent quality assurance, small errors in AI-drawn contours near the tumor can result in meaningfully different radiation doses to sensitive organs. A monitoring system that detects these cases automatically -- before treatment is delivered -- would substantially improve patient safety.
The study introduced a new geometric metric called the Local Surface Dice, which evaluates contour similarity only within two centimeters of the tumor target volume. Standard metrics like the Surface Dice assess the entire organ boundary, meaning small but clinically critical errors near the tumor can be diluted by accurate performance elsewhere on the organ surface.
Alongside this geometric metric, the team developed an Absolute Dose Difference (ADD) measure that quantifies how much the radiation dose reported at specific critical tissue volumes would change depending on whether the AI-drawn or clinician-corrected contour is used. This directly links contour errors to their potential impact on treatment.
By combining these two approaches, cases could be classified into four zones: Zone 1 (AI contour essentially correct, small dose impact), Zone 2 (minor geometric error but larger dose impact near high-dose regions), Zone 3 (larger geometric deviation but less dose impact), and Zone 4 (both geometric and dose errors significant). This framework guides clinical review and flags cases for model retraining.
Fifty prostate cancer patients treated at the Iridium Netwerk in Belgium were analyzed. All patients received SBRT under the PACE Trial protocol, with a dose of 7.25 Gy per fraction over five fractions targeting the prostate gland. CT simulation was performed on Philips or Siemens CT scanners and all planning was done in the RayStation treatment planning system.
For each patient, the AI-generated contours were reviewed and corrected by a radiation oncologist before treatment. This created a paired dataset -- the original AI output and the clinically approved version -- which could be systematically compared using the new monitoring metrics.
The AI planning model was originally trained on data from a Canadian hospital using a slightly larger tumor margin than was used at the Belgian study site. This deliberate mismatch allowed the team to test whether the monitoring system could detect cases where AI model behavior was influenced by differences between the training and current clinical populations -- a common real-world challenge called out-of-distribution (OOD) detection.
For the rectum (anorectum), the Local Surface Dice scored 0.93 on average, significantly lower than the Standard Surface Dice score of 0.98 (p = 0.006). This confirms that the localized metric catches errors that the standard metric misses -- cases where the AI drew the organ boundary accurately overall but made clinically meaningful errors in the region closest to the radiation target.
The Absolute Dose Difference for the rectum at the most commonly used dose-constraint volume (D1cm3) averaged 0.9 Gy, with a standard deviation of 0.8 Gy. This means that in some patients, using the AI contour rather than the corrected contour would have caused the reported maximum rectal dose to differ by nearly 2 Gy -- a meaningful difference in a high-dose SBRT context.
The zone classification system effectively separated patients who needed only minor corrections (zone 1) from those with contour changes that significantly affected the dose metrics (zones 2 and 4). This provides a ranked priority list of cases for clinical review, focusing attention on the patients where AI errors matter most rather than those with minor cosmetic contour differences.
The AI planning model fulfilled all mandatory clinical dose goals in 90% of the 50 cases -- an impressive baseline performance. However, when stricter optimal dose constraints were evaluated, the AI plan achieved the targets in only 32% of cases, compared to 74% for clinician-optimized plans. This demonstrates a meaningful gap in plan quality that routine review must address.
A key finding was that the cases where the AI plan failed were predictable from geometry. Patients with larger organ volumes or greater overlap between the prostate target volume and nearby organs consistently produced lower-quality AI plans. When the bladder or rectum extended significantly into the treatment region, the optimization problem became too complex for the AI to solve without human guidance.
The monitoring system also detected a systematic difference between the training and clinical populations: the average prostate volume was significantly smaller in clinical patients than in the training data (120 vs. 154 cm3, p less than 0.001), and bladder and rectum volumes were also different. Cases falling outside the statistical range of the training data correlated with cases where the AI plan required the most modification, suggesting that volume-based monitoring can predict which patients need closer attention before the plan is even reviewed.
The study confirms that no single metric can adequately monitor AI performance in a complex clinical setting. Geometric metrics alone can show that a contour is slightly different without revealing whether that difference matters for treatment. Dose-volume metrics alone can flag dose changes without indicating where the contour error occurred or how large it was.
Together, the Local Surface Dice and Absolute Dose Difference create a complementary system. The Local Dice catches large geometric deviations close to the target (zones 3 and 4), while ADD is sensitive to small geometric changes in regions receiving the highest doses (zone 2). Neither metric alone would detect all clinically relevant cases.
An important insight from the dose analysis is that the critical volume threshold matters. Metrics evaluated at very small volumes (0.03 cm3) were insensitive to contour shifts in homogeneous dose regions, while metrics at very large volumes averaged out local errors. The study recommends using a minimum critical volume of 1 cm3 for reliable detection of AI segmentation errors in dose-volume analysis.
One of the most practical contributions of this work is the identification of which cases are most valuable for model retraining. Cases in zone 4 (large geometric error, large dose impact) and those where organ volumes exceed the training data distribution are the best candidates to add to the AI training dataset, as they represent the failure modes most likely to improve model robustness when addressed.
This approach connects monitoring to what the authors call active learning -- a strategy where the AI system itself helps identify the cases from which it would benefit most from additional human annotation. Rather than labeling data randomly or in chronological order, active learning targets the cases most likely to improve future model performance.
The framework developed here is not specific to prostate cancer or SBRT. The general principle -- combining geometric metrics focused on the target vicinity with dose-based impact metrics -- could be applied to AI-assisted radiotherapy planning for any tumor type, providing a reusable blueprint for clinical AI quality assurance programs.