Prostate cancer is the most common non-skin cancer in American men and the second leading cause of cancer death. Despite its prevalence, many men remain unaware of their screening options, risk factors, and the nuanced tradeoffs involved in deciding whether to pursue PSA testing -- particularly the risk of overdiagnosis and overtreatment of slow-growing tumors that may never become life-threatening.
Publicly available information about prostate cancer screening varies widely in accuracy and accessibility. Patients searching general health websites often encounter outdated content, conflicting guidelines, or text written at reading levels beyond what many adults can comfortably process. This creates a knowledge gap at exactly the moment when men need reliable guidance to engage in informed conversations with their doctors.
Large language models (LLMs) such as GPT-4.0 offer a potential solution: AI chatbots that can answer direct questions in plain, conversational language, available around the clock without appointment delays. However, prior research on commercial chatbots like ChatGPT has found accuracy rates as low as 74% and completeness rates around 63% for prostate cancer topics, underscoring the need for purpose-built, guideline-anchored tools.
This study evaluates Prostate Cancer Info (PCI), a custom GPT chatbot built on OpenAI's GPT-4.0 platform and restricted to four authoritative sources -- the American Cancer Society (ACS), American Urological Association (AUA), US Preventive Services Task Force (USPSTF), and Centers for Disease Control (CDC). By constraining the chatbot to trusted, current guidelines, the developers aimed to build a tool that is both accurate and safe for patient-facing use.
The research team developed Prostate Cancer Info (PCI) using OpenAI's GPT Builder platform, which allows creation of customized chatbots restricted to specified knowledge sources. PCI's knowledge base was limited to documents from ACS, AUA, USPSTF, and CDC -- four organizations that publish nationally recognized, evidence-based guidelines for prostate cancer screening and management.
To evaluate PCI, the team compiled 23 frequently asked questions (FAQs) about prostate cancer screening drawn from the same four guideline sources. These questions covered a broad range of topics including PSA testing thresholds, risk stratification by age and race, digital rectal examination, shared decision-making, follow-up after abnormal results, and the potential harms of overdiagnosis.
Two board-certified urologists independently reviewed PCI's answers to all 23 questions and scored each response on three criteria: accuracy (is the answer factually correct?), completeness (does the answer include all relevant information from the guidelines?), and readability (measured using the Flesch Reading Ease scale, where higher scores indicate easier text and a score of 60-70 corresponds roughly to an 8th-grade level).
Inter-rater reliability between the two urologists was measured using Cohen's kappa, a statistical measure of agreement that accounts for chance. The study was designed as a cross-sectional evaluation, meaning each question was asked once and scored once, providing a snapshot of PCI's performance under typical patient-use conditions.
PCI answered all 23 questions with 100% accuracy -- every response was factually correct according to the reviewing urologists. This represents a meaningful improvement over commercial chatbots previously evaluated for prostate cancer screening information, which achieved accuracy rates as low as 74% to 89%. The restriction to curated, authoritative guideline documents appears to have effectively eliminated hallucination and factual errors.
On completeness, PCI scored 74% -- providing fully complete answers to 17 of the 23 questions. The six incomplete responses shared a common pattern: they correctly addressed the primary question but omitted secondary details such as statistical probabilities associated with specific PSA levels, geographic and regional risk factors, and nuances around timing of treatment decisions after diagnosis. These gaps were consistent with the boundaries of what the restricted source documents explicitly covered.
The mean Flesch Reading Ease score was 64.5 (standard deviation 8.7), placing PCI's responses at approximately an 8th-grade reading level. This is slightly above the ACS-recommended 6th-grade target for patient health materials but substantially better than the 12th-grade level reported for general-purpose ChatGPT outputs. Inter-rater reliability between the two urologists was strong, with Cohen's kappa values indicating good to excellent agreement on both accuracy and completeness assessments.
Incomplete answers fell into identifiable categories. Three questions about PSA level thresholds lacked probabilistic information (e.g., the likelihood that a given PSA value corresponds to cancer). One question about risk factors was missing data on geographic variation in prostate cancer incidence. One question on treatment timing lacked detail about the window between diagnosis and initiation of active treatment, and one question about shared decision-making for African-American men was incomplete regarding culturally specific risk elevation messaging.
The 100% accuracy rate demonstrates that constraining an LLM to curated, authoritative sources effectively eliminates the hallucination problem that plagues general-purpose chatbots in medical contexts. When PCI could not find information within its restricted knowledge base, it acknowledged the limitation rather than generating plausible-sounding but unsupported answers -- a behavior that the researchers highlight as a key safety advantage for patient-facing deployment.
The 74% completeness rate reflects an inherent limitation of the restricted-source approach: the chatbot can only be as complete as the documents it has access to. Some guideline documents do not include probabilistic statistics or geographic risk data, meaning PCI cannot provide that information even when it would be clinically relevant. This points to the importance of careful source selection and ongoing curation as guidelines are updated and expanded.
The study authors note that PCI's 8th-grade readability level, while better than commercial alternatives, still falls short of the recommended 6th-grade threshold for patient education materials. This gap has equity implications: many prostate cancer patients, particularly older men or those from lower-literacy backgrounds, may still struggle with the language level. The authors suggest that readability prompting -- instructing the model to simplify its outputs -- could close this gap without sacrificing accuracy.
Compared to other chatbots evaluated in published literature, PCI performs favorably. Earlier studies of commercial ChatGPT versions for prostate cancer screening reported accuracy between 74% and 89% and completeness around 63%. PCI's 100% accuracy and 74% completeness, achieved without any additional prompt engineering beyond source restriction, suggest that the custom GPT approach has meaningful clinical advantages over deploying off-the-shelf language models.
The study envisions PCI being deployed as a pre-appointment educational tool. The proposed workflow involves sending patients a QR code via email or SMS before their urology appointment, which opens PCI directly in a mobile browser. Patients could then ask their own questions in natural language, arriving at the appointment with a baseline understanding of PSA testing, risk categories, and the shared decision-making process -- reducing the burden on clinic staff and improving the quality of the patient-physician conversation.
This model addresses a recognized problem in prostate cancer care: many men receive a PSA result without adequate context for interpreting it. They arrive at specialist appointments without knowing what questions to ask, which leads to rushed consultations, poor shared decision-making, and anxiety that could have been managed with earlier education. A chatbot that can answer common questions at any hour -- in plain language, grounded in guidelines -- fills a real gap in the care continuum.
The authors also propose a feedback loop for ongoing improvement: tracking which questions users ask most frequently, flagging responses that generate follow-up clarification requests, and using these signals to identify gaps in PCI's knowledge base. This kind of iterative refinement could allow the tool to improve over time as new guidelines are published and as real-world usage reveals edge cases not anticipated during development.
Limitations of the study include its cross-sectional design (questions were evaluated once, not across multiple use sessions), the use of only urologist evaluators rather than patients themselves, and the restriction to English-language content. Future research should assess PCI's performance with diverse patient populations, test its effectiveness in improving patient knowledge retention, and evaluate whether pre-appointment use of the tool reduces consultation length or improves patient satisfaction scores.
This study contributes evidence that custom, guideline-restricted GPT tools represent a more reliable approach to AI-assisted patient education than general-purpose commercial chatbots. The key design decision -- limiting PCI's knowledge base to four authoritative sources -- appears to be the primary driver of its superior accuracy. This approach is replicable for other cancer types or clinical domains where high-quality guideline documents exist.
The practical implications for healthcare organizations are significant. Building a custom GPT via OpenAI's GPT Builder requires no specialized machine learning expertise and can be accomplished with access to PDF or web versions of relevant guidelines. The development investment is low compared to the potential benefit of providing patients with round-the-clock, accurate, guideline-aligned answers to their most common questions.
However, the authors caution that accuracy alone is not sufficient for clinical deployment. Completeness gaps, readability above recommended levels, and the absence of patient-centered evaluation are real limitations. Before broad clinical adoption, PCI or similar tools should be validated in prospective studies with actual prostate cancer patients, measuring not just information quality but downstream outcomes -- including whether chatbot use improves informed consent, reduces decision conflict, or changes screening uptake patterns.
As AI tools continue to evolve, the standard of comparison will shift. GPT-4.0 represents the current state of the technology, but newer models will offer improved reasoning, multilingual support, and better calibration to clinical complexity. The evaluation framework developed in this study -- accuracy, completeness, and readability scored against guideline standards -- provides a transferable methodology for benchmarking future tools as they emerge.