The gold standard for diagnosing most cancers is a pathologist examining tissue under a microscope. AI systems for digital pathology have achieved impressive results on specific tasks -- detecting cancer, grading tumors, predicting outcomes -- but they are almost all trained separately, each requiring its own large dataset of expert-labeled examples. Scaling this approach to the thousands of possible diagnoses a pathologist might encounter is impractical.
A deeper problem is that existing pathology AI models rely exclusively on image data, ignoring how pathologists actually learn and communicate. Pathology education is inseparable from language: textbooks describe visual patterns with words, pathology reports use natural language to convey findings, and teaching happens through image-description pairs. AI that cannot connect images to language is fundamentally limited compared to how humans reason about pathology.
Foundation models are large AI systems pretrained on massive datasets that can then be adapted to many downstream tasks without being retrained from scratch. In general AI, tools like CLIP and DALL-E showed that training on image-text pairs produces models that can classify, retrieve, and reason across both modalities. Building a pathology-specific equivalent requires a large, domain-specific dataset of paired pathology images and descriptions -- which did not previously exist at scale.
This paper introduces CONCH (CONtrastive learning from Captions for Histopathology), a visual-language foundation model pretrained on over 1.17 million histopathology image-caption pairs from biomedical literature and educational resources, designed to work across many pathology tasks without task-specific retraining.
The training dataset was assembled from two sources: educational pathology materials and the PubMed Central Open Access collection of biomedical research articles. An automated pipeline using object detection, AI-based caption splitting, and CLIP-based image-text alignment was used to extract and clean over 1.17 million human histopathology image-caption pairs -- the largest such dataset constructed for pathology to date.
CONCH's architecture consists of three components: an image encoder (a Vision Transformer that converts slide images into numerical representations), a text encoder (a language model trained on pathology reports and literature to understand pathology terminology), and a multimodal decoder that can generate text descriptions of images. This design is based on CoCa, a state-of-the-art general-purpose visual-language framework.
The model is trained with two complementary objectives. Contrastive learning pushes the image and text representations of matching pairs together in a shared numerical space while separating non-matching pairs. A captioning objective trains the model to generate accurate text descriptions of images, further reinforcing the connection between visual patterns and language.
Before joint training, both the image encoder and text encoder were pretrained separately. The image encoder was pretrained on 16 million unlabeled pathology image tiles from over 350 cancer subtypes. The text encoder was trained on over 550,000 pathology reports from Massachusetts General Hospital and 400,000 pathology-relevant research abstracts, giving it deep domain-specific language knowledge.
Zero-shot classification means the model can categorize a new image using only a text description of the category -- without ever having seen labeled examples for that specific task. This is analogous to a pathologist being able to recognize a cancer subtype after reading a description of it, rather than having seen thousands of labeled slides first.
On four slide-level cancer classification tasks, CONCH outperformed all competing models. For non-small-cell lung cancer (NSCLC) subtyping, it achieved 90.7% zero-shot accuracy -- 12% better than the next-best model. For breast cancer (BRCA) subtyping, CONCH achieved 91.3% while competing models essentially guessed at random (50-55% accuracy, near chance level).
For prostate cancer Gleason pattern classification -- directly relevant to staging prostate cancer aggressiveness -- CONCH achieved a zero-shot quadratic kappa score of 0.690, outperforming BiomedCLIP by 0.140. This means CONCH could correctly stratify prostate tissue into cancer grades without any task-specific training examples, simply by being shown text descriptions of each Gleason pattern.
CONCH also demonstrated the ability to generate visual heatmaps that highlight which regions of a slide were most influential in a classification decision. These heatmaps aligned closely with pathologist-annotated tumor regions, providing interpretability that is critical for building clinical trust in AI systems.
In most clinical and research scenarios, some labeled examples will be available, but rarely in large quantities. Few-shot learning tests whether a model can reach high accuracy with very few labeled training examples. CONCH showed exceptional label efficiency: using only 8 labeled examples per cancer class, CONCH outperformed competing models trained on 64 labeled examples per class -- an 8-fold reduction in annotation burden.
Rare diseases are particularly important because they are, by definition, underrepresented in training datasets. CONCH was evaluated on a 30-category brain tumor classification task using only rare cancer types. In zero-shot mode, CONCH achieved 37.1% accuracy on this challenging problem (random chance would be 3.3%), and when combined with weakly supervised learning on embeddings, it achieved 68.2% -- significantly outperforming all other encoders tested.
For rare diseases, other models required roughly four times as many labeled training examples to match CONCH's performance. This is especially significant because collecting expert pathology annotations for rare diseases is extremely costly -- any reduction in labeling requirements has direct practical value for developing AI tools for uncommon cancers.
Cross-modal retrieval -- finding the right image given a text query, or the right description given an image -- is directly useful in clinical and research contexts. A pathologist could type a description like 'cribriform prostatic adenocarcinoma' and retrieve visually matching cases from a database of millions of slides, assisting with diagnosis or education. CONCH achieved a text-to-image mean recall of 44.0%, outperforming the next-best model by 17.3%.
The paper demonstrated that CONCH can retrieve images matching complex pathology descriptions -- including prostate cancer patterns -- even though these descriptions were not part of evaluation datasets during training. A pathologist confirmed that retrieved images visually matched complex query terms, validating the model's ability to connect nuanced pathology language to visual tissue patterns.
Zero-shot tissue segmentation -- identifying cancer versus normal tissue at the pixel level without any segmentation-labeled training data -- is another breakthrough capability. On prostate cancer tissue segmentation (SICAP dataset), CONCH achieved a Dice score of 0.601, meaningfully outperforming competing models. On colorectal cancer tissue segmentation, it achieved 0.615. These results suggest CONCH can localize tumors within slides without pixel-level annotations, which are among the most expensive type of pathology label to obtain.
Prostate cancer Gleason grading is one of the most important -- and most labor-intensive -- pathology tasks. The researchers assembled a benchmark of over 228,000 labeled prostate tissue image patches from multiple public datasets, covering four categories: non-cancerous (NC), Gleason 3, Gleason 4, and Gleason 5 patterns.
When CONCH was fine-tuned on this dataset end-to-end, it outperformed all competing models, including larger Vision Transformer models, domain-specific encoders like CTransPath and KimiaNet, and standard CNNs. Notably, CONCH outperformed a ViT-Large model that has 3.5 times more parameters, demonstrating that better pretraining can compensate for smaller model size.
Even when the training data was dramatically reduced to 10% or 1% of labels, CONCH maintained its advantage over other models, reinforcing its superior label efficiency for prostate cancer grading specifically. This is particularly meaningful because Gleason grading currently suffers from high interobserver variability -- a consistent, objective AI tool for grading could directly reduce inconsistency in prostate cancer staging and treatment decisions.
Despite strong results, CONCH has limitations. Its 1.17 million training pairs, while the largest in histopathology, pale in comparison to billion-scale datasets used in general AI. Zero-shot performance on complex tasks involving many rare classes remains limited. The model has not been tested for robustness across different staining protocols, scanner types, or tissue preparation methods -- all of which vary between hospitals.
The model's current capabilities are primarily at the image tile level rather than individual cell or subcellular resolution. Tasks like mitosis counting, fine-grained cell segmentation, or organelle-level analysis currently remain outside its scope. These finer-grained tasks are important for complete pathology workflows and represent a direction for future development.
The long-term clinical vision for CONCH and similar foundation models is a single, generalizable AI system that can assist pathologists across the full breadth of their work -- classifying cancer types, retrieving similar cases, generating draft reports, localizing tumors, and supporting rare disease diagnosis -- all from a shared model that understands both images and language the way pathologists do.