Neoadjuvant chemotherapy (NAC) is chemotherapy given before surgery, with the goal of shrinking the tumor to allow better surgical outcomes, reduce cancer spread, and identify which drug regimens are effective for a given patient. While NAC offers substantial benefits for locally advanced breast cancer, patients who do not respond to the drugs experience unnecessary side effects from chemotherapy that ultimately does not help them.
Predicting NAC response before treatment begins from pre-treatment needle biopsy tissue is the clinical problem this study addresses. If AI can accurately predict which patients will achieve a strong response, oncologists can make more confident treatment decisions and spare non-responders from ineffective chemotherapy, advancing the goal of personalized medicine in breast cancer.
The key innovation in this study is a multi-model pipeline that combines three independent AI models: a convolutional neural network (CNN) that analyzes tissue structural patterns, a support vector machine (SVM) that analyzes fine-grained nuclear features, and a random forest (RF) that provides nonlinear classification as a fallback. Each model targets different aspects of cancer atypia that are biologically linked to chemotherapy sensitivity.
Tested on 103 unseen cases from three institutions across Japan and the United States, the integrated pipeline predicted NAC response grade with 95.15% accuracy, substantially outperforming any single model used alone and demonstrating that combining complementary models captures more biologically relevant information than any single AI approach.
The morphological features that predict chemotherapy response in breast cancer fall into two broad categories. Structural atypia refers to the abnormal architectural patterns of cancer tissue: how cells arrange themselves into glandular structures, how they invade stroma, and how tumor regions are organized at the tissue level. Nuclear atypia refers to abnormalities in the cell nucleus itself: enlarged size, irregular contours, dense chromatin, and prominent nucleoli. Both structural and nuclear features carry independent biological information about tumor aggressiveness and treatment sensitivity.
Standard AI approaches to digital pathology use a single model, typically either a deep learning CNN or a classical machine learning method such as an SVM. The authors' prior work showed that CNNs excel at learning structural patterns from image patches, while SVMs combined with explicit feature extraction perform better at capturing the fine-grained quantitative nuclear characteristics that are important for classification tasks where exact measurements of nuclear shape, texture, and size matter.
The fundamental problem with using a single model for a task as complex as NAC response prediction is that cancer tissue heterogeneity creates scenarios where different aspects of the tissue morphology are most informative for different cases. A model that learns structural patterns may confidently classify cases where structural features are dominant, but struggle on cases where the discriminating signal is primarily in nuclear characteristics, and vice versa.
The study's hypothesis was that building separate specialized models for each type of atypia, then combining their outputs intelligently based on model confidence, would outperform any individual model. This ensemble pipeline approach, inspired by how pathologists sequentially examine slides at low and then high magnification, proved correct: the pipeline achieved approximately 10% higher accuracy than any single model alone.
The study enrolled 310 female breast cancer patients from three institutions in Japan and the US (Yamaguchi University, Tokyo Medical University, and Weill Cornell Medicine), all treated with NAC. Response was graded using the Japanese Breast Cancer Society (JBCS) criteria into four categories: RG0 (no response), RG1 (slight response), RG2 (marked response), and RG3 (complete response). All four breast cancer subtypes were represented: triple negative, hormone receptor positive, HER2 positive, and hormone receptor plus HER2 positive.
The CNN model used ResNeXt (an improved ResNet-50 architecture) trained on 256x256 pixel image patches extracted automatically from needle biopsy WSIs at 20x magnification. Color normalization was applied using the Macenko method to reduce staining variability between institutions. Multi-task learning added a cancer/non-cancer classification output alongside the four-class RG prediction, improving the model's ability to focus on diagnostically relevant tissue features. The CNN was trained on over 606,000 patches from 207 training cases.
The SVM and random forest models operated on nuclear features extracted from larger 2,048x2,048 pixel regions at 40x magnification. A pathologist manually segmented cancer from non-cancer areas, and automated nuclear segmentation software identified individual cell nuclei within cancer regions. From these nuclei, 82 morphological and texture features were extracted per cell, including nucleus size, contour length, roundness, eccentricity, and chromatin texture measures. These 82 features were expanded to 960 using the authors' novel Cell Feature Level Co-occurrence Matrix (CFLCM) method, which captures spatial relationships between nuclear features across the tissue.
A critical finding was that adding cancer subtype information to the SVM and RF models substantially improved their accuracy. Because molecular subtype (e.g., HER2 positive versus triple negative) is strongly linked to both nuclear morphology and chemotherapy sensitivity, combining subtype with morphological features created 15 distinct classes instead of 4, improving feature separation as confirmed by t-SNE visualization showing well-separated clusters when subtype was incorporated.
The CNN model achieved 86.4% case-level accuracy with Cohen's Kappa of 0.81 on the 103 test cases, with AUC values above 0.9 for each response grade class. The CNN's high performance reflected its ability to learn the variety of structural patterns in breast cancer tissue that are associated with different degrees of chemotherapy sensitivity, including glandular architecture disruption and nuclear organization at the tissue scale.
The SVM model trained with both response grade and subtype information achieved 85.4% accuracy and kappa 0.78 at the case level. Adding subtype information improved the SVM from 82.5% accuracy without subtype to 85.4% with it, confirming that nuclear morphological patterns are partly subtype-specific. The random forest model using the same 15-class framework achieved the highest individual model accuracy of 97.1% with kappa 0.96, though its out-of-bag error estimate suggested a potential generalization accuracy of 82.24%, indicating some caution about overfitting.
The integrated four-stage pipeline processed each test case sequentially: confident CNN predictions were returned immediately (38 of 103 cases), uncertain cases proceeded to SVM evaluation (51 cases), and cases the SVM could not confidently classify fell through to the random forest (22 cases). This confidence-gated design meant each model was only applied where its strengths were most likely to be useful. The final pipeline accuracy was 95.15%, with only 5 of 103 cases incorrectly classified.
For comparison, a simpler majority voting approach combining the three models achieved 94.17% accuracy, slightly below the pipeline. While the pipeline's advantage of about 1 percentage point over majority voting is modest, the authors note that majority voting is a more generalizable approach that does not require empirically determined thresholds, making it a practical alternative for implementations where pipeline tuning is not feasible.
The pipeline's design philosophy mirrors how pathologists examine tissue specimens: the CNN performs a broad structural assessment analogous to reviewing the slide at low magnification to understand tissue organization, while the SVM and RF models perform a more focused nuclear analysis analogous to examining specific areas at high magnification to assess individual cell abnormalities. This hierarchical confidence-based approach routes cases through progressively more specialized analyses when the first model is uncertain.
The finding that incorporating cancer subtype improved nuclear classification by both the SVM and RF models is biologically meaningful. Triple negative cancers, which lack expression of hormone receptors and HER2, tend to have different nuclear morphological characteristics from HER2-positive cancers, which overexpress a growth receptor protein. These subtype-specific nuclear patterns are informative for predicting which cancers will respond to specific chemotherapy regimens, and the AI models successfully learned to use this information.
A key limitation the authors identify is that the SVM and RF models still require manual cancer region annotation by a pathologist to identify which areas of the tissue to extract nuclear features from. For fully automated clinical deployment, this step would need to be replaced by the CNN's cancer/non-cancer classification output, so that the entire pipeline from raw slide to response prediction requires no manual input. The authors describe this as a planned next step for future work.
The cross-institutional design, spanning hospitals in Japan and the United States with different patient demographics, staining protocols, and chemotherapy regimens, provides meaningful evidence of generalizability. The fact that the model achieved high accuracy across these diverse settings suggests it has learned biologically fundamental tissue features rather than institution-specific technical artifacts.
This study demonstrates that combining complementary AI models targeting different aspects of cancer morphology can achieve high accuracy for a clinically important prediction task that single models struggle with. The 95.15% accuracy in predicting NAC response from pre-treatment biopsy tissue is clinically meaningful: if validated prospectively, such a tool could guide chemotherapy selection before treatment begins, potentially sparing non-responders from ineffective and toxic therapy.
The result that adding cancer subtype information substantially improved the nuclear classification models points toward a broader principle: AI models in pathology benefit from incorporating clinical context alongside morphological features. Future iterations could incorporate additional contextual data such as patient age, tumor size, and imaging findings to further improve prediction accuracy.
For clinical deployment, the next essential development is full automation of the nuclear analysis steps, replacing the current manually annotated cancer region inputs with outputs from the CNN's cancer detection capabilities. This would create a genuinely end-to-end pipeline that requires only a pre-treatment biopsy scan and molecular subtype information, both of which are obtained as part of standard clinical care.
The pipeline approach developed here represents a generalizable framework applicable beyond breast cancer NAC prediction. Any pathology task where different morphological characteristics carry complementary information, including grading, subtype classification, and recurrence risk prediction across various cancer types, could potentially benefit from multi-model pipelines that route ambiguous cases through progressively more specialized analyses rather than relying on a single universal model.