Pathological complete response (pCR) after neoadjuvant chemotherapy (NAC) is a strong surrogate for improved disease-free survival in breast cancer. The ability to predict pCR from a pre-treatment biopsy would allow clinicians to identify patients unlikely to benefit from standard chemotherapy before they are exposed to its toxicity.
Predicting outcomes from pre-treatment images is substantially harder than from post-treatment images. For example, deep learning models applied to MRI before chemotherapy achieved an AUC of only 0.55, compared to 0.97 when using post-treatment images. This gap reflects that pre-treatment tumors contain complex information that is not yet clearly organized around treatment response.
The tumor immune microenvironment, particularly tumor-infiltrating lymphocytes (TILs), has been repeatedly linked to chemotherapy response. Higher TIL levels are associated with pCR in multiple breast cancer subtypes. Specific immune markers including CD8+ cytotoxic T cells, CD163+ macrophages, and PD-L1 expression have each been studied, but prior analyses typically examined them manually and independently, missing the spatial and combinatorial information available in whole slide images.
This study addressed these gaps by building an automated pipeline called IMPRESS (IMage-based Pathological REgistration and Segmentation Statistics) that systematically extracts 36 quantitative features from both H&E-stained and multiplex immunohistochemistry (IHC) whole slide images, then uses machine learning to predict NAC response in HER2-positive and triple-negative breast cancer.
The study included 62 HER2-positive (HER2+) and 64 triple-negative breast cancer (TNBC) female patients treated with NAC from 2011 to 2016. All patients underwent biopsy before treatment and surgical resection after NAC. HER2+ patients received anthracycline-cyclophosphamide combined with taxol and trastuzumab, while TNBC patients received standard anthracycline-taxane chemotherapy.
In the HER2+ cohort, 38 patients (61 percent) achieved pCR and 24 (39 percent) had residual tumor. In the TNBC cohort, 27 patients (42 percent) achieved pCR and 37 (58 percent) had residual tumor. For patients with residual disease, the Residual Cancer Burden (RCB) index was calculated as a continuous quantitative measure of remaining disease extent.
Two independent external validation cohorts, each containing 20 patients (10 pCR and 10 residual), were used to test whether models trained on the primary cohort could generalize to different patient populations. This external validation is a critical methodological requirement before any AI tool can be considered for clinical use.
Multiplex IHC staining with antibodies against CD8 (for cytotoxic T cells), CD163 (for macrophages), and PD-L1 was performed on freshly cut sections from pre-treatment biopsies. Standard H&E slides were prepared in parallel, providing two complementary image types for the same tissue sample from each patient.
The IMPRESS pipeline begins with tissue segmentation of H&E whole slide images using a deep neural network called DeepLabV3, trained on 151 annotated breast cancer images from The Cancer Genome Atlas. The model automatically identifies three tissue regions: the stromal region, the tumoral region, and the lymphocytes aggregated region, with segmentation accuracy (dice coefficients) ranging from 0.70 to 0.94 when validated against pathologist labels.
IHC slide images are then co-registered with their corresponding H&E images using non-rigid image registration. Because H&E and IHC slides are cut from adjacent but non-identical tissue sections, they are not perfectly aligned. The registration algorithm reduces median alignment error from over 278 micrometers to under 19 micrometers in the HER2+ cohort, enabling accurate spatial overlay of immune marker locations onto tissue region maps.
Individual IHC markers (CD8, CD163, PD-L1) are segmented from registered IHC images using color-based K-means clustering in LAB color space, which maximizes contrast between different stain colors. From the combined H&E segmentation and IHC segmentation, three categories of spatial features are calculated for each marker in each tissue region: area ratio (how much of a region expresses the marker), proportion (what fraction of that marker is in each region), and purity (what fraction of immune markers in a region are of that type).
This produces 36 IMPRESS features per patient in total: 11 features for each of the three IHC markers plus 3 features describing overall tissue region proportions. These features are interpretable and tied to specific biological meanings, in contrast to black-box deep learning features that cannot be traced back to tissue biology.
A LASSO-regularized logistic regression model trained on IMPRESS features combined with clinical data achieved an AUC of 0.8975 for predicting pCR in HER2+ breast cancer, significantly outperforming the model trained on pathologist-assessed IHC features (AUC = 0.7880, p-value less than 2e-40). In TNBC, IMPRESS achieved an AUC of 0.7674, comparable to pathologist assessment (AUC = 0.7626), though without statistically significant difference.
Combining both H&E and IHC features consistently outperformed either modality alone. In HER2+ patients, the combined IMPRESS model significantly exceeded H&E-only features (AUC = 0.8118) and IHC-only features (AUC = 0.7746). Similar patterns held for TNBC, though with smaller absolute differences. This confirms that H&E morphology and immune marker information provide complementary rather than redundant information.
In external validation, the HER2+ model maintained strong performance with an AUC of 0.90, closely matching its training cohort AUC of 0.90. The TNBC model showed substantially reduced performance in external validation (AUC = 0.59), suggesting the HER2+ model generalizes well while the TNBC model may require larger and more diverse training data to overcome the heterogeneity of that subtype.
Despite strong AUC values, negative predictive values (NPV) were moderate in both cohorts, meaning the models produced a meaningful number of false negatives. This limitation indicates that while IMPRESS represents a significant advance over manual assessment, further improvements and larger training datasets are needed before it could function as a reliable clinical triage tool.
Feature importance analysis revealed that in HER2+ breast cancer, the most favorable predictors of pCR were immune features in the lymphocytes aggregated region, specifically CD8 ratio, CD163 ratio, and PD-L1 ratio in that region. High CD8+ and CD163+ immune cell density in lymphocyte-rich areas of the tumor predicted better chemotherapy response. The HER2/CEP17 ratio was also a strong favorable predictor, consistent with prior literature.
In contrast, the most adverse predictors in HER2+ patients included older age, estrogen receptor positivity (ER ratio and PR status), and notably, high stromal CD8 proportion, meaning a large fraction of CD8+ T cells located specifically within the stroma rather than in lymphocyte aggregates or tumor regions. This finding suggests that the spatial location of immune cells, not just their overall quantity, is crucial for predicting chemotherapy benefit.
In TNBC, favorable features were dominated by PD-L1 expression: Lymph PD-L1 ratio and proportion were the top favorable markers. Tumor CD8 proportion and Lymph CD163 proportion also appeared as favorable signals. The TNBC feature profile differed meaningfully from HER2+ patients, suggesting that these two subtypes have distinct immunological determinants of chemotherapy sensitivity despite both benefiting from high TIL levels generally.
Strikingly, stromal CD8 proportion emerged as a common adverse marker in both subtypes. This counterintuitive finding, that CD8+ cytotoxic T cells in the stroma predict worse response, contrasts with the favorable role of CD8+ cells in lymphocyte aggregates and tumor regions. It implies that T cell functional exclusion or spatial mislocation within the stroma may reflect immune dysfunction rather than active anti-tumor immunity.
Beyond predicting pCR as a binary outcome, IMPRESS features were also correlated with the continuous Residual Cancer Burden (RCB) index in patients with incomplete responses. In HER2+ patients, 13 of the 36 IMPRESS features showed significant correlations with RCB, while only 3 features did so in TNBC patients, consistent with the better overall performance of IMPRESS in HER2+.
In HER2+ patients, features including Lymph CD8 ratio, Lymph CD163 ratio, Lymph PD-L1 ratio, and Lymph CD8 proportion were all negatively correlated with RCB, meaning higher values of these features predicted lower residual disease burden. In contrast, stromal CD8 proportion and tumor CD163 purity were positively correlated with RCB, predicting more extensive residual disease.
The consistency between machine learning feature importance analysis and independent correlation analysis with RCB strengthens confidence in the biological relevance of these findings. Features that machine learning identified as predictors of pCR were the same features that correlated quantitatively with the degree of residual disease in non-complete responders.
The primary advantage of the IMPRESS approach over manual pathologist assessment lies in three dimensions: comprehensiveness, objectivity, and reproducibility. Manual IHC scoring by pathologists captures the approximate percentage of positively stained cells but cannot systematically quantify the spatial relationships between different cell types and tissue regions that IMPRESS captures across all 36 feature dimensions simultaneously.
AI-based analysis of whole slide images avoids sampling bias inherent in manual spot assessments of a small number of fields. IMPRESS evaluates every tissue pixel in the entire slide, weighting findings proportionally to tissue area rather than relying on a pathologist's visual selection of representative regions. This approach is especially advantageous for spatially heterogeneous tumors.
Reproducibility is another key advantage. Manual pathologist assessments, even with consensus review, carry inter-observer variability that makes it difficult to standardize results across institutions. Automated pipelines with fixed algorithms produce identical results for identical inputs, which is a prerequisite for multicenter clinical implementation and regulatory approval.
Interpretability is a critical feature that distinguishes IMPRESS from pure deep learning black-box approaches. Each of the 36 IMPRESS features has a defined biological meaning traceable to a specific immune marker in a specific tissue region. This allows clinicians and researchers to understand which pathological findings are driving the prediction, rather than accepting opaque numerical outputs from end-to-end neural networks.
The most significant limitation is cohort size. With 62 HER2+ and 64 TNBC patients, the study is too small to fully capture the biological heterogeneity of these subtypes, particularly TNBC. The TNBC model's drop in external validation performance from AUC 0.77 to 0.59 likely reflects this limitation, as a simple linear model cannot learn complex patterns from a small, heterogeneous dataset. Larger multicenter datasets are needed.
The IHC panel was limited to three markers: CD8, CD163, and PD-L1. The full tumor immune microenvironment involves dozens of cell types and molecules. Adding markers such as Ki-67, FOXP3 for regulatory T cells, or markers for natural killer cells and dendritic cells could substantially enrich the feature set and potentially improve prediction for TNBC.
The current pipeline requires separate H&E and IHC slides that must be co-registered computationally. Emerging multiplex imaging technologies such as CODEX, MIBI, and spatial transcriptomics can simultaneously measure many markers on a single tissue section, potentially enabling richer IMPRESS-like features without the registration challenge and with greater spatial precision at the single-cell level.
Future work should also explore extracting regional rather than whole-slide features, enabling spatial mapping of predictive microenvironments within tumors. This would allow pathologists to visualize specific regions driving the AI prediction, making findings more actionable for clinical decision-making and providing a clearer link between computational analysis and pathological interpretation.