Breast cancer diagnosis relies heavily on histopathological image analysis - the microscopic examination of tissue samples stained with dyes like haematoxylin-eosin. Pathologists evaluate tissue structure, cell morphology, staining patterns, and spatial relationships between cells and tissue layers to classify tumors. This process is time-consuming, subject to inter-observer variability, and challenging to scale given the volume of specimens in clinical practice.
Convolutional neural networks (CNNs) have shown strong performance on many medical imaging tasks but have specific limitations for histopathological images. They process images locally, building up features through hierarchical convolutions, but struggle to capture spatial relationships between cells and tissue layers that are distributed across larger image regions. They also do not inherently handle the variability in staining properties between different laboratory preparations - a major source of domain shift in histopathology.
Vision Transformers (ViTs) address CNNs' locality limitation through global self-attention, but existing transformer-based approaches either process only part of the image or fail to adequately leverage stain-specific features. Neither pure CNN nor pure transformer approaches fully exploit the complementary information available in both the original RGB histopathology images and their stain-normalized counterparts.
This study proposed DFViT (Deep Fusion-based Vision Transformer), a novel hybrid architecture that combines: (1) image-level fusion of original RGB and stain-normalized histopathology images, and (2) feature-level fusion of CNN-extracted local features with transformer-extracted global features, aiming to overcome the limitations of both approaches through complementary integration.
DFViT implements fusion at two distinct levels. The first is image-level fusion: each histopathological image is processed through stain normalization, which standardizes the color and brightness of H&E stains to a common reference, eliminating laboratory-to-laboratory variability. The normalized image is then concatenated with the original RGB image to create a fused input that preserves both standardized features (for consistency) and original staining details (which may carry diagnostically relevant information that normalization could erase).
The fused image is then processed by two parallel branches for feature-level fusion. The first branch uses VGG16, a pre-trained CNN model with deep 3x3 convolution layers, with its final classification layer removed so it outputs a feature vector capturing local spatial patterns, textures, and edges - the kinds of low-level features CNNs excel at learning. The second branch uses a Vision Transformer (ViT) that processes the image as a sequence of patches and applies multi-head self-attention to capture long-range contextual relationships across the entire image.
The feature vectors from VGG16 and ViT are then concatenated into a single combined vector Z = [X, Y], where X has n features from VGG16 and Y has m features from ViT, producing a vector with n+m features. This combined representation retains both local low-level spatial information from the CNN and high-level global contextual information from the transformer - two sources of information that are inherently complementary for understanding complex tissue architecture.
The fused feature vector is processed by a multi-layer perceptron (MLP) with three hidden layers of 512 neurons each, using GELU activation and dropout (rate 0.3) to prevent overfitting. The MLP applies Softmax activation for multi-class outputs and Sigmoid for binary classification. Dropout randomly deactivates neurons during training, reducing the model's tendency to memorize training data rather than learning generalizable features - essential given the relatively small size of available histopathology datasets.
The model was evaluated on three distinct datasets representing different classification scenarios. The BreakHis dataset contains 7,909 breast cancer histopathology images from 82 patients at four different microscope magnification levels (40x, 100x, 200x, 400x), with 2,480 benign and 5,429 malignant tumor images. This binary classification dataset is the most widely used benchmark for breast histopathology classification algorithms.
The BACH dataset (Breast Cancer Histology) contains 400 microscope images categorized into four classes: normal tissue, benign lesion, carcinoma in situ (non-invasive cancer), and invasive carcinoma, with 100 images per class. This balanced multi-class dataset represents the more challenging problem of distinguishing between multiple disease states, not just benign vs. malignant.
The UC dataset from UCSC Cancer Genomics contains 3,302 endometrial specimens across four categories: endometrial hyperplasia (798 images), endometrioid cancer (535 images), endometrial polyps (636 images), and normal endometrium (1,333 images). This unbalanced multi-class dataset tests the model's ability to generalize beyond breast tissue to a related but distinct tissue type and pathology classification task.
The diversity of these three datasets - spanning binary vs. multi-class tasks, balanced vs. unbalanced class distributions, different tissue types, and multiple microscopy magnification levels - provides a comprehensive evaluation of the model's generalizability. Models that perform consistently well across all three settings are more likely to be clinically useful in diverse real-world scenarios.
On the BACH dataset (4-class breast classification), DFViT achieved 94% accuracy, 98% precision, 95% recall, and 96.48% F1-score. This substantially outperformed all baseline models: the best-performing prior model (DeconVit) achieved only 79% accuracy and 75.99% F1-score. Traditional CNN models including ResNet (85.47% F1), VGG16 (82% F1), and VGG19 (79.62% F1) all fell well short of DFViT's performance.
On the UC dataset (4-class endometrial classification), DFViT achieved 84.56% accuracy, 82% precision, 81% recall, and 81.51% F1-score, again outperforming all competitors. The prior best model (DeconVit) achieved 81.36% accuracy and 77.05% F1-score. Notably, this result demonstrates the model's ability to generalize to a tissue type (endometrium) and pathology set not used during the original design process.
On the BreakHis dataset (binary benign/malignant classification at multiple magnifications), DFViT achieved 95.29% accuracy, 97.0% precision, 96.91% recall, and 97.68% F1-score. This surpassed all comparison models including the previous best performers: Traditional ML with optimized deep features (95.45% accuracy, 95.45% F1) and SELF stacked ensemble learning (95.1% accuracy, 94.5% F1). Performance was consistent across all four magnification levels (40x, 100x, 200x, 400x), indicating robustness to image resolution variation.
Across all magnification levels in BreakHis, the model showed higher accuracy on malignant images than benign images, largely reflecting the class imbalance in the dataset (5,429 malignant vs. 2,480 benign). At 200x magnification, the model predicted 520/539 malignant images correctly (3 errors) but misclassified 44/244 benign images, consistent with the tendency of class-imbalanced models to favor the majority class - an acknowledged limitation requiring further class balancing strategies.
The superior performance of DFViT over both pure CNN and pure vision transformer approaches reflects the complementary nature of the fused information sources. VGG16 contributes low-level spatial features - edges, textures, shapes of individual cells and gland structures - while the ViT contributes high-level contextual awareness of how tissue regions relate to each other across the entire image. Neither alone captures the full picture that pathologists use for diagnosis.
The image-level fusion of RGB and stain-normalized images addresses a known challenge in histopathology AI: staining variability between laboratories. Stain normalization improves consistency but may discard subtle staining differences that have diagnostic significance. By feeding both the normalized and original images to the model, DFViT retains the benefits of normalization (reduced domain shift) while preserving original staining details that pure normalization approaches would sacrifice.
The approach of leveraging pre-trained VGG16 features also helps the ViT component perform well on the limited histopathology training sets. Vision transformers typically require large amounts of data to train effectively from scratch. By providing the ViT with a rich stream of pre-learned visual features from VGG16 (trained on ImageNet), the fusion approach effectively reduces the ViT's data hunger - making the combined model more practical for the relatively small datasets available in medical imaging.
The persistent challenge of class imbalance - particularly visible in BreakHis where malignant cases outnumber benign cases by more than 2:1 - limits performance on the minority class. This is a systemic issue in medical imaging datasets where pathological cases are typically better represented than normal cases. Future work incorporating targeted oversampling or loss function weighting could further improve benign case detection.
DFViT establishes a new approach to breast cancer histopathology classification by implementing fusion at both the image level (RGB + stain-normalized) and the feature level (CNN + transformer). The results on three benchmark datasets demonstrate that this dual fusion approach consistently outperforms single-architecture models, achieving state-of-the-art performance across binary and multi-class tasks at multiple tissue types and magnification levels.
The clinical implications of such a system are significant. A tool that can accurately and consistently classify histopathological images could serve as a quality control measure alongside pathologist review, flag potential diagnostic errors, and provide second opinions in settings where specialist pathology expertise is limited. The model's generalizability across tissue types (breast and endometrium) suggests broader applicability across cancer pathology workflows.
Key limitations acknowledged by the authors include potential overfitting on small datasets, possible noise introduced by the image fusion process, and increased computational complexity from the hybrid transformer-CNN architecture. These factors could limit scalability in high-throughput clinical laboratory settings.
Future directions include advanced data augmentation strategies tailored to histopathology images, ensemble methods combining multiple classifiers, and investigation of model explainability tools such as attention visualization to identify which tissue features drive classification decisions. Integration with Internet of Medical Things (IoMT) infrastructure and cybersecurity considerations for clinical deployment are also identified as important future research areas.