Multiparametric MRI (mpMRI) has demonstrated superior ability to detect and localize clinically significant prostate cancer compared with traditional ultrasound-guided biopsy. However, biopsies are performed under real-time transrectal ultrasound (TRUS) guidance -- which is cheaper, faster, and more widely available than MRI -- creating a fundamental gap between where cancer is identified (on MRI) and where the biopsy needle is guided (using ultrasound).
MR-TRUS fusion biopsy addresses this gap by overlaying pre-biopsy MRI lesion targets onto the real-time ultrasound field during biopsy, allowing the urologist to direct the needle toward MRI-identified suspicious areas. While this spatial overlay has been shown to improve high-risk cancer detection rates, the fusion currently transfers only the location labels from MRI -- not the rich quantitative tissue information within each imaging modality.
A novel ultrasound-based technology called temporal enhanced ultrasound (TeUS) analyzes a time series of radio frequency (RF) ultrasound data from a stationary tissue location, capturing dynamic tissue properties that standard single-frame ultrasound images cannot detect. Prior studies using TeUS with deep learning models achieved AUCs as high as 0.84 for differentiating prostate cancer from healthy tissue -- suggesting that TeUS captures tissue biology complementary to what MRI measures.
No prior study had quantitatively integrated the information content of both MRI and TeUS within a machine learning framework. This study, conducted by researchers from Queen's University, UBC, Harvard, and the NIH, aimed to investigate whether deep learning-based fusion of TeUS and ADC images (from diffusion-weighted MRI) could outperform either modality alone for detecting prostate cancer foci during biopsy.
The study used data from 107 patients who underwent MR-TRUS fusion-guided prostate biopsy at the NIH. For each biopsy core, the team captured 5 seconds of real-time ultrasound RF data (100 frames) at the biopsy target location while the ultrasound probe was held stationary. This produced a TeUS time series that encodes how the tissue's acoustic response changes over time -- a property related to tissue microstructure and cellularity.
The dataset included 145 biopsy cores in total: 51 cancer-positive and 94 benign. The ground truth for each core was established by histopathology of the biopsied tissue. Only ADC images (from diffusion-weighted MRI) were available alongside TeUS -- not the full multiparametric MRI suite -- giving the study a biparametric rather than full multiparametric scope. Data were split 65/35 into training/validation and test sets using stratified sampling on tumor length.
TeUS processing was technically complex. The raw RF time series data were first converted to the frequency domain using fast Fourier transform (FFT) to extract frequency-domain tissue signatures. An autoencoder (AE) neural network was then trained to compress the high-dimensional TeUS features into a single 2D feature map matching the ADC image dimensions, making both modalities compatible for joint deep learning training.
Label generation was necessary because biopsy provides only point-level tissue labels at the needle tip, not pixel-level labels for the entire imaging plane. A disk-shaped region of interest was generated around each biopsy target location, with radius proportional to the tumor-in-core length, to create approximate spatial labels for training the fully convolutional segmentation networks.
Three information fusion strategies were compared: early fusion (combining raw modality inputs before the network), late fusion (training separate models per modality and averaging their output probabilities), and intermediate fusion (sharing a representation layer partway through the network while keeping separate encoders and decoders for each modality). The study hypothesized that intermediate fusion would best capture the complementary information each modality provides while avoiding the challenges of aligning imperfectly registered images early in the pipeline.
Intermediate fusion was implemented through a dual-stream U-Net: each modality (ADC and TeUS) fed into its own separate encoder network through four scales of convolutions. At the deepest encoding layer, the two streams shared weights, forcing the network to learn a common abstract representation of cancer-relevant features present in both modalities. Separate decoders then recovered spatial predictions for each modality's coordinate space independently, avoiding confusion from the different field-of-view between MRI and ultrasound.
An attention U-Net variant was also tested, which adds spatial attention gates that weight the importance of different spatial regions when combining encoder and decoder features. Attention mechanisms have shown benefit in segmentation tasks where the target structure is small and surrounded by varied tissue -- potentially relevant for detecting small prostate cancer foci amid benign gland tissue.
Training used a partial cross-entropy loss function, which computes the training signal only at biopsy-proven labeled locations rather than for every pixel in the image. This approach acknowledges that ground truth is only available at biopsy needle sites, not across the entire imaging plane, and allows fully convolutional networks to be trained on sparse biopsy supervision data.
The multimodal intermediate fusion model outperformed both unimodal models (ADC alone and TeUS alone) for prostate cancer detection. The best result -- achieved with the intermediate fusion Attention U-Net -- was an AUC of 0.76 for detecting all cancer-positive cores, and an AUC of 0.89 for detecting larger cancer foci (those with greater tumor-in-core length). These results represent a meaningful improvement over either modality used independently.
The late fusion strategy -- simply averaging the predictions of two separately trained unimodal models -- yielded modest improvements over the weaker unimodal model but did not fully capitalize on the complementary information available. Weighted averaging of the two modalities was also explored but produced no significant additional improvement, suggesting that the complementary signal between modalities requires learned feature-level integration rather than output-level combination.
The improvement was more pronounced for smaller cancer foci in the intermediate fusion setting. The authors hypothesize that small tumors are more difficult to detect from either MRI or TeUS alone, so the complementary tissue characterization each modality offers becomes more valuable precisely in the cases where either single-modality approach is least reliable. This is clinically important because small, early-stage cancer foci are the hardest to biopsy accurately.
The computation time for the most complex model (intermediate fusion with two-stream U-Net) was 44 milliseconds on a consumer-grade GPU, demonstrating that the approach is compatible with near-real-time clinical deployment during an active biopsy procedure -- an important practical requirement for intraoperative use.
This study is the first to demonstrate that quantitative deep learning integration of MRI diffusion data and temporal ultrasound signals improves prostate cancer foci detection over single-modality approaches. By generating per-pixel cancer likelihood maps across the entire imaging plane rather than evaluating each candidate region separately, the fully convolutional approach provides spatially rich guidance that could be displayed in real time during biopsy procedures.
The practical significance is substantial: during a TRUS-guided biopsy, a clinician currently relies on visual MRI overlay to aim the needle. Adding an AI-generated cancer likelihood map derived from both the MRI tissue information and the live TeUS signal could dynamically highlight the most suspicious tissue regions in the real-time ultrasound field, potentially improving cancer detection yield while reducing the number of cores required.
Future directions highlighted by the authors include handling missing modality scenarios -- since many biopsy centers do not perform pre-biopsy MRI, the system would need to function on TeUS alone when MRI data are unavailable. Additionally, incorporating transfer learning from larger publicly available MRI datasets and exploring alternative late fusion mechanisms could further improve performance in larger patient cohorts.