DAMM-Diffusion: Learning Divergence-Aware Multi-Modal Diffusion Model for Nanoparticles Distribution Prediction

The prediction of nanoparticles (NPs) distribution is crucial for the diagnosis and treatment of tumors. Recent studies indicate that the heterogeneity of tumor microenvironment (TME) highly affects the distribution of NPs across tumors. Hence, it has become a research hotspot to generate the NPs distribution by the aid of multi-modal TME components. However, the distribution divergence among multi-modal TME components may cause side effects i.e., the best uni-modal model may outperform the joint generative model. To address the above issues, we propose a \textbf{D}ivergence-\textbf{A}ware \textbf{M}ulti-\textbf{M}odal \textbf{Diffusion} model (i.e., \textbf{DAMM-Diffusion}) to adaptively generate the prediction results from uni-modal and multi-modal branches in a unified network. In detail, the uni-modal branch is composed of the U-Net architecture while the multi-modal branch extends it by introducing two novel fusion modules i.e., Multi-Modal Fusion Module (MMFM) and Uncertainty-Aware Fusion Module (UAFM). Specifically, the MMFM is proposed to fuse features from multiple modalities, while the UAFM module is introduced to learn the uncertainty map for cross-attention computation. Following the individual prediction results from each branch, the Divergence-Aware Multi-Modal Predictor (DAMMP) module is proposed to assess the consistency of multi-modal data with the uncertainty map, which determines whether the final prediction results come from multi-modal or uni-modal predictions. We predict the NPs distribution given the TME components of tumor vessels and cell nuclei, and the experimental results show that DAMM-Diffusion can generate the distribution of NPs with higher accuracy than the comparing methods. Additional results on the multi-modal brain image synthesis task further validate the effectiveness of the proposed method.

GANDA: A deep generative adversarial network predicts the spatial distribution of nanoparticles in tumor pixelly

Intratumoral nanoparticles (NPs) distribution is critical for the diagnostic and therapeutic effect, but methods to predict the distribution remain unavailable due to the complex bio-nano interactions. Here, we developed a Generative Adversarial Network for Distribution Analysis (GANDA) to make pixels-to-pixels prediction of the NPs distribution across tumors. This predictive model used deep learning approaches to automatically learn the features of tumor vessels and cell nuclei from whole-slide images of tumor sections. We showed that the GANDA could generate images of NPs distribution with the same spatial resolution as original images of tumor vessels and nuclei. The GANDA enabled quantitative analysis of NPs distribution (R2=0.93) and extravasation without knowing their real distribution. This model provides opportunities to investigate how influencing factors affect NPs distribution in individual tumors and may guide nanomedicine optimization for personalized treatments.