Waveform Design of Multi-User-Multi-Target ISAC System based on Kullback-Leibler Divergence

This paper presents a novel framework that leverages the Kullback-Leibler divergence (KLD) to analyze and optimize performance trade-offs in integrated sensing and communication (ISAC) systems. We consider a multiple-input-multiple-output (MIMO) base station that simultaneously serves communication user equipments (UEs) and detects multiple targets using shared antenna deployment. The proposed KLD-based approach provides a unified performance measure encompassing both UE error rate and target detection capability. We apply this approach on two well-known communication beamforming techniques, maximum ratio transmission (MRT) and zero-forcing (ZF), and evaluate the effect on the radar subsystem. Furthermore, two optimization problems are formulated and solved. The first one optimizes the KLD of the radar subsystem for given constraints on the communication KLD, whereas the second one focuses on communication waveform KLD-based optimization and constrained radar KLD. These optimization problems are solved using a projected gradient method with an adaptive penalty for the radar waveform and a gradient-assisted interior point method for the communication waveform. Through theoretical derivations and extensive simulations, it is demonstrated that our approach can be a powerful tool for characterizing and optimizing the performance trade-offs of ISAC under various configurations. The results also show significant improvements in both sensing and communication performance by the KLD-optimized system compared to well-known benchmarks, such as conventional MRT and ZF for the communication subsystem, and the conventional identity covariance design for the radar subsystem. These findings support the holistic design and optimization of ISAC in next-generation wireless networks.