Cross-Band Modulation Design for Hybrid RF-Optical Systems

We present a novel cross-band modulation framework that combines 3D modulation in the RF domain with intensity modulation and direct detection in the optical domain, the first such integration to enhance communication reliability. By harnessing cross-band diversity, the framework optimizes symbol mapping across RF and optical links, significantly boosting mutual information (MI) and reducing symbol error probability (SEP). Two practical modulation schemes implement this framework, both using quadrature amplitude modulation in the RF subsystem. The first is a linear cross-band mapping scheme, where RF symbols are mapped to optical intensity values via an analytically tractable optimization that ensures O(1) detection complexity while minimizing SEP. The second employs a deep neural network-generated (DNN-Gen) 3D constellation with a custom loss function that adaptively optimizes symbol placement to maximize MI and minimize SEP. Although DNN-Gen incurs higher computational complexity than the linear approach, it adapts the 3D constellation to varying signal-to-noise ratios, yielding significant performance gains. Furthermore, we derive a theoretical MI benchmark for the linear scheme, offering insights into the fundamental limits of RF-optical cross-band communication. Extensive Monte Carlo simulations confirm that both schemes outperform SoA cross-band modulation techniques, including cross-band pulse amplitude modulation, with notable improvements. Additionally, DNN-Gen maintains high performance over a range of RF SNRs, lessening the need for exhaustive training at every operating condition. Overall, these results establish our cross-band modulation framework as a scalable, high-performance solution for next-generation hybrid RF-optical networks, balancing low complexity with optimized symbol mapping to maximize system reliability and efficiency.