The Importance of Being Earnest: Performance of Modulation Classification for Real RF Signals

Digital modulation classification (DMC) can be highly valuable for equipping radios with increased spectrum awareness in complex emerging wireless networks. However, as the existing literature is overwhelmingly based on theoretical or simulation results, it is unclear how well DMC performs in practice. In this paper we study the performance of DMC in real-world wireless networks, using an extensive RF signal dataset of 250,000 over-the-air transmissions with heterogeneous transceiver hardware and co-channel interference. Our results show that DMC can achieve a high classification accuracy even under the challenging real-world conditions of modulated co-channel interference and low-grade hardware. However, this only holds if the training dataset fully captures the variety of interference and hardware types in the real radio environment; otherwise, the DMC performance deteriorates significantly. Our work has two important engineering implications. First, it shows that it is not straightforward to exchange learned classifier models among dissimilar radio environments and devices in practice. Second, our analysis suggests that the key missing link for real-world deployment of DMC is designing signal features that generalize well to diverse wireless network scenarios. We are making our RF signal dataset publicly available as a step towards a unified framework for realistic DMC evaluation.