Fail-Safe Human Detection for Drones Using a Multi-Modal Curriculum Learning Approach

Drones are currently being explored for safety-critical applications where human agents are expected to evolve in their vicinity. In such applications, robust people avoidance must be provided by fusing a number of sensing modalities in order to avoid collisions. Currently however, people detection systems used on drones are solely based on standard cameras besides an emerging number of works discussing the fusion of imaging and event-based cameras. On the other hand, radar-based systems provide up-most robustness towards environmental conditions but do not provide complete information on their own and have mainly been investigated in automotive contexts, not for drones. In order to enable the fusion of radars with both event-based and standard cameras, we present KUL-UAVSAFE, a first-of-its-kind dataset for the study of safety-critical people detection by drones. In addition, we propose a baseline CNN architecture with cross-fusion highways and introduce a curriculum learning strategy for multi-modal data termed SAUL, which greatly enhances the robustness of the system towards hard RGB failures and provides a significant gain of 15% in peak F1 score compared to the use of BlackIn, previously proposed for cross-fusion networks. We demonstrate the real-time performance and feasibility of the approach by implementing the system in an edge-computing unit. We release our dataset and additional material in the project home page.

A 2-$μ$J, 12-class, 91% Accuracy Spiking Neural Network Approach For Radar Gesture Recognition

Radar processing via spiking neural networks (SNNs) has recently emerged as a solution in the field of ultra-low-power wireless human-computer interaction. Compared to traditional energy- and area-hungry deep learning methods, SNNs are significantly more energy efficient and can be deployed in the growing number of compact SNN accelerator chips, making them a better solution for ubiquitous IoT applications. We propose a novel SNN strategy for radar gesture recognition, achieving more than 91% of accuracy on two different radar datasets. Our work significantly differs from previous approaches as 1) we use a novel radar-SNN training strategy, 2) we use quantized weights, enabling power-efficient implementation in real-world SNN hardware, and 3) we report the SNN energy consumption per classification, clearly demonstrating the real-world feasibility and power savings induced by SNN-based radar processing. We release evaluation code to help future research.