Neuromorphic Optical Tracking and Imaging of Randomly Moving Targets through Strongly Scattering Media

Tracking and acquiring simultaneous optical images of randomly moving targets obscured by scattering media remains a challenging problem of importance to many applications that require precise object localization and identification. In this work we develop an end-to-end neuromorphic optical engineering and computational approach to demonstrate how to track and image normally invisible objects by combining an event detecting camera with a multistage neuromorphic deep learning strategy. Photons emerging from dense scattering media are detected by the event camera and converted to pixel-wise asynchronized spike trains - a first step in isolating object-specific information from the dominant uninformative background. Spiking data is fed into a deep spiking neural network (SNN) engine where object tracking and image reconstruction are performed by two separate yet interconnected modules running in parallel in discrete time steps over the event duration. Through benchtop experiments we demonstrate tracking and imaging randomly moving objects in dense turbid media as well as image reconstruction of spatially stationary but optically dynamic objects. Standardized character sets serve as representative proxies for geometrically complex objects, underscoring the method's generality. The results highlight the advantages of a fully neuromorphic approach in meeting a major imaging technology with high computational efficiency and low power consumption.

Event-Driven Imaging in Turbid Media: A Confluence of Optoelectronics and Neuromorphic Computation

In this paper a new optical-computational method is introduced to unveil images of targets whose visibility is severely obscured by light scattering in dense, turbid media. The targets of interest are taken to be dynamic in that their optical properties are time-varying whether stationary in space or moving. The scheme, to our knowledge the first of its kind, is human vision inspired whereby diffuse photons collected from the turbid medium are first transformed to spike trains by a dynamic vision sensor as in the retina, and image reconstruction is then performed by a neuromorphic computing approach mimicking the brain. We combine benchtop experimental data in both reflection (backscattering) and transmission geometries with support from physics-based simulations to develop a neuromorphic computational model and then apply this for image reconstruction of different MNIST characters and image sets by a dedicated deep spiking neural network algorithm. Image reconstruction is achieved under conditions of turbidity where an original image is unintelligible to the human eye or a digital video camera, yet clearly and quantifiable identifiable when using the new neuromorphic computational approach.

An Asynchronous Wireless Network for Capturing Event-Driven Data from Large Populations of Autonomous Sensors

We introduce a wireless RF network concept for capturing sparse event-driven data from large populations of spatially distributed autonomous microsensors, possibly numbered in the thousands. Each sensor is assumed to be a microchip capable of event detection in transforming time-varying inputs to spike trains. Inspired by brain information processing, we have developed a spectrally efficient, low-error rate asynchronous networking concept based on a code-division multiple access method. We characterize the network performance of several dozen submillimeter-size silicon microchips experimentally, complemented by larger scale in silico simulations. A comparison is made between different implementations of on-chip clocks. Testing the notion that spike-based wireless communication is naturally matched with downstream sensor population analysis by neuromorphic computing techniques, we then deploy a spiking neural network (SNN) machine learning model to decode data from eight thousand spiking neurons in the primate cortex for accurate prediction of hand movement in a cursor control task.