Vision Foundation Models for Domain Generalisable Cross-View Localisation in Planetary Ground-Aerial Robotic Teams

Accurate localisation in planetary robotics enables the advanced autonomy required to support the increased scale and scope of future missions. The successes of the Ingenuity helicopter and multiple planetary orbiters lay the groundwork for future missions that use ground-aerial robotic teams. In this paper, we consider rovers using machine learning to localise themselves in a local aerial map using limited field-of-view monocular ground-view RGB images as input. A key consideration for machine learning methods is that real space data with ground-truth position labels suitable for training is scarce. In this work, we propose a novel method of localising rovers in an aerial map using cross-view-localising dual-encoder deep neural networks. We leverage semantic segmentation with vision foundation models and high volume synthetic data to bridge the domain gap to real images. We also contribute a new cross-view dataset of real-world rover trajectories with corresponding ground-truth localisation data captured in a planetary analogue facility, plus a high volume dataset of analogous synthetic image pairs. Using particle filters for state estimation with the cross-view networks allows accurate position estimation over simple and complex trajectories based on sequences of ground-view images.

Real-Time Instrument Planning and Perception for Novel Measurements of Dynamic Phenomena

Advancements in onboard computing mean remote sensing agents can employ state-of-the-art computer vision and machine learning at the edge. These capabilities can be leveraged to unlock new rare, transient, and pinpoint measurements of dynamic science phenomena. In this paper, we present an automated workflow that synthesizes the detection of these dynamic events in look-ahead satellite imagery with autonomous trajectory planning for a follow-up high-resolution sensor to obtain pinpoint measurements. We apply this workflow to the use case of observing volcanic plumes. We analyze classification approaches including traditional machine learning algorithms and convolutional neural networks. We present several trajectory planning algorithms that track the morphological features of a plume and integrate these algorithms with the classifiers. We show through simulation an order of magnitude increase in the utility return of the high-resolution instrument compared to baselines while maintaining efficient runtimes.

Demonstrating Onboard Inference for Earth Science Applications with Spectral Analysis Algorithms and Deep Learning

In partnership with Ubotica Technologies, the Jet Propulsion Laboratory is demonstrating state-of-the-art data analysis onboard CogniSAT-6/HAMMER (CS-6). CS-6 is a satellite with a visible and near infrared range hyperspectral instrument and neural network acceleration hardware. Performing data analysis at the edge (e.g. onboard) can enable new Earth science measurements and responses. We will demonstrate data analysis and inference onboard CS-6 for numerous applications using deep learning and spectral analysis algorithms.

AI Space Cortex: An Experimental System for Future Era Space Exploration

Our Robust, Explainable Autonomy for Scientific Icy Moon Operations (REASIMO) effort contributes to NASA's Concepts for Ocean worlds Life Detection Technology (COLDTech) program, which explores science platform technologies for ocean worlds such as Europa and Enceladus. Ocean world missions pose significant operational challenges. These include long communication lags, limited power, and lifetime limitations caused by radiation damage and hostile conditions. Given these operational limitations, onboard autonomy will be vital for future Ocean world missions. Besides the management of nominal lander operations, onboard autonomy must react appropriately in the event of anomalies. Traditional spacecraft rely on a transition into 'safe-mode' in which non-essential components and subsystems are powered off to preserve safety and maintain communication with Earth. For a severely time-limited Ocean world mission, resolutions to these anomalies that can be executed without Earth-in-the-loop communication and associated delays are paramount for completion of the mission objectives and science goals. To address these challenges, the REASIMO effort aims to demonstrate a robust level of AI-assisted autonomy for such missions, including the ability to detect and recover from anomalies, and to perform missions based on pre-trained behaviors rather than hard-coded, predetermined logic like all prior space missions. We developed an AI-assisted, personality-driven, intelligent framework for control of an Ocean world mission by combining a mix of advanced technologies. To demonstrate the capabilities of the framework, we perform tests of autonomous sampling operations on a lander-manipulator testbed at the NASA Jet Propulsion Laboratory, approximating possible surface conditions such a mission might encounter.

Active Inference in Contextual Multi-Armed Bandits for Autonomous Robotic Exploration

Autonomous selection of optimal options for data collection from multiple alternatives is challenging in uncertain environments. When secondary information about options is accessible, such problems can be framed as contextual multi-armed bandits (CMABs). Neuro-inspired active inference has gained interest for its ability to balance exploration and exploitation using the expected free energy objective function. Unlike previous studies that showed the effectiveness of active inference based strategy for CMABs using synthetic data, this study aims to apply active inference to realistic scenarios, using a simulated mineralogical survey site selection problem. Hyperspectral data from AVIRIS-NG at Cuprite, Nevada, serves as contextual information for predicting outcome probabilities, while geologists' mineral labels represent outcomes. Monte Carlo simulations assess the robustness of active inference against changing expert preferences. Results show that active inference requires fewer iterations than standard bandit approaches with real-world noisy and biased data, and performs better when outcome preferences vary online by adapting the selection strategy to align with expert shifts.

Non-myopic Planetary Exploration Combining In Situ and Remote Measurements

Remote sensing can provide crucial information for planetary rovers. However, they must validate these orbital observations with in situ measurements. Typically, this involves validating hyperspectral data using a spectrometer on-board the field robot. In order to achieve this, the robot must visit sampling locations that jointly improve a model of the environment while satisfying sampling constraints. However, current planners follow sub-optimal greedy strategies that are not scalable to larger regions. We demonstrate how the problem can be effectively defined in an MDP framework and propose a planning algorithm based on Monte Carlo Tree Search, which is devoid of the common drawbacks of existing planners and also provides superior performance. We evaluate our approach using hyperspectral imagery of a well-studied geologic site in Cuprite, Nevada.