AI-Powered Augmented Reality for Satellite Assembly, Integration and Test

The integration of Artificial Intelligence (AI) and Augmented Reality (AR) is set to transform satellite Assembly, Integration, and Testing (AIT) processes by enhancing precision, minimizing human error, and improving operational efficiency in cleanroom environments. This paper presents a technical description of the European Space Agency's (ESA) project "AI for AR in Satellite AIT," which combines real-time computer vision and AR systems to assist technicians during satellite assembly. Leveraging Microsoft HoloLens 2 as the AR interface, the system delivers context-aware instructions and real-time feedback, tackling the complexities of object recognition and 6D pose estimation in AIT workflows. All AI models demonstrated over 70% accuracy, with the detection model exceeding 95% accuracy, indicating a high level of performance and reliability. A key contribution of this work lies in the effective use of synthetic data for training AI models in AR applications, addressing the significant challenges of obtaining real-world datasets in highly dynamic satellite environments, as well as the creation of the Segmented Anything Model for Automatic Labelling (SAMAL), which facilitates the automatic annotation of real data, achieving speeds up to 20 times faster than manual human annotation. The findings demonstrate the efficacy of AI-driven AR systems in automating critical satellite assembly tasks, setting a foundation for future innovations in the space industry.

The ReSWARM Microgravity Flight Experiments: Planning, Control, and Model Estimation for On-Orbit Close Proximity Operations

On-orbit close proximity operations involve robotic spacecraft maneuvering and making decisions for a growing number of mission scenarios demanding autonomy, including on-orbit assembly, repair, and astronaut assistance. Of these scenarios, on-orbit assembly is an enabling technology that will allow large space structures to be built in-situ, using smaller building block modules. However, robotic on-orbit assembly involves a number of technical hurdles such as changing system models. For instance, grappled modules moved by a free-flying "assembler" robot can cause significant shifts in system inertial properties, which has cascading impacts on motion planning and control portions of the autonomy stack. Further, on-orbit assembly and other scenarios require collision-avoiding motion planning, particularly when operating in a "construction site" scenario of multiple assembler robots and structures. These complicating factors, relevant to many autonomous microgravity robotics use cases, are tackled in the ReSWARM flight experiments as a set of tests on the International Space Station using NASA's Astrobee robots. RElative Satellite sWarming and Robotic Maneuvering, or ReSWARM, demonstrates multiple key technologies for close proximity operations and on-orbit assembly: (1) global long-horizon planning, accomplished using offline and online sampling-based planner options that consider the system dynamics; (2) on-orbit reconfiguration model learning, using the recently-proposed RATTLE information-aware planning framework; and (3) robust control tools to provide low-level control robustness using current system knowledge. These approaches are detailed individually and in an "on-orbit assembly scenario" of multi-waypoint tracking on-orbit. Additionally, detail is provided discussing the practicalities of hardware implementation and unique aspects of working with Astrobee in microgravity.

The RATTLE Motion Planning Algorithm for Robust Online Parametric Model Improvement with On-Orbit Validation

Certain forms of uncertainty that robotic systems encounter can be explicitly learned within the context of a known model, like parametric model uncertainties such as mass and moments of inertia. Quantifying such parametric uncertainty is important for more accurate prediction of the system behavior, leading to safe and precise task execution. In tandem, providing a form of robustness guarantee against prevailing uncertainty levels like environmental disturbances and current model knowledge is also desirable. To that end, the authors' previously proposed RATTLE algorithm, a framework for online information-aware motion planning, is outlined and extended to enhance its applicability to real robotic systems. RATTLE provides a clear tradeoff between information-seeking motion and traditional goal-achieving motion and features online-updateable models. Additionally, online-updateable low level control robustness guarantees and a new method for automatic adjustment of information content down to a specified estimation precision is proposed. Results of extensive experimentation in microgravity using the Astrobee robots aboard the International Space Station and practical implementation details are presented, demonstrating RATTLE's capabilities for real-time, robust, online-updateable, and model information-seeking motion planning capabilities under parametric uncertainty.

Symplectic Momentum Neural Networks -- Using Discrete Variational Mechanics as a prior in Deep Learning

With deep learning being gaining attention from the research community for prediction and control of real physical systems, learning important representations is becoming now more than ever mandatory. It is of extremely importance that deep learning representations are coherent with physics. When learning from discrete data this can be guaranteed by including some sort of prior into the learning, however not all discretization priors preserve important structures from the physics. In this paper we introduce Symplectic Momentum Neural Networks (SyMo) as models from a discrete formulation of mechanics for non-separable mechanical systems. The combination of such formulation leads SyMos to be constrained towards preserving important geometric structures such as momentum and a symplectic form and learn from limited data. Furthermore, it allows to learn dynamics only from the poses as training data. We extend SyMos to include variational integrators within the learning framework by developing an implicit root-find layer which leads to End-to-End Symplectic Momentum Neural Networks (E2E-SyMo). Through experimental results, using the pendulum and cartpole we show that such combination not only allows these models tol earn from limited data but also provides the models with the capability of preserving the symplectic form and show better long-term behaviour.

Online Information-Aware Motion Planning with Inertial Parameter Learning for Robotic Free-Flyers

Space free-flyers like the Astrobee robots currently operating aboard the International Space Station must operate with inherent system uncertainties. Parametric uncertainties like mass and moment of inertia are especially important to quantify in these safety-critical space systems and can change in scenarios such as on-orbit cargo movement, where unknown grappled payloads significantly change the system dynamics. Cautiously learning these uncertainties en route can potentially avoid time- and fuel-consuming pure system identification maneuvers. Recognizing this, this work proposes RATTLE, an online information-aware motion planning algorithm that explicitly weights parametric model-learning coupled with real-time replanning capability that can take advantage of improved system models. The method consists of a two-tiered (global and local) planner, a low-level model predictive controller, and an online parameter estimator that produces estimates of the robot's inertial properties for more informed control and replanning on-the-fly; all levels of the planning and control feature online update-able models. Simulation results of RATTLE for the Astrobee free-flyer grappling an uncertain payload are presented alongside results of a hardware demonstration showcasing the ability to explicitly encourage model parametric learning while achieving otherwise useful motion.

COSMIC: fast closed-form identification from large-scale data for LTV systems

We introduce a closed-form method for identification of discrete-time linear time-variant systems from data, formulating the learning problem as a regularized least squares problem where the regularizer favors smooth solutions within a trajectory. We develop a closed-form algorithm with guarantees of optimality and with a complexity that increases linearly with the number of instants considered per trajectory. The COSMIC algorithm achieves the desired result even in the presence of large volumes of data. Our method solved the problem using two orders of magnitude less computational power than a general purpose convex solver and was about 3 times faster than a Stochastic Block Coordinate Descent especially designed method. Computational times of our method remained in the order of magnitude of the second even for 10k and 100k time instants, where the general purpose solver crashed. To prove its applicability to real world systems, we test with spring-mass-damper system and use the estimated model to find the optimal control path. Our algorithm was applied to both a Low Fidelity and Functional Engineering Simulators for the Comet Interceptor mission, that requires precise pointing of the on-board cameras in a fast dynamics environment. Thus, this paper provides a fast alternative to classical system identification techniques for linear time-variant systems, while proving to be a solid base for applications in the Space industry and a step forward to the incorporation of algorithms that leverage data in such a safety-critical environment.

Decision Support Models for Predicting and Explaining Airport Passenger Connectivity from Data

Predicting if passengers in a connecting flight will lose their connection is paramount for airline profitability. We present novel machine learning-based decision support models for the different stages of connection flight management, namely for strategic, pre-tactical, tactical and post-operations. We predict missed flight connections in an airline's hub airport using historical data on flights and passengers, and analyse the factors that contribute additively to the predicted outcome for each decision horizon. Our data is high-dimensional, heterogeneous, imbalanced and noisy, and does not inform about passenger arrival/departure transit time. We employ probabilistic encoding of categorical classes, data balancing with Gaussian Mixture Models, and boosting. For all planning horizons, our models attain an AUC of the ROC higher than 0.93. SHAP value explanations of our models indicate that scheduled/perceived connection times contribute the most to the prediction, followed by passenger age and whether border controls are required.

Conjunction Data Messages behave as a Poisson Process

Space debris is a major problem in space exploration. International bodies continuously monitor a large database of orbiting objects and emit warnings in the form of conjunction data messages. An important question for satellite operators is to estimate when fresh information will arrive so that they can react timely but sparingly with satellite maneuvers. We propose a statistical learning model of the message arrival process, allowing us to answer two important questions: (1) Will there be any new message in the next specified time interval? (2) When exactly and with what uncertainty will the next message arrive? The average prediction error for question (2) of our Bayesian Poisson process model is smaller than the baseline in more than 3 hours in a test set of 50k close encounter events.

Safe and Uncertainty-Aware Robotic Motion Planning Techniques for Agile On-Orbit Assembly

As access to space and robotic autonomy capabilities move forward, there is simultaneously a growing interest in deploying large, complex space structures to provide new on-orbit capabilities. New space-borne observatories, large orbital outposts, and even futuristic on-orbit manufacturing will be enabled by robotic assembly of space structures using techniques like on-orbit additive manufacturing which can provide flexibility in constructing and even repairing complex hardware. However, the dynamics underlying the robotic assembler during manipulation may operate under inertial uncertainties. Thus, inertial estimation of the robot and the manipulated component system must be considered during structural assembly. The contribution of this work is to address both the motion planning and control for robotic assembly with consideration of the inertial estimation of the combined free-flying robotic assembler and additively manufactured component system. Specifically, the Linear Quadratic Regulator Rapidly-Exploring Randomized Trees (LQR-RRT*) and dynamically feasible path smoothing are used to obtain obstacle-free trajectories for the system. Further, model learning is incorporated explicitly into the planning stages via approximation of the continuous system and accompanying reward of performing safe, objective-oriented motion. Remaining uncertainty can then be dealt with using robust tube model predictive control. By obtaining controlled trajectories that consider both obstacle avoidance and learning of the inertial properties of the free-flyer and manipulated component system, the free-flyer rapidly considers and plans the construction of space structures with enhanced system knowledge. The approach naturally generalizes to repairing, refueling, and re-provisioning space structure components while providing optimal collision-free trajectories under e.g., inertial uncertainty.

Human-Robot Collaboration in Microgravity: The Object Handover Problem

Collaborative space robots are an emerging research field with high impact as robots facilitate servicing functions in collaboration with astronauts with higher precision during lengthy tasks, on a tight schedule, with less risk and without requiring life support, making them more efficient and economically more viable. However, human-robot collaboration in space is still a challenge concerning key issues such as manipulation of weightless objects and mobility due to the peculiar motion dynamics of the robot and the manipulator on a microgravity environment. Therefore, this paper formulates and validates an algorithm that enables a free-flyer robot equipped with a manipulator to perform a successful, fluent, and dynamic robot-to-human and human-to-robot object handover. Furthermore, a virtual reality user interaction interface is developed using a Leap Motion device and a systematic user study is conducted. This study aims to analyse the user's preferences between a rigid and compliant impedance robot behavior during the object handover and to understand the impact of those behaviors on the success of the task. The results showed that the rigid behavior was overall more preferable and registered higher transfer success during the tasks.