A Simulation Pipeline to Facilitate Real-World Robotic Reinforcement Learning Applications

Reinforcement learning (RL) has gained traction for its success in solving complex tasks for robotic applications. However, its deployment on physical robots remains challenging due to safety risks and the comparatively high costs of training. To avoid these problems, RL agents are often trained on simulators, which introduces a new problem related to the gap between simulation and reality. This paper presents an RL pipeline designed to help reduce the reality gap and facilitate developing and deploying RL policies for real-world robotic systems. The pipeline organizes the RL training process into an initial step for system identification and three training stages: core simulation training, high-fidelity simulation, and real-world deployment, each adding levels of realism to reduce the sim-to-real gap. Each training stage takes an input policy, improves it, and either passes the improved policy to the next stage or loops it back for further improvement. This iterative process continues until the policy achieves the desired performance. The pipeline's effectiveness is shown through a case study with the Boston Dynamics Spot mobile robot used in a surveillance application. The case study presents the steps taken at each pipeline stage to obtain an RL agent to control the robot's position and orientation.

Real-Time Fast Marching Tree for Mobile Robot Motion Planning in Dynamic Environments

This paper proposes the Real-Time Fast Marching Tree (RT-FMT), a real-time planning algorithm that features local and global path generation, multiple-query planning, and dynamic obstacle avoidance. During the search, RT-FMT quickly looks for the global solution and, in the meantime, generates local paths that can be used by the robot to start execution faster. In addition, our algorithm constantly rewires the tree to keep branches from forming inside the dynamic obstacles and to maintain the tree root near the robot, which allows the tree to be reused multiple times for different goals. Our algorithm is based on the planners Fast Marching Tree (FMT*) and Real-time Rapidly-Exploring Random Tree (RT-RRT*). We show via simulations that RT-FMT outperforms RT- RRT* in both execution cost and arrival time, in most cases. Moreover, we also demonstrate via simulation that it is worthwhile taking the local path before the global path is available in order to reduce arrival time, even though there is a small possibility of taking an inferior path.

This is the Way: Mitigating the Roll of an Autonomous Uncrewed Surface Vessel in Wavy Conditions Using Model Predictive Control

Though larger vessels may be well-equipped to deal with wavy conditions, smaller vessels are often more susceptible to disturbances. This paper explores the development of a nonlinear model predictive control (NMPC) system for Uncrewed Surface Vessels (USVs) in wavy conditions to minimize average roll. The NMPC is based on a prediction method that uses information about the vessel's dynamics and an assumed wave model. This method is able to mitigate the roll of an under-actuated USV in a variety of conditions by adjusting the weights of the cost function. The results show a reduction of 39% of average roll with a tuned controller in conditions with 1.75-metre sinusoidal waves. A general and intuitive tuning strategy is established. This preliminary work is a proof of concept which sets the stage for the leveraging of wave prediction methodologies to perform planning and control in real time for USVs in real-world scenarios and field trials.

OtterROS: Picking and Programming an Uncrewed Surface Vessel for Experimental Field Robotics Research with ROS 2

There exist a wide range of options for field robotics research using ground and aerial mobile robots, but there are comparatively few robust and research-ready uncrewed surface vessels (USVs). This workshop paper starts with a snapshot of USVs currently available to the research community and then describes "OtterROS", an open source ROS 2 solution for the Otter USV. Field experiments using OtterROS are described, which highlight the utility of the Otter USV and the benefits of using ROS 2 in aquatic robotics research. For those interested in USV research, the paper details recommended hardware to run OtterROS and includes an example ROS 2 package using OtterROS, removing unnecessary non-recurring engineering from field robotics research activities.