Reinforcement Learning Control for Autonomous Hydraulic Material Handling Machines with Underactuated Tools

The precise and safe control of heavy material handling machines presents numerous challenges due to the hard-to-model hydraulically actuated joints and the need for collision-free trajectory planning with a free-swinging end-effector tool. In this work, we propose an RL-based controller that commands the cabin joint and the arm simultaneously. It is trained in a simulation combining data-driven modeling techniques with first-principles modeling. On the one hand, we employ a neural network model to capture the highly nonlinear dynamics of the upper carriage turn hydraulic motor, incorporating explicit pressure prediction to handle delays better. On the other hand, we model the arm as velocity-controllable and the free-swinging end-effector tool as a damped pendulum using first principles. This combined model enhances our simulation environment, enabling the training of RL controllers that can be directly transferred to the real machine. Designed to reach steady-state Cartesian targets, the RL controller learns to leverage the hydraulic dynamics to improve accuracy, maintain high speeds, and minimize end-effector tool oscillations. Our controller, tested on a mid-size prototype material handler, is more accurate than an inexperienced operator and causes fewer tool oscillations. It demonstrates competitive performance even compared to an experienced professional driver.

Dynamic Throwing with Robotic Material Handling Machines

Automation of hydraulic material handling machinery is currently limited to semi-static pick-and-place cycles. Dynamic throwing motions which utilize the passive joints, can greatly improve time efficiency as well as increase the dumping workspace. In this work, we use Reinforcement Learning (RL) to design dynamic controllers for material handlers with underactuated arms as commonly used in logistics. The controllers are tested both in simulation and in real-world experiments on a 12-ton test platform. The method is able to exploit the passive joints of the gripper to perform dynamic throwing motions. With the proposed controllers, the machine is able to throw individual objects to targets outside the static reachability zone with good accuracy for its practical applications. The work demonstrates the possibility of using RL to perform highly dynamic tasks with heavy machinery, suggesting a potential for improving the efficiency and precision of autonomous material handling tasks.

A Unified and Modular Model Predictive Control Framework for Soft Continuum Manipulators under Internal and External Constraints

The emerging field of fluidically actuated soft robotic control has promising capabilities such as inherent compliance and user safety. However, these are counterbalanced by issues not common to rigid robots, like nonlinear actuation dynamics, motion constraints, workspace limitations, and variable shape stiffness. In this work, we have adapted Model Predictive Control (MPC), that has recently seen an exponential rise in popularity and fields of applications, to a soft robotic arm called SoPrA. We have addressed the problems that current control methods are facing, trying to propose a unique environment to handle them in a modular way. This work shows, both with simulation and experimental results, that Task-Space MPC can be successfully implemented for dynamic soft robotic control, while past research has usually focused on Joint-Space references. We have provided a way to couple the Piece-wise Constant Curvature and the Augmented Rigid Body Model assumptions with internal and external constraints and actuation dynamics, delivering an algorithm that can manage all these information and optimize over them. We believe that an MPC implementation based on our approach could be the way to address most of model-based soft robotics control issues within a unified and modular framework, while allowing to include improvements that usually belong to other control domains such as learning techniques.