Q-learning Based System for Path Planning with UAV Swarms in Obstacle Environments

Path Planning methods for autonomous control of Unmanned Aerial Vehicle (UAV) swarms are on the rise because of all the advantages they bring. There are more and more scenarios where autonomous control of multiple UAVs is required. Most of these scenarios present a large number of obstacles, such as power lines or trees. If all UAVs can be operated autonomously, personnel expenses can be decreased. In addition, if their flight paths are optimal, energy consumption is reduced. This ensures that more battery time is left for other operations. In this paper, a Reinforcement Learning based system is proposed for solving this problem in environments with obstacles by making use of Q-Learning. This method allows a model, in this particular case an Artificial Neural Network, to self-adjust by learning from its mistakes and achievements. Regardless of the size of the map or the number of UAVs in the swarm, the goal of these paths is to ensure complete coverage of an area with fixed obstacles for tasks, like field prospecting. Setting goals or having any prior information aside from the provided map is not required. For experimentation, five maps of different sizes with different obstacles were used. The experiments were performed with different number of UAVs. For the calculation of the results, the number of actions taken by all UAVs to complete the task in each experiment is taken into account. The lower the number of actions, the shorter the path and the lower the energy consumption. The results are satisfactory, showing that the system obtains solutions in fewer movements the more UAVs there are. For a better presentation, these results have been compared to another state-of-the-art approach.

Robust Biped Locomotion Using Deep Reinforcement Learning on Top of an Analytical Control Approach

This paper proposes a modular framework to generate robust biped locomotion using a tight coupling between an analytical walking approach and deep reinforcement learning. This framework is composed of six main modules which are hierarchically connected to reduce the overall complexity and increase its flexibility. The core of this framework is a specific dynamics model which abstracts a humanoid's dynamics model into two masses for modeling upper and lower body. This dynamics model is used to design an adaptive reference trajectories planner and an optimal controller which are fully parametric. Furthermore, a learning framework is developed based on Genetic Algorithm (GA) and Proximal Policy Optimization (PPO) to find the optimum parameters and to learn how to improve the stability of the robot by moving the arms and changing its center of mass (COM) height. A set of simulations are performed to validate the performance of the framework using the official RoboCup 3D League simulation environment. The results validate the performance of the framework, not only in creating a fast and stable gait but also in learning to improve the upper body efficiency.

A CPG-Based Agile and Versatile Locomotion Framework Using Proximal Symmetry Loss

Humanoid robots are made to resemble humans but their locomotion abilities are far from ours in terms of agility and versatility. When humans walk on complex terrains, or face external disturbances, they combine a set of strategies, unconsciously and efficiently, to regain stability. This paper tackles the problem of developing a robust omnidirectional walking framework, which is able to generate versatile and agile locomotion on complex terrains. The Linear Inverted Pendulum Model and Central Pattern Generator concepts are used to develop a closed-loop walk engine that is combined with a reinforcement learning module. This module learns to regulate the walk engine parameters adaptively and generates residuals to adjust the robot's target joint positions (residual physics). Additionally, we propose a proximal symmetry loss to increase the sample efficiency of the Proximal Policy Optimization algorithm by leveraging model symmetries. The effectiveness of the proposed framework was demonstrated and evaluated across a set of challenging simulation scenarios. The robot was able to generalize what it learned in one scenario, by displaying human-like locomotion skills in unforeseen circumstances, even in the presence of noise and external pushes.

A Hybrid Biped Stabilizer System Based on Analytical Control and Learning of Symmetrical Residual Physics

Although humanoid robots are made to resemble humans, their stability is not yet comparable to ours. When facing external disturbances, humans efficiently and unconsciously combine a set of strategies to regain stability. This work deals with the problem of developing a robust hybrid stabilizer system for biped robots. The Linear Inverted Pendulum (LIP) and Divergent Component of Motion (DCM) concepts are used to formulate the biped locomotion and stabilization as an analytical control framework. On top of that, a neural network with symmetric partial data augmentation learns residuals to adjust the joint's position, and thus improving the robot's stability when facing external perturbations. The performance of the proposed framework was evaluated across a set of challenging simulation scenarios. The results show a considerable improvement over the baseline in recovering from large external forces. Moreover, the produced behaviors are human-like and robust to considerably noisy environments.

A Robust Model-Based Biped Locomotion Framework Based on Three-Mass Model: From Planning to Control

Biped robots are inherently unstable because of their complex kinematics as well as dynamics. Despite types of research in developing biped locomotion, the performance of biped locomotion is still far from the expectations. This paper proposes a model-based framework to generate stable biped locomotion. The core of this framework is an abstract dynamics model which is composed of three masses to consider the dynamics of stance leg, torso and swing leg for minimizing the tracking problems. According to this dynamics model, we propose a modular walking reference trajectories planner which takes into account obstacles to plan all the references. Moreover, this dynamics model is used to formulate the low-level controller as a Model Predictive Control~(MPC) scheme which can consider some constraints in the states of the system, inputs, outputs and also mixed input-output. The performance and the robustness of the proposed framework are validated by performing several simulations using~\mbox{MATLAB}. The simulation results show that the proposed framework is capable of generating the biped locomotion robustly.

A Hierarchical Framework to Generate Robust Biped Locomotion Based on Divergent Component of Motion

Keeping the stability can be counted as the essential ability of a humanoid robot to step out of the laboratory to work in our real environment. Since humanoid robots have similar kinematic to a human, humans expect these robots to be robustly capable of stabilizing even in a challenging situation like while a severe push is applied. This paper presents a robust walking framework which not only takes into account the traditional push recovery approaches (e.g., ankle, hip and step strategies) but also uses the concept of Divergent Component of the Motion (DCM) to adjust next step timing and location. The control core of the proposed framework is composed of a Linear-Quadratic-Gaussian (LQG) controller and two proportional controllers. In this framework, the LQG controller tries to track the reference trajectories and the proportional controllers are designed to adjust the next step timing and location that allow the robot to recover from a severe push. The robustness and the performance of the proposed framework have been validated by performing a set of simulations, including walking and push recovery using MATLAB. The simulation results verified that the proposed framework is capable of providing a robust walking even in very challenging situations.

A Robust Closed-Loop Biped Locomotion Planner Based on Time Varying Model Predictive Control

Developing robust locomotion for humanoid robots is a complex task due to the unstable nature of these robots and also to the unpredictability of the terrain. A robust locomotion planner is one of the fundamental components for generating stable biped locomotion. This paper presents an optimal closed-loop biped locomotion planner which can plan reference trajectories even in challenging conditions. The proposed planner is designed based on a Time-Varying Model Predictive Control~(TVMPC) scheme to be able to consider some constraints in the states, inputs and outputs of the system and also mixed input-output. Moreover, the proposed planner takes into account the vertical motion of the Center of Mass~(COM) to generate walking with mostly stretched knees which is more human-like. Additionally, the planner uses the concept of Divergent Component of Motion~(DCM) to modify the reference ZMP online to improve the withstanding level of the robot in the presence of severe disturbances. The performance and also the robustness of the proposed planner are validated by performing several simulations using~\mbox{MATLAB}. The simulation results show that the proposed planner is capable of generating the biped locomotion robustly.

A Fast and Stable Omnidirectional Walking Engine for the Nao Humanoid Robot

This paper proposes a framework designed to generate a closed-loop walking engine for a humanoid robot. In particular, the core of this framework is an abstract dynamics model which is composed of two masses that represent the lower and the upper body of a humanoid robot. Moreover, according to the proposed dynamics model, the low-level controller is formulated as a Linear-Quadratic-Gaussian (LQG) controller that is able to robustly track the desired trajectories. Besides, this framework is fully parametric which allows using an optimization algorithm to find the optimum parameters. To examine the performance of the proposed framework, a set of simulation using a simulated Nao robot in the RoboCup 3D simulation environment has been carried out. Simulation results show that the proposed framework is capable of providing fast and reliable omnidirectional walking. After optimizing the parameters using genetic algorithm (GA), the maximum forward walking velocity that we have achieved was $80.5cm/s$.

A Robust Biped Locomotion Based on Linear-Quadratic-Gaussian Controller and Divergent Component of Motion

Generating robust locomotion for a humanoid robot in the presence of disturbances is difficult because of its high number of degrees of freedom and its unstable nature. In this paper, we used the concept of Divergent Component of Motion~(DCM) and propose an optimal closed-loop controller based on Linear-Quadratic-Gaussian to generate a robust and stable walking for humanoid robots. The biped robot dynamics has been approximated using the Linear Inverted Pendulum Model~(LIPM). Moreover, we propose a controller to adjust the landing location of the swing leg to increase the withstanding level of the robot against a severe external push. The performance and also the robustness of the proposed controller is analyzed and verified by performing a set of simulations using~\mbox{MATLAB}. The simulation results showed that the proposed controller is capable of providing a robust walking even in the presence of disturbances and in challenging situations.

A Model-Based Balance Stabilization System for Biped Robot

This paper presents a model-based balance stabilization system which takes into account not only the stable part of COM dynamics but also the unstable part. In this system, the overall dynamics of a humanoid robot is approximated using a Linear Inverted Pendulum Plus Flywheel Model (LIPPFM). Moreover, Divergent Component of Motion~(DCM) is used to define when and where a robot should take a step to prevent falling. The proposed system has been successfully tested by performing several simulations using MATLAB. The simulation results show this system is capable of stabilizing the balance of the robot in various conditions.