Modeling Survival in model-based Reinforcement Learning

Although recent model-free reinforcement learning algorithms have been shown to be capable of mastering complicated decision-making tasks, the sample complexity of these methods has remained a hurdle to utilizing them in many real-world applications. In this regard, model-based reinforcement learning proposes some remedies. Yet, inherently, model-based methods are more computationally expensive and susceptible to sub-optimality. One reason is that model-generated data are always less accurate than real data, and this often leads to inaccurate transition and reward function models. With the aim to mitigate this problem, this work presents the notion of survival by discussing cases in which the agent's goal is to survive and its analogy to maximizing the expected rewards. To that end, a substitute model for the reward function approximator is introduced that learns to avoid terminal states rather than to maximize accumulated rewards from safe states. Focusing on terminal states, as a small fraction of state-space, reduces the training effort drastically. Next, a model-based reinforcement learning method is proposed (Survive) to train an agent to avoid dangerous states through a safety map model built upon temporal credit assignment in the vicinity of terminal states. Finally, the performance of the presented algorithm is investigated, along with a comparison between the proposed and current methods.

Exploration and Coordination of Complementary Multi-Robot Teams In a Hunter and Gatherer Scenario

This paper considers the problem of dynamic task allocation, where tasks are unknowingly distributed over an environment. We aim to address the multi-robot exploration aspect of the problem while solving the task-allocation aspect. To that end, we first propose a novel nature-inspired approach called "hunter and gatherer". We consider each task comprised of two sequential subtasks: detection and completion, where each subtask can only be carried out by a certain type of agent. Thus, this approach employs two complementary teams of agents: one agile in detecting (hunters) and another dexterous in completing (gatherers) the tasks. Then, we propose a multi-robot exploration algorithm for hunters and a multi-robot task allocation algorithm for gatherers, both in a distributed manner and based on innovative notions of "certainty and uncertainty profit margins". Statistical analysis of simulation results confirms the efficacy of the proposed algorithms. Besides, it is statistically proven that the proposed solutions function fairly, i.e. for each type of agent, the overall workload is distributed equally.

Multi-Agent Task Allocation in Complementary Teams: A Hunter and Gatherer Approach

Consider a dynamic task allocation problem, where tasks are unknowingly distributed over an environment. This paper considers each task comprised of two sequential subtasks: detection and completion, where each subtask can only be carried out by a certain type of agent. We address this problem using a novel nature-inspired approach called "hunter and gatherer". The proposed method employs two complementary teams of agents: one agile in detecting (hunters) and another dexterous in completing (gatherers) the tasks. To minimize the collective cost of task accomplishments in a distributed manner, a game-theoretic solution is introduced to couple agents from complementary teams. We utilize market-based negotiation models to develop incentive-based decision-making algorithms relying on innovative notions of "certainty and uncertainty profit margins". The simulation results demonstrate that employing two complementary teams of hunters and gatherers can effectually improve the number of tasks completed by agents compared to conventional methods, while the collective cost of accomplishments is minimized. In addition, the stability and efficacy of the proposed solutions are studied using Nash equilibrium analysis and statistical analysis respectively. It is also numerically shown that the proposed solutions function fairly, i.e. for each type of agent, the overall workload is distributed equally.

Design, Modeling, and Control of Norma: a Slider & Pendulum-Driven Spherical Robot

This paper discusses the design, modeling, and control of Norma, a novel 2 DOF mobile spherical robot (SR). The propelling mechanism of this robot consists of two actuators: a slider, and a rotational pendulum located on the SR's diagonal shaft. The slider can translate along the shaft and shift the robot's center of gravity towards the robot's sides. The pendulum rotates around the shaft to propel the SR to roll forward and backward. These two actuators enable the SR to perform both rolling and turning maneuvers as a nonholonomic robot. The advantage of the proposed mechanical design lies in its convenience of physical implementation, agility, and accurate mathematical model. The Euler Lagrange approach is utilized to derive the dynamics of the proposed mechanical structure using minimum simplifications possible. Further, a path tracking control scheme is introduced for a smooth trajectory. Finally, simulations are carried out in MATLAB to verify the accuracy of the mathematical model and the effectiveness of the controller against experimental results.

Kinematics of Spherical Robots Rolling Over 3D Terrains

Although the kinematics and dynamics of spherical robots (SRs) on flat horizontal and inclined 2D surfaces are thoroughly investigated, their rolling behavior on generic 3D terrains has remained unexplored. This paper derives the kinematics equations of the most common SRs configurations rolling over 3D surfaces. First, the kinematics equations for a geometrical sphere rolling over a 3D surface are derived along with the characterization of the modeling method. Next, a brief review of current SR mechanical configurations is presented as well as a novel classification for spherical robots based on their kinematics. Then, considering the mechanical constraints of each category, the kinematics equations for each group of spherical robots are presented. Afterwards, a path tracking method is utilized for a desired 3D trajectory. Finally, simulations are carried out to validate the developed models and the effectiveness of the proposed control scheme.