Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning

Entropy-based objectives are widely used to perform state space exploration in reinforcement learning (RL) and dataset generation for offline RL. Behavioral entropy (BE), a rigorous generalization of classical entropies that incorporates cognitive and perceptual biases of agents, was recently proposed for discrete settings and shown to be a promising metric for robotic exploration problems. In this work, we propose using BE as a principled exploration objective for systematically generating datasets that provide diverse state space coverage in complex, continuous, potentially high-dimensional domains. To achieve this, we extend the notion of BE to continuous settings, derive tractable $k$-nearest neighbor estimators, provide theoretical guarantees for these estimators, and develop practical reward functions that can be used with standard RL methods to learn BE-maximizing policies. Using standard MuJoCo environments, we experimentally compare the performance of offline RL algorithms for a variety of downstream tasks on datasets generated using BE, R\'{e}nyi, and Shannon entropy-maximizing policies, as well as the SMM and RND algorithms. We find that offline RL algorithms trained on datasets collected using BE outperform those trained on datasets collected using Shannon entropy, SMM, and RND on all tasks considered, and on 80% of the tasks compared to datasets collected using R\'{e}nyi entropy.

Robotic Exploration using Generalized Behavioral Entropy

This work presents and evaluates a novel strategy for robotic exploration that leverages human models of uncertainty perception. To do this, we introduce a measure of uncertainty that we term ``Behavioral entropy'', which builds on Prelec's probability weighting from Behavioral Economics. We show that the new operator is an admissible generalized entropy, analyze its theoretical properties and compare it with other common formulations such as Shannon's and Renyi's. In particular, we discuss how the new formulation is more expressive in the sense of measures of sensitivity and perceptiveness to uncertainty introduced here. Then we use Behavioral entropy to define a new type of utility function that can guide a frontier-based environment exploration process. The approach's benefits are illustrated and compared in a Proof-of-Concept and ROS-unity simulation environment with a Clearpath Warthog robot. We show that the robot equipped with Behavioral entropy explores faster than Shannon and Renyi entropies.

Robot Navigation in Risky, Crowded Environments: Understanding Human Preferences

Risky and crowded environments (RCE) contain abstract sources of risk and uncertainty, which are perceived differently by humans, leading to a variety of behaviors. Thus, robots deployed in RCEs, need to exhibit diverse perception and planning capabilities in order to interpret other human agents' behavior and act accordingly in such environments. To understand this problem domain, we conducted a study to explore human path choices in RCEs, enabling better robotic navigational explainable AI (XAI) designs. We created a novel COVID-19 pandemic grocery shopping scenario which had time-risk tradeoffs, and acquired users' path preferences. We found that participants showcase a variety of path preferences: from risky and urgent to safe and relaxed. To model users' decision making, we evaluated three popular risk models (Cumulative Prospect Theory (CPT), Conditional Value at Risk (CVAR), and Expected Risk (ER). We found that CPT captured people's decision making more accurately than CVaR and ER, corroborating theoretical results that CPT is more expressive and inclusive than CVaR and ER. We also found that people's self assessments of risk and time-urgency do not correlate with their path preferences in RCEs. Finally, we conducted thematic analysis of open-ended questions, providing crucial design insights for robots is RCE. Thus, through this study, we provide novel and critical insights about human behavior and perception to help design better navigational explainable AI (XAI) in RCEs.

Risk-perception-aware control design under dynamic spatial risks

This work proposes a novel risk-perception-aware (RPA) control design using non-rational perception of risks associated with uncertain dynamic spatial costs. We use Cumulative Prospect Theory (CPT) to model the risk perception of a decision maker (DM) and use it to construct perceived risk functions that transform the uncertain dynamic spatial cost to deterministic perceived risks of a DM. These risks are then used to build safety sets which can represent risk-averse to risk-insensitive perception. We define a notions of "inclusiveness" and "versatility" based on safety sets and use it to compare with other models such as Conditional value at Risk (CVaR) and Expected risk (ER). We theoretically prove that CPT is the most "inclusive" and "versatile" model of the lot in the context of risk-perception-aware controls. We further use the perceived risk function along with ideas from control barrier functions (CBF) to construct a class of perceived risk CBFs. For a class of truncated-Gaussian costs, we find sufficient geometric conditions for the validity of this class of CBFs, thus guaranteeing safety. Then, we generate perceived-safety-critical controls using a Quadratic program (QP) to guide an agent safely according to a given perceived risk model. We present simulations in a 2D environment to illustrate the performance of the proposed controller.

Planning under risk and uncertainty based on Prospect-theoretic models

In this work, we develop a novel sampling-based motion planing approach to generate plans in a risky and uncertain environment. To model a variety of risk-sensitivity profiles, we propose an adaption of Cumulative Prospect Theory (CPT) to the setting of path planning. This leads to the definition of a non-rational continuous cost envelope (as well as a continuous uncertainty envelope) associated with an obstacle environment. We use these metrics along with standard costs like path length to formulate path planning problems. Building on RRT*, we then develop a sampling-based motion planner that generates desirable paths from the perspective of a given risk sensitive profile. Since risk sensitivity can greatly vary, we provide a tuning knob to appease a diversity of decision makers (DM), ranging from totally risk-averse to risk-indifferent. Additionally, we adapt a Simultaneous Perturbation Stochastic Approximation (SPSA)-based algorithm to learn the CPT parameters that can best represent a certain DM. Simulations are presented in a 2D environment to evaluate the modeling approach and algorithm's performance.

Gesture based Human-Swarm Interactions for Formation Control using interpreters

We propose a novel Human-Swarm Interaction (HSI) framework which enables the user to control a swarm shape and formation. The user commands the swarm utilizing just arm gestures and motions which are recorded by an off-the-shelf wearable armband. We propose a novel interpreter system, which acts as an intermediary between the user and the swarm to simplify the user's role in the interaction. The interpreter takes in a high level input drawn using gestures by the user, and translates it into low level swarm control commands. This interpreter employs machine learning, Kalman filtering and optimal control techniques to translate the user input into swarm control parameters. A notion of Human Interpretable dynamics is introduced, which is used by the interpreter for planning as well as to provide feedback to the user. The dynamics of the swarm are controlled using a novel decentralized formation controller based on distributed linear iterations and dynamic average consensus. The framework is demonstrated theoretically as well as experimentally in a 2D environment, with a human controlling a swarm of simulated robots in real time.

Brain-Swarm Interface : Controlling a Swarm of Robots with Brain and Eye Signals from an EEG Headset

This work presents a novel marriage of Swarm Robotics and Brain Computer Interface technology to produce an interface which connects a user to a swarm of robots. The proposed interface enables the user to control the swarm's size and motion employing just thoughts and eye movements. The thoughts and eye movements are recorded as electrical signals from the scalp by an off-the-shelf Electroencephalogram (EEG) headset. Signal processing techniques are used to filter out noise and decode the user's eye movements from raw signals, while a Hidden Markov Model technique is employed to decipher the user's thoughts from filtered signals. The dynamics of the robots are controlled using a swarm controller based on potential fields. The shape and motion parameters of the potential fields are modulated by the human user through the brain-swarm interface to move the robots. The method is demonstrated experimentally with a human controlling a swarm of three M3pi robots in a laboratory environment, as well as controlling a swarm of 128 robots in a computer simulation.