Reinforcement Learning Based Oscillation Dampening: Scaling up Single-Agent RL algorithms to a 100 AV highway field operational test

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. We explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

A Middle Way to Traffic Enlightenment

This paper introduces a novel approach that seeks a middle ground for traffic control in multi-lane congestion, where prevailing traffic speeds are too fast, and speed recommendations designed to dampen traffic waves are too slow. Advanced controllers that modify the speed of an automated car for wave-dampening, eco-driving, or other goals, typically are designed with forward collision safety in mind. Our approach goes further, by considering how dangerous it can be for a controller to drive so slowly relative to prevailing traffic that it creates a significant issue for safety and comfort. This paper explores open-road scenarios where large gaps between prevailing speeds and desired speeds can exist, specifically when infrastructure-based variable speed limit systems are not strictly followed at all times by other drivers. Our designed, implemented, and deployed algorithm is able to follow variable speed limits when others also follow it, avoid collisions with vehicles ahead, and adapt to prevailing traffic when other motorists are traveling well above the posted speeds. The key is to reject unsafe speed recommendations from infrastructure-based traffic smoothing systems, based on real-time local traffic conditions observed by the vehicle under control. This solution is implemented and deployed on two control vehicles in heavy multi-lane highway congestion. The results include analysis from system design, and field tests that validate the system's performance using an existing Variable Speed Limit system as the external source for speed recommendations, and the on-board sensors of a stock Toyota Rav4 for inputs that estimate the prevailing speed of traffic around the vehicle under control.

SAILing CAVs: Speed-Adaptive Infrastructure-Linked Connected and Automated Vehicles

This work demonstrates a new capability in roadway control: Speed-adaptive, infrastructure-linked connected and automated vehicles. We develop and deploy a lightly modified vehicle that is able to dynamically adjust the vehicle speed in response to posted variable speed limit messages generated by the infrastructure using LTE connectivity. This work describes the open source hardware and software platform that enables integration between infrastructure-based variable posted speed limits, and existing vehicle platforms for automated control. The vehicle is deployed in heavy morning traffic on I-24 in Nashville, TN. The control vehicle follows the posted variable speed limits, resulting in as much as a 25% reduction in speed variability compared to a human-piloted vehicle in the same traffic stream.

The CAT Vehicle Testbed: A Simulator with Hardware in the Loop for Autonomous Vehicle Applications

This paper presents the CAT Vehicle (Cognitive and Autonomous Test Vehicle) Testbed: a research testbed comprised of a distributed simulation-based autonomous vehicle, with straightforward transition to hardware in the loop testing and execution, to support research in autonomous driving technology. The evolution of autonomous driving technology from active safety features and advanced driving assistance systems to full sensor-guided autonomous driving requires testing of every possible scenario. However, researchers who want to demonstrate new results on a physical platform face difficult challenges, if they do not have access to a robotic platform in their own labs. Thus, there is a need for a research testbed where simulation-based results can be rapidly validated through hardware in the loop simulation, in order to test the software on board the physical platform. The CAT Vehicle Testbed offers such a testbed that can mimic dynamics of a real vehicle in simulation and then seamlessly transition to reproduction of use cases with hardware. The simulator utilizes the Robot Operating System (ROS) with a physics-based vehicle model, including simulated sensors and actuators with configurable parameters. The testbed allows multi-vehicle simulation to support vehicle to vehicle interaction. Our testbed also facilitates logging and capturing of the data in the real time that can be played back to examine particular scenarios or use cases, and for regression testing. As part of the demonstration of feasibility, we present a brief description of the CAT Vehicle Challenge, in which student researchers from all over the globe were able to reproduce their simulation results with fewer than 2 days of interfacing with the physical platform.