Reinforcement Learning Based Oscillation Dampening: Scaling up Single-Agent RL algorithms to a 100 AV highway field operational test

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. We explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

A Middle Way to Traffic Enlightenment

This paper introduces a novel approach that seeks a middle ground for traffic control in multi-lane congestion, where prevailing traffic speeds are too fast, and speed recommendations designed to dampen traffic waves are too slow. Advanced controllers that modify the speed of an automated car for wave-dampening, eco-driving, or other goals, typically are designed with forward collision safety in mind. Our approach goes further, by considering how dangerous it can be for a controller to drive so slowly relative to prevailing traffic that it creates a significant issue for safety and comfort. This paper explores open-road scenarios where large gaps between prevailing speeds and desired speeds can exist, specifically when infrastructure-based variable speed limit systems are not strictly followed at all times by other drivers. Our designed, implemented, and deployed algorithm is able to follow variable speed limits when others also follow it, avoid collisions with vehicles ahead, and adapt to prevailing traffic when other motorists are traveling well above the posted speeds. The key is to reject unsafe speed recommendations from infrastructure-based traffic smoothing systems, based on real-time local traffic conditions observed by the vehicle under control. This solution is implemented and deployed on two control vehicles in heavy multi-lane highway congestion. The results include analysis from system design, and field tests that validate the system's performance using an existing Variable Speed Limit system as the external source for speed recommendations, and the on-board sensors of a stock Toyota Rav4 for inputs that estimate the prevailing speed of traffic around the vehicle under control.

Robustness Verification for Knowledge-Based Logic of Risky Driving Scenes

Many decision-making scenarios in modern life benefit from the decision support of artificial intelligence algorithms, which focus on a data-driven philosophy and automated programs or systems. However, crucial decision issues related to security, fairness, and privacy should consider more human knowledge and principles to supervise such AI algorithms to reach more proper solutions and to benefit society more effectively. In this work, we extract knowledge-based logic that defines risky driving formats learned from public transportation accident datasets, which haven't been analyzed in detail to the best of our knowledge. More importantly, this knowledge is critical for recognizing traffic hazards and could supervise and improve AI models in safety-critical systems. Then we use automated verification methods to verify the robustness of such logic. More specifically, we gather 72 accident datasets from Data.gov and organize them by state. Further, we train Decision Tree and XGBoost models on each state's dataset, deriving accident judgment logic. Finally, we deploy robustness verification on these tree-based models under multiple parameter combinations.

Enabling Mixed Autonomy Traffic Control

We demonstrate a new capability of automated vehicles: mixed autonomy traffic control. With this new capability, automated vehicles can shape the traffic flows composed of other non-automated vehicles, which has the promise to improve safety, efficiency, and energy outcomes in transportation systems at a societal scale. Investigating mixed autonomy mobile traffic control must be done in situ given that the complex dynamics of other drivers and their response to a team of automated vehicles cannot be effectively modeled. This capability has been blocked because there is no existing scalable and affordable platform for experimental control. This paper introduces an extensible open-source hardware and software platform, enabling a team of 100 vehicles to execute several different vehicular control algorithms as a collaborative fleet, composed of three different makes and models, which drove 22752 miles in a combined 1022 hours, over 5 days in Nashville, TN in November 2022.

SAILing CAVs: Speed-Adaptive Infrastructure-Linked Connected and Automated Vehicles

This work demonstrates a new capability in roadway control: Speed-adaptive, infrastructure-linked connected and automated vehicles. We develop and deploy a lightly modified vehicle that is able to dynamically adjust the vehicle speed in response to posted variable speed limit messages generated by the infrastructure using LTE connectivity. This work describes the open source hardware and software platform that enables integration between infrastructure-based variable posted speed limits, and existing vehicle platforms for automated control. The vehicle is deployed in heavy morning traffic on I-24 in Nashville, TN. The control vehicle follows the posted variable speed limits, resulting in as much as a 25% reduction in speed variability compared to a human-piloted vehicle in the same traffic stream.

So you think you can track?

This work introduces a multi-camera tracking dataset consisting of 234 hours of video data recorded concurrently from 234 overlapping HD cameras covering a 4.2 mile stretch of 8-10 lane interstate highway near Nashville, TN. The video is recorded during a period of high traffic density with 500+ objects typically visible within the scene and typical object longevities of 3-15 minutes. GPS trajectories from 270 vehicle passes through the scene are manually corrected in the video data to provide a set of ground-truth trajectories for recall-oriented tracking metrics, and object detections are provided for each camera in the scene (159 million total before cross-camera fusion). Initial benchmarking of tracking-by-detection algorithms is performed against the GPS trajectories, and a best HOTA of only 9.5% is obtained (best recall 75.9% at IOU 0.1, 47.9 average IDs per ground truth object), indicating the benchmarked trackers do not perform sufficiently well at the long temporal and spatial durations required for traffic scene understanding.

Prototyping Vehicle Control Applications Using the CAT Vehicle Simulator

This paper demonstrates the integration model-based design approaches or vehicle control, with validation in a freely available open-source simulator. Continued interest in autonomous vehicles and their deployment is driven by the potential benefits of their use. However, it can be challenging to transition new theoretical approaches into unknown simulation environments. Thus, it is critical for experts from other fields, whose insights may be necessary to continue to advance autonomy, to be able to create control applications with the potential to transition to practice. In this article, we will explain how to use the CAT Vehicle simulator and ROS packages to create and test vehicle controllers. The methodology of developing the control system in this article takes the approach of model-based design using Simulink, and the ROS Toolbox, followed by code generation to create a standalone C++ ROS node. Such ROS nodes can be integrated through roslaunch in the CAT Vehicle ROS package.

Reachability Analysis for FollowerStopper: Safety Analysis and Experimental Results

Motivated by earlier work and the developer of a new algorithm, the FollowerStopper, this article uses reachability analysis to verify the safety of the FollowerStopper algorithm, which is a controller designed for dampening stop- and-go traffic waves. With more than 1100 miles of driving data collected by our physical platform, we validate our analysis results by comparing it to human driving behaviors. The FollowerStopper controller has been demonstrated to dampen stop-and-go traffic waves at low speed, but previous analysis on its relative safety has been limited to upper and lower bounds of acceleration. To expand upon previous analysis, reachability analysis is used to investigate the safety at the speeds it was originally tested and also at higher speeds. Two formulations of safety analysis with different criteria are shown: distance-based and time headway-based. The FollowerStopper is considered safe with distance-based criterion. However, simulation results demonstrate that the FollowerStopper is not representative of human drivers - it follows too closely behind vehicles, specifically at a distance human would deem as unsafe. On the other hand, under the time headway-based safety analysis, the FollowerStopper is not considered safe anymore. A modified FollowerStopper is proposed to satisfy time-based safety criterion. Simulation results of the proposed FollowerStopper shows that its response represents human driver behavior better.

The CAT Vehicle Testbed: A Simulator with Hardware in the Loop for Autonomous Vehicle Applications

This paper presents the CAT Vehicle (Cognitive and Autonomous Test Vehicle) Testbed: a research testbed comprised of a distributed simulation-based autonomous vehicle, with straightforward transition to hardware in the loop testing and execution, to support research in autonomous driving technology. The evolution of autonomous driving technology from active safety features and advanced driving assistance systems to full sensor-guided autonomous driving requires testing of every possible scenario. However, researchers who want to demonstrate new results on a physical platform face difficult challenges, if they do not have access to a robotic platform in their own labs. Thus, there is a need for a research testbed where simulation-based results can be rapidly validated through hardware in the loop simulation, in order to test the software on board the physical platform. The CAT Vehicle Testbed offers such a testbed that can mimic dynamics of a real vehicle in simulation and then seamlessly transition to reproduction of use cases with hardware. The simulator utilizes the Robot Operating System (ROS) with a physics-based vehicle model, including simulated sensors and actuators with configurable parameters. The testbed allows multi-vehicle simulation to support vehicle to vehicle interaction. Our testbed also facilitates logging and capturing of the data in the real time that can be played back to examine particular scenarios or use cases, and for regression testing. As part of the demonstration of feasibility, we present a brief description of the CAT Vehicle Challenge, in which student researchers from all over the globe were able to reproduce their simulation results with fewer than 2 days of interfacing with the physical platform.