RAVE Checklist: Recommendations for Overcoming Challenges in Retrospective Safety Studies of Automated Driving Systems

The public, regulators, and domain experts alike seek to understand the effect of deployed SAE level 4 automated driving system (ADS) technologies on safety. The recent expansion of ADS technology deployments is paving the way for early stage safety impact evaluations, whereby the observational data from both an ADS and a representative benchmark fleet are compared to quantify safety performance. In January 2024, a working group of experts across academia, insurance, and industry came together in Washington, DC to discuss the current and future challenges in performing such evaluations. A subset of this working group then met, virtually, on multiple occasions to produce this paper. This paper presents the RAVE (Retrospective Automated Vehicle Evaluation) checklist, a set of fifteen recommendations for performing and evaluating retrospective ADS performance comparisons. The recommendations are centered around the concepts of (1) quality and validity, (2) transparency, and (3) interpretation. Over time, it is anticipated there will be a large and varied body of work evaluating the observed performance of these ADS fleets. Establishing and promoting good scientific practices benefits the work of stakeholders, many of whom may not be subject matter experts. This working group's intentions are to: i) strengthen individual research studies and ii) make the at-large community more informed on how to evaluate this collective body of work.

A Flexible Cell Classification for ML Projects in Jupyter Notebooks

Jupyter Notebook is an interactive development environment commonly used for rapid experimentation of machine learning (ML) solutions. Describing the ML activities performed along code cells improves the readability and understanding of Notebooks. Manual annotation of code cells is time-consuming and error-prone. Therefore, tools have been developed that classify the cells of a notebook concerning the ML activity performed in them. However, the current tools are not flexible, as they work based on look-up tables that have been created, which map function calls of commonly used ML libraries to ML activities. These tables must be manually adjusted to account for new or changed libraries. This paper presents a more flexible approach to cell classification based on a hybrid classification approach that combines a rule-based and a decision tree classifier. We discuss the design rationales and describe the developed classifiers in detail. We implemented the new flexible cell classification approach in a tool called JupyLabel. Its evaluation and the obtained metric scores regarding precision, recall, and F1-score are discussed. Additionally, we compared JupyLabel with HeaderGen, an existing cell classification tool. We were able to show that the presented flexible cell classification approach outperforms this tool significantly.

Kinematic Characterization of Micro-Mobility Vehicles During Evasive Maneuvers

There is an increasing need to comprehensively characterize the kinematic performances of different Micromobility Vehicles (MMVs). This study aims to: 1) characterize the kinematic behaviors of different MMVs during emergency maneuvers; 2) explore the influence of different MMV power sources on the device performances; 3) investigate if piecewise linear models are suitable for modeling MMV trajectories. A test track experiment where 40 frequent riders performed emergency braking and swerving maneuvers riding a subset of electric MMVs, their traditional counterparts, and, in some cases, behaving as running pedestrians. A second experiment was conducted to determine the MMVs swerving lower boundaries. Device power source resulted having a statistically significant influence on kinematic capabilities of the MMVs: while e-MMVs displayed superior braking capabilities compared to their traditional counterparts, the opposite was observed in terms of swerving performance. Furthermore, performances varied significantly across the different MMV typologies, with handlebar-based devices consistently outperforming the handlebar-less devices across the metrics considered. The piecewise linear models used for braking profiles fit well for most MMVs, except for skateboards and pedestrians due to foot-ground engagement. These findings underscore that the effectiveness of steering or braking in preventing collisions may vary depending on the type and power source of the device. This study also demonstrates the applicability of piecewise linear models for generating parameterized functions that accurately model braking trajectories, providing a valuable resource for automated systems developers. The model, however, also reveals that the single brake ramp assumption does not apply for certain types of MMVs or for pedestrians, indicating the necessity for further improvements.