Enhancing Pavement Sensor Data Acquisition for AI-Driven Transportation Research

Effective strategies for sensor data management are essential for advancing transportation research, especially in the current data-driven era, due to the advent of novel applications in artificial intelligence. This paper presents comprehensive guidelines for managing transportation sensor data, encompassing both archived static data and real-time data streams. The real-time system architecture integrates various applications with data acquisition systems (DAQ). By deploying the in-house designed, open-source Avena software platform alongside the NATS messaging system as a secure communication broker, reliable data exchange is ensured. While robust databases like TimescaleDB facilitate organized storage, visualization platforms like Grafana provide real-time monitoring capabilities. In contrast, static data standards address the challenges in handling unstructured, voluminous datasets. The standards advocate for a combination of cost-effective bulk cloud storage for unprocessed sensor data and relational databases for recording summarized analyses. They highlight the role of cloud data transfer tools like FME for efficient migration of sensor data from local storages onto the cloud. Further, integration of robust visualization tools into the framework helps in deriving patterns and trends from these complex datasets. The proposals were applied to INDOT's real-world case studies involving the I-65 and I-69 Greenfield districts. For real-time data collection, Campbell Scientific DAQ systems were used, enabling continuous generation and monitoring of sensor metrics. In the case of the archived I-69 database, summary data was compiled in Oracle, while the unprocessed data was stored in SharePoint. The results underline the effectiveness of the proposed guidelines and motivate their adoption in research projects.