ETHcavation: A Dataset and Pipeline for Panoptic Scene Understanding and Object Tracking in Dynamic Construction Environments

Construction sites are challenging environments for autonomous systems due to their unstructured nature and the presence of dynamic actors, such as workers and machinery. This work presents a comprehensive panoptic scene understanding solution designed to handle the complexities of such environments by integrating 2D panoptic segmentation with 3D LiDAR mapping. Our system generates detailed environmental representations in real-time by combining semantic and geometric data, supported by Kalman Filter-based tracking for dynamic object detection. We introduce a fine-tuning method that adapts large pre-trained panoptic segmentation models for construction site applications using a limited number of domain-specific samples. For this use case, we release a first-of-its-kind dataset of 502 hand-labeled sample images with panoptic annotations from construction sites. In addition, we propose a dynamic panoptic mapping technique that enhances scene understanding in unstructured environments. As a case study, we demonstrate the system's application for autonomous navigation, utilizing real-time RRT* for reactive path planning in dynamic scenarios. The dataset (https://leggedrobotics.github.io/panoptic-scene-understanding.github.io/) and code (https://github.com/leggedrobotics/rsl_panoptic_mapping) for training and deployment are publicly available to support future research.

HEAP -- The autonomous walking excavator

The demand and the potential for automation in the construction sector is unmatched, particularly for increasing environmental sustainability, improving worker safety and reducing labor shortages. We have developed an autonomous walking excavator - based one of the most versatile machines found on construction sites - as one way to begin fulfilling this potential. This article describes the process of converting an off-the-shelf construction machine into an autonomous robotic system. First we outline the necessary sensing equipment for full autonomy and the novel actuation of the legs, and compare three different complementary actuation principles for the excavator's arm. Second, we solve the state estimation problem for a general wheeled-legged robot. Beside kinematic measurements, it includes GNSS-RTK, to absolutely reference the machine on a construction site. Third, we developed individual controllers for driving, chassis balancing and arm motions allowing for fully autonomous operation. Lastly, we highlight the machine's potential in four different real-world applications, e.g. autonomous trench digging, autonomous assembly of dry stone walls, autonomous forestry work and semi-autonomous teleoperation. On top, we also share some development insights and possible future research directions.

Towards Autonomous Robotic Precision Harvesting

This paper presents an integrated system for performing precision harvesting missions using a walking harvester. Our harvester performs the challenging task of autonomous navigation and tree grabbing in a confined, GPS denied forest environment. Strategies for mapping, localization, planning, and control are proposed and integrated into a fully autonomous system. The mission starts with a human or a mobile robot mapping the area of interest using a custom-made sensor module. Subsequently, a human expert or a data-supported algorithm selects the trees for harvesting. The sensor module is then mounted on the machine and used for localization within the given map. A planning algorithm searches for both an approach pose and a path in a single path planning problem. We design a path following controller leveraging the walking harvester's capabilities for negotiating rough terrain. Upon reaching the approach pose, the machine grabs a tree with a general-purpose gripper. This process repeats for all the trees selected by the operator (algorithm). Our system has been tested on a testing field with tree trunks and in a natural forest. To the best of our knowledge, this is the first time this level of autonomy has been shown on a full-size hydraulic machine operating in a realistic environment.

Combined Sampling and Optimization Based Planning for Legged-Wheeled Robots

Planning for legged-wheeled machines is typically done using trajectory optimization because of many degrees of freedom, thus rendering legged-wheeled planners prone to falling prey to bad local minima. We present a combined sampling and optimization-based planning approach that can cope with challenging terrain. The sampling-based stage computes whole-body configurations and contact schedule, which speeds up the optimization convergence. The optimization-based stage ensures that all the system constraints, such as non-holonomic rolling constraints, are satisfied. The evaluations show the importance of good initial guesses for optimization. Furthermore, they suggest that terrain/collision (avoidance) constraints are more challenging than the robot model's constraints. Lastly, we extend the optimization to handle general terrain representations in the form of elevation maps.