Are Open-Vocabulary Models Ready for Detection of MEP Elements on Construction Sites

The construction industry has long explored robotics and computer vision, yet their deployment on construction sites remains very limited. These technologies have the potential to revolutionize traditional workflows by enhancing accuracy, efficiency, and safety in construction management. Ground robots equipped with advanced vision systems could automate tasks such as monitoring mechanical, electrical, and plumbing (MEP) systems. The present research evaluates the applicability of open-vocabulary vision-language models compared to fine-tuned, lightweight, closed-set object detectors for detecting MEP components using a mobile ground robotic platform. A dataset collected with cameras mounted on a ground robot was manually annotated and analyzed to compare model performance. The results demonstrate that, despite the versatility of vision-language models, fine-tuned lightweight models still largely outperform them in specialized environments and for domain-specific tasks.

GNN-based Decentralized Perception in Multirobot Systems for Predicting Worker Actions

In industrial environments, predicting human actions is essential for ensuring safe and effective collaboration between humans and robots. This paper introduces a perception framework that enables mobile robots to understand and share information about human actions in a decentralized way. The framework first allows each robot to build a spatial graph representing its surroundings, which it then shares with other robots. This shared spatial data is combined with temporal information to track human behavior over time. A swarm-inspired decision-making process is used to ensure all robots agree on a unified interpretation of the human's actions. Results show that adding more robots and incorporating longer time sequences improve prediction accuracy. Additionally, the consensus mechanism increases system resilience, making the multi-robot setup more reliable in dynamic industrial settings.

Learning Multi-agent Multi-machine Tending by Mobile Robots

Robotics can help address the growing worker shortage challenge of the manufacturing industry. As such, machine tending is a task collaborative robots can tackle that can also highly boost productivity. Nevertheless, existing robotics systems deployed in that sector rely on a fixed single-arm setup, whereas mobile robots can provide more flexibility and scalability. In this work, we introduce a multi-agent multi-machine tending learning framework by mobile robots based on Multi-agent Reinforcement Learning (MARL) techniques with the design of a suitable observation and reward. Moreover, an attention-based encoding mechanism is developed and integrated into Multi-agent Proximal Policy Optimization (MAPPO) algorithm to boost its performance for machine tending scenarios. Our model (AB-MAPPO) outperformed MAPPO in this new challenging scenario in terms of task success, safety, and resources utilization. Furthermore, we provided an extensive ablation study to support our various design decisions.

Robotic deployment on construction sites: considerations for safety and productivity impact

Deploying mobile robots in construction sites to collaborate with workers or perform automated tasks such as surveillance and inspections carries the potential to greatly increase productivity, reduce human errors, and save costs. However ensuring human safety is a major concern, and the rough and dynamic construction environments pose multiple challenges for robot deployment. In this paper, we present the insights we obtained from our collaborations with construction companies in Canada and discuss our experiences deploying a semi-autonomous mobile robot in real construction scenarios.

From the Lab to the Theater: An Unconventional Field Robotics Journey

Artistic performances involving robotic systems present unique technical challenges akin to those encountered in other field deployments. In this paper, we delve into the orchestration of robotic artistic performances, focusing on the complexities inherent in communication protocols and localization methods. Through our case studies and experimental insights, we demonstrate the breadth of technical requirements for this type of deployment, and, most importantly, the significant contributions of working closely with non-experts.

Spherical Rolling Robots Design, Modeling, and Control: A Systematic Literature Review

Spherical robots have garnered increasing interest for their applications in exploration, tunnel inspection, and extraterrestrial missions. Diverse designs have emerged, including barycentric configurations, pendulum-based mechanisms, etc. In addition, a wide spectrum of control strategies has been proposed, ranging from traditional PID approaches to cutting-edge neural networks. Our systematic review aims to comprehensively identify and categorize locomotion systems and control schemes employed by spherical robots, spanning the years 1996 to 2023. A meticulous search across five databases yielded a dataset of 3189 records. As a result of our exhaustive analysis, we identified a collection of novel designs and control strategies. Leveraging the insights garnered, we provide valuable recommendations for optimizing the design and control aspects of spherical robots, supporting both novel design endeavors and the advancement of field deployments. Furthermore, we illuminate key research directions that hold the potential to unlock the full capabilities of spherical robots

See as a Bee: UV Sensor for Aerial Strawberry Crop Monitoring

Precision agriculture aims to use technological tools for the agro-food sector to increase productivity, cut labor costs, and reduce the use of resources. This work takes inspiration from bees vision to design a remote sensing system tailored to incorporate UV-reflectance into a flower detector. We demonstrate how this approach can provide feature-rich images for deep learning strawberry flower detection and we apply it to a scalable, yet cost effective aerial monitoring robotic system in the field. We also compare the performance of our UV-G-B image detector with a similar work that utilizes RGB images.

The eyes and hearts of UAV pilots: observations of physiological responses in real-life scenarios

The drone industry is diversifying and the number of pilots increases rapidly. In this context, flight schools need adapted tools to train pilots, most importantly with regard to their own awareness of their physiological and cognitive limits. In civil and military aviation, pilots can train themselves on realistic simulators to tune their reaction and reflexes, but also to gather data on their piloting behavior and physiological states. It helps them to improve their performances. Opposed to cockpit scenarios, drone teleoperation is conducted outdoor in the field, thus with only limited potential from desktop simulation training. This work aims to provide a solution to gather pilots behavior out in the field and help them increase their performance. We combined advance object detection from a frontal camera to gaze and heart-rate variability measurements. We observed pilots and analyze their behavior over three flight challenges. We believe this tool can support pilots both in their training and in their regular flight tasks. A demonstration video is available on https://www.youtube.com/watch?v=eePhjd2qNiI

Semantic navigation with domain knowledge

Several deployment locations of mobile robotic systems are human made (i.e. urban firefighter, building inspection, property security) and the manager may have access to domain-specific knowledge about the place, which can provide semantic contextual information allowing better reasoning and decision making. In this paper we propose a system that allows a mobile robot to operate in a location-aware and operator-friendly way, by leveraging semantic information from the deployment location and integrating it to the robots localization and navigation systems. We integrate Building Information Models (BIM) into the Robotic Operating System (ROS), to generate topological and metric maps fed to an layered path planner (global and local). A map merging algorithm integrates newly discovered obstacles into the metric map, while a UWB-based localization system detects equipment to be registered back into the semantic database. The results are validated in simulation and real-life deployments in buildings and construction sites.

Semantic Navigation Using Building Information on Construction Sites

With the growth in automated data collection of construction projects, the need for semantic navigation of mobile robots is increasing. In this paper, we propose an infrastructure to leverage building-related information for smarter, safer and more precise robot navigation during construction phase. Our use of Building Information Models (BIM) in robot navigation is twofold: (1) the intuitive semantic information enables non-experts to deploy robots and (2) the semantic data exposed to the navigation system allows optimal path planning (not necessarily the shortest one). Our Building Information Robotic System (BIRS) uses Industry Foundation Classes (IFC) as the interoperable data format between BIM and the Robotic Operating System (ROS). BIRS generates topological and metric maps from BIM for ROS usage. An optimal path planer, integrating critical components for construction assessment is proposed using a cascade strategy (global versus local). The results are validated through series of experiments in construction sites.