Precise Workcell Sketching from Point Clouds Using an AR Toolbox

Capturing real-world 3D spaces as point clouds is efficient and descriptive, but it comes with sensor errors and lacks object parametrization. These limitations render point clouds unsuitable for various real-world applications, such as robot programming, without extensive post-processing (e.g., outlier removal, semantic segmentation). On the other hand, CAD modeling provides high-quality, parametric representations of 3D space with embedded semantic data, but requires manual component creation that is time-consuming and costly. To address these challenges, we propose a novel solution that combines the strengths of both approaches. Our method for 3D workcell sketching from point clouds allows users to refine raw point clouds using an Augmented Reality (AR) interface that leverages their knowledge and the real-world 3D environment. By utilizing a toolbox and an AR-enabled pointing device, users can enhance point cloud accuracy based on the device's position in 3D space. We validate our approach by comparing it with ground truth models, demonstrating that it achieves a mean error within 1cm - significant improvement over standard LiDAR scanner apps.

RobotGraffiti: An AR tool for semi-automated construction of workcell models to optimize robot deployment

Improving robot deployment is a central step towards speeding up robot-based automation in manufacturing. A main challenge in robot deployment is how to best place the robot within the workcell. To tackle this challenge, we combine two knowledge sources: robotic knowledge of the system and workcell context awareness of the user, and intersect them with an Augmented Reality interface. RobotGraffiti is a unique tool that empowers the user in robot deployment tasks. One simply takes a 3D scan of the workcell with their mobile device, adds contextual data points that otherwise would be difficult to infer from the system, and receives a robot base position that satisfies the automation task. The proposed approach is an alternative to expensive and time-consuming digital twins, with a fast and easy-to-use tool that focuses on selected workcell features needed to run the placement optimization algorithm. The main contributions of this paper are the novel user interface for robot base placement data collection and a study comparing the traditional offline simulation with our proposed method. We showcase the method with a robot base placement solution and obtain up to 16 times reduction in time.

Enabling Waypoint Generation for Collaborative Robots using LLMs and Mixed Reality

Programming a robotic is a complex task, as it demands the user to have a good command of specific programming languages and awareness of the robot's physical constraints. We propose a framework that simplifies robot deployment by allowing direct communication using natural language. It uses large language models (LLM) for prompt processing, workspace understanding, and waypoint generation. It also employs Augmented Reality (AR) to provide visual feedback of the planned outcome. We showcase the effectiveness of our framework with a simple pick-and-place task, which we implement on a real robot. Moreover, we present an early concept of expressive robot behavior and skill generation that can be used to communicate with the user and learn new skills (e.g., object grasping).

AR Training App for Energy Optimal Programming of Cobots

Worldwide most factories aim for low-cost and fast production ignoring resources and energy consumption. But, high revenues have been accompanied by environmental degradation. The United Nations reacted to the ecological problem and proposed the Sustainable Development Goals, and one of them is Sustainable Production (Goal 12). In addition, the participation of lightweight robots, such as collaborative robots, in modern industrial production is increasing. The energy consumption of a single collaborative robot is not significant, however, the consumption of more and more cobots worldwide is representative. Consequently, our research focuses on strategies to reduce the energy consumption of lightweight robots aiming for sustainable production. Firstly, the energy consumption of the lightweight robot UR10e is assessed by a set of experiments. We analyzed the results of the experiments to describe the relationship between the energy consumption and the evaluation parameters, thus paving the way to optimization strategies. Next, we propose four strategies to reduce energy consumption: 1) optimal standby position, 2) optimal robot instruction, 3) optimal motion time, and 4) reduction of dissipative energy. The results show that cobots potentially reduce from 3\% up to 37\% of their energy consumption, depending on the optimization technique. To disseminate the results of our research, we developed an AR game in which the users learn how to energy-efficiently program cobots.

Designing Internet of Behaviors Systems

The Internet of Behaviors (IoB) puts human behavior at the core of engineering intelligent connected systems. IoB links the digital world to human behavior to establish human-driven design, development, and adaptation processes. This paper defines the novel concept by an IoB model based on a collective effort interacting with software engineers, human-computer interaction scientists, social scientists, and cognitive science communities. The model for IoB is created based on an exploratory study that synthesizes state-of-the-art analysis and experts interviews. The architecture of a real industry 4.0 manufacturing infrastructure helps to explain the IoB model and it's application. The conceptual model was used to successfully implement a socio-technical infrastructure for a crowd monitoring and queue management system for the Uffizi Galleries, Florence, Italy. The experiment, which started in the fall of 2016 and was operational in the fall of 2018, used a data-driven approach to feed the system with real-time sensory data. It also incorporated prediction models on visitors' mobility behavior. The system's main objective was to capture human behavior, model it, and build a mechanism that considers changes, adapts in real-time, and continuously learns from repetitive behaviors. In addition to the conceptual model and the real-life evaluation, this paper provides recommendations from experts and gives future directions for IoB to become a significant technological advancement in the coming few years.