RGB-D Semantic SLAM for Surgical Robot Navigation in the Operating Room

Gaining spatial awareness of the Operating Room (OR) for surgical robotic systems is a key technology that can enable intelligent applications aiming at improved OR workflow. In this work, we present a method for semantic dense reconstruction of the OR scene using multiple RGB-D cameras attached and registered to the da Vinci Xi surgical system. We developed a novel SLAM approach for robot pose tracking in dynamic OR environments and dense reconstruction of the static OR table object. We validated our techniques in a mock OR by collecting data sequences with corresponding optical tracking trajectories as ground truth and manually annotated 100 frame segmentation masks. The mean absolute trajectory error is $11.4\pm1.9$ mm and the mean relative pose error is $1.53\pm0.48$ degrees per second. The segmentation DICE score is improved from 0.814 to 0.902 by using our SLAM system compared to single frame. Our approach effectively produces a dense OR table reconstruction in dynamic clinical environments as well as improved semantic segmentation on individual image frames.

A Robotic 3D Perception System for Operating Room Environment Awareness

Purpose: We describe a 3D multi-view perception system for the da Vinci surgical system to enable Operating room (OR) scene understanding and context awareness. Methods: Our proposed system is comprised of four Time-of-Flight (ToF) cameras rigidly attached to strategic locations on the daVinci Xi patient side cart (PSC). The cameras are registered to the robot's kinematic chain by performing a one-time calibration routine and therefore, information from all cameras can be fused and represented in one common coordinate frame. Based on this architecture, a multi-view 3D scene semantic segmentation algorithm is created to enable recognition of common and salient objects/equipment and surgical activities in a da Vinci OR. Our proposed 3D semantic segmentation method has been trained and validated on a novel densely annotated dataset that has been captured from clinical scenarios. Results: The results show that our proposed architecture has acceptable registration error ($3.3\%\pm1.4\%$ of object-camera distance) and can robustly improve scene segmentation performance (mean Intersection Over Union - mIOU) for less frequently appearing classes ($\ge 0.013$) compared to a single-view method. Conclusion: We present the first dynamic multi-view perception system with a novel segmentation architecture, which can be used as a building block technology for applications such as surgical workflow analysis, automation of surgical sub-tasks and advanced guidance systems.

Preliminary Studies on Force/Motion Control of Intelligent Mechanical Systems

To rationalize the relatively high investment that industrial automation systems entail, research in the field of intelligent machines should target high value functions such as fettling, die-finishing, deburring, and fixtureless manufacturing. For achieving this goal, past work has concentrated on force control algorithms at the system level with limited focus on performance expansion at the actuator level. We present a comprehensive literature review on robot force control, including algorithms, specialized actuators, and robot control software. A robot force control testbed was developed using Schunk's PowerCube 6-DOF Arm and a six-axis ATI force/torque sensor. Using parameter identification experiments, manipulator module inertias and the motor torque constant were estimated. Experiments were conducted to study the practical issues involved in implementing stable contact transitions and programmable endpoint impedance. Applications to human augmentation, virtual fixtures, and teleoperation are discussed. These experiments are used as a vehicle to understand the performance improvement achievable at the actuator level. The approach at UTRRG has been to maximize the choices within the actuator to enhance its intelligence. Drawing on this 20-year research history in electromechanical actuator architecture, we propose a new concept that mixes two inputs, distinct in their velocity ratios, within the same dual actuator called a Force/Motion Actuator (FMA). Detailed kinematic and dynamic models of this dual actuator are developed. The actuator performance is evaluated using simulations with an output velocity specification and resolving input trajectories using a minimum-norm solution. It is shown that a design choice of 14:1 motion scaling between the two inputs results in good sensitivity to output force disturbances without compromising velocity tracking performance.

Study of the Dynamic Coupling Term in Parallel Force/Velocity Actuated Systems

Presented in this paper is an actuator concept, called a Parallel Force/Velocity Actuator (PFVA), that combines two fundamentally distinct actuators (one using low gear reduction or even direct drive, which we will call a Force Actuator (FA) and the other with a high reduction gear train that we will refer to as a Velocity Actuator (VA)). The objective of this work is to evaluate the effect of the relative scale factor, RSF, (ratio of gear reductions) between these inputs on their dynamic coupling. We conceptually describe a Parallel Force/Velocity Actuator (PFVA) based on a Dual-Input-Single- Output (DISO) epicyclic gear train. We then present an analytical formulation for the variation of the dynamic coupling term w.r.t. RSF. Conclusions from this formulation are illustrated through a numerical example involving a 1-DOF four-bar linkage. It is shown, both analytically and numerically, that as we increase the RSF, the two inputs to the PFVA are decoupled w.r.t. the inertia torques. This understanding can serve as an important design guideline for PFVAs. The paper also presents two limitations of this study and suggests future work based on these caveats.