Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariants

Recognizing surgical gestures in real-time is a stepping stone towards automated activity recognition, skill assessment, intra-operative assistance, and eventually surgical automation. The current robotic surgical systems provide us with rich multi-modal data such as video and kinematics. While some recent works in multi-modal neural networks learn the relationships between vision and kinematics data, current approaches treat kinematics information as independent signals, with no underlying relation between tool-tip poses. However, instrument poses are geometrically related, and the underlying geometry can aid neural networks in learning gesture representation. Therefore, we propose combining motion invariant measures (curvature and torsion) with vision and kinematics data using a relational graph network to capture the underlying relations between different data streams. We show that gesture recognition improves when combining invariant signals with tool position, achieving 90.3\% frame-wise accuracy on the JIGSAWS suturing dataset. Our results show that motion invariant signals coupled with position are better representations of gesture motion compared to traditional position and quaternion representations. Our results highlight the need for geometric-aware modeling of kinematics for gesture recognition.

Sensorless Remote Center of Motion Misalignment Estimation

Laparoscopic surgery constrains instrument motion around a fixed pivot point at the incision into a patient to minimize tissue trauma. Surgical robots achieve this through either hardware to software-based remote center of motion (RCM) constraints. However, accurate RCM alignment is difficult due to manual trocar placement, patient motion, and tissue deformation. Misalignment between the robot's RCM point and the patient incision site can cause unsafe forces at the incision site. This paper presents a sensorless force estimation-based framework for dynamically assessing and optimizing RCM misalignment in robotic surgery. Our experiments demonstrate that misalignment exceeding 20 mm can generate large enough forces to potentially damage tissue, emphasizing the need for precise RCM positioning. For misalignment $D\geq $ 20 mm, our optimization algorithm estimates the RCM offset with an absolute error within 5 mm. Accurate RCM misalignment estimation is a step toward automated RCM misalignment compensation, enhancing safety and reducing tissue damage in robotic-assisted laparoscopic surgery.

A Feasibility Study of a Soft, Low-Cost, 6-Axis Load Cell for Haptics

Haptic devices have shown to be valuable in supplementing surgical training, especially when providing haptic feedback based on user performance metrics such as wrench applied by the user on the tool. However, current 6-axis force/torque sensors are prohibitively expensive. This paper presents the design and calibration of a low-cost, six-axis force/torque sensor specially designed for laparoscopic haptic training applications. The proposed design uses Hall-effect sensors to measure the change in the position of magnets embedded in a silicone layer that results from an applied wrench to the device. Preliminary experimental validation demonstrates that these sensors can achieve an accuracy of 0.45 N and 0.014 Nm, and a theoretical XY range of +/-50N, Z range of +/-20N, and torque range of +/-0.2Nm. This study indicates that the proposed low-cost 6-axis force/torque sensor can accurately measure user force and provide useful feedback during laparoscopic training on a haptic device.

Endovascular Detection of Catheter-Thrombus Contact by Vacuum Excitation

Objective: The objective of this work is to introduce and demonstrate the effectiveness of a novel sensing modality for contact detection between an off-the-shelf aspiration catheter and a thrombus. Methods: A custom robotic actuator with a pressure sensor was used to generate an oscillatory vacuum excitation and sense the pressure inside the extracorporeal portion of the catheter. Vacuum pressure profiles and robotic motion data were used to train a support vector machine (SVM) classification model to detect contact between the aspiration catheter tip and a mock thrombus. Validation consisted of benchtop accuracy verification, as well as user study comparison to the current standard of angiographic presentation. Results: Benchtop accuracy of the sensing modality was shown to be 99.67%. The user study demonstrated statistically significant improvement in identifying catheter-thrombus contact compared to the current standard. The odds ratio of successful detection of clot contact was 2.86 (p=0.03) when using the proposed sensory method compared to without it. Conclusion: The results of this work indicate that the proposed sensing modality can offer intraoperative feedback to interventionalists that can improve their ability to detect contact between the distal tip of a catheter and a thrombus. Significance: By offering a relatively low-cost technology that affords off-the-shelf aspiration catheters as clot-detecting sensors, interventionalists can improve the first-pass effect of the mechanical thrombectomy procedure while reducing procedural times and mental burden.

Design Considerations and Robustness to Parameter Uncertainty in Wire-Wrapped Cam Mechanisms

Collaborative robots must simultaneously be safe enough to operate in close proximity to human operators and powerful enough to assist users in industrial tasks such as lifting heavy equipment. The requirement for safety necessitates that collaborative robots are designed with low-powered actuators. However, some industrial tasks may require the robot to have high payload capacity and/or long reach. For collaborative robot designs to be successful, they must find ways of addressing these conflicting design requirements. One promising strategy for navigating this tradeoff is through the use of static balancing mechanisms to offset the robot's self weight, thus enabling the selection of lower-powered actuators. In this paper, we introduce a novel, 2 degree of freedom static balancing mechanism based on spring-loaded, wire-wrapped cams. We also present an optimization-based cam design method that guarantees the cams stay convex, ensures the springs stay below their extensions limits, and minimizes sensitivity to unmodeled deviations from the nominal spring constant. Additionally, we present a model of the effect of friction between the wire and the cam. Lastly, we show experimentally that the torque generated by the cam mechanism matches the torque predicted in our modeling approach. Our results also suggest that the effects of wire-cam friction are significant for non-circular cams.

Task and Configuration Space Compliance of Continuum Robots via Lie Group and Modal Shape Formulations

Continuum robots suffer large deflections due to internal and external forces. Accurate modeling of their passive compliance is necessary for accurate environmental interaction, especially in scenarios where direct force sensing is not practical. This paper focuses on deriving analytic formulations for the compliance of continuum robots that can be modeled as Kirchhoff rods. Compared to prior works, the approach presented herein is not subject to the constant-curvature assumptions to derive the configuration space compliance, and we do not rely on computationally-expensive finite difference approximations to obtain the task space compliance. Using modal approximations over curvature space and Lie group integration, we obtain closed-form expressions for the task and configuration space compliance matrices of continuum robots, thereby bridging the gap between constant-curvature analytic formulations of configuration space compliance and variable curvature task space compliance. We first present an analytic expression for the compliance of a single Kirchhoff rod. We then extend this formulation for computing both the task space and configuration space compliance of a tendon-actuated continuum robot. We then use our formulation to study the tradeoffs between computation cost and modeling accuracy as well as the loss in accuracy from neglecting the Jacobian derivative term in the compliance model. Finally, we experimentally validate the model on a tendon-actuated continuum segment, demonstrating the model's ability to predict passive deflections with error below 11.5\% percent of total arc length.

Design Considerations for 3RRR Parallel Robots with Lightweight, Approximate Static-Balancing

Balancing parallel robots throughout their workspace while avoiding the use of balancing masses and respecting design practicality constraints is difficult. Medical robots demand such compact and lightweight designs. This paper considers the difficult task of achieving optimal approximate balancing of a parallel robot throughout a desired task-based dexterous workspace using balancing springs only. While it is possible to achieve perfect balancing in a path, only approximate balancing may be achieved without the addition of balancing masses. Design considerations for optimal robot base placement and the effects of placement of torsional balancing springs are presented. Using a modal representation for the balancing torque requirements, we use recent results on the design of wire-wrapped cam mechanisms to achieve balancing throughout a task-based workspace. A simulation study shows that robot base placement can have a detrimental effect on the attainability of a practical design solution for static balancing. We also show that optimal balancing using torsional springs is best achieved when all springs are at the actuated joints and that the wire-wrapped cam design can significantly improve the performance of static balancing. The methodology presented in this paper provides practical design solutions that yield simple, lightweight and compact designs suitable for medical applications where such traits are paramount.

Model-Based Pose Estimation of Steerable Catheters under Bi-Plane Image Feedback

Small catheters undergo significant torsional deflections during endovascular interventions. A key challenge in enabling robot control of these catheters is the estimation of their bending planes. This paper considers approaches for estimating these bending planes based on bi-plane image feedback. The proposed approaches attempt to minimize error between either the direct (position-based) or instantaneous (velocity-based) kinematics with the reconstructed kinematics from bi-plane image feedback. A comparison between these methods is carried out on a setup using two cameras in lieu of a bi-plane fluoroscopy setup. The results show that the position-based approach is less susceptible to segmentation noise and works best when the segment is in a non-straight configuration. These results suggest that estimation of the bending planes can be accompanied with errors under 30 degrees. Considering that the torsional buildup of these catheters can be more than 180 degrees, we believe that this method can be used for catheter control with improved safety due to the reduction of this uncertainty.

Lie Group Formulation and Sensitivity Analysis for Shape Sensing of Variable Curvature Continuum Robots with General String Encoder Routing

This paper considers a combination of actuation tendons and measurement strings to achieve accurate shape sensing and direct kinematics of continuum robots. Assuming general string routing, a methodical Lie group formulation for the shape sensing of these robots is presented. The shape kinematics is expressed using arc-length-dependent curvature distributions parameterized by modal functions, and the Magnus expansion for Lie group integration is used to express the shape as a product of exponentials. The tendon and string length kinematic constraints are solved for the modal coefficients and the configuration space and body Jacobian are derived. The noise amplification index for the shape reconstruction problem is defined and used for optimizing the string/tendon routing paths, and a planar simulation study shows the minimal number of strings/tendons needed for accurate shape reconstruction. A torsionally stiff continuum segment is used for experimental evaluation, demonstrating mean (maximal) end-effector absolute position error of less than 2% (5%) of total length. Finally, a simulation study of a torsionally compliant segment demonstrates the approach for general deflections and string routings. We believe that the methods of this paper can benefit the design process, sensing and control of continuum and soft robots.

Torque-Limited Manipulation Planning through Contact by Interleaving Graph Search and Trajectory Optimization

Robots often have to perform manipulation tasks in close proximity to people. As such, it is desirable to use a robot arm that has limited joint torques so as to not injure the nearby person. Unfortunately, these limited torques then limit the payload capability of the arm. By using contact with the environment, robots can expand their reachable workspace that, otherwise, would be inaccessible due to exceeding actuator torque limits. We adapt our recently developed INSAT algorithm \cite{insat} to tackle the problem of torque-limited whole arm manipulation planning through contact. INSAT requires no prior over contact mode sequence and no initial template or seed for trajectory optimization. INSAT achieves this by interleaving graph search to explore the manipulator joint configuration space with incremental trajectory optimizations seeded by neighborhood solutions to find a dynamically feasible trajectory through contact. We demonstrate our results on a variety of manipulators and scenarios in simulation. We also experimentally show our planner exploiting robot-environment contact for the pick and place of a payload using a Kinova Gen3 robot. In comparison to the same trajectory running in free space, we experimentally show that the utilization of bracing contacts reduces the overall torque required to execute the trajectory.