Memorized action chunking with Transformers: Imitation learning for vision-based tissue surface scanning

Optical sensing technologies are emerging technologies used in cancer surgeries to ensure the complete removal of cancerous tissue. While point-wise assessment has many potential applications, incorporating automated large area scanning would enable holistic tissue sampling. However, such scanning tasks are challenging due to their long-horizon dependency and the requirement for fine-grained motion. To address these issues, we introduce Memorized Action Chunking with Transformers (MACT), an intuitive yet efficient imitation learning method for tissue surface scanning tasks. It utilizes a sequence of past images as historical information to predict near-future action sequences. In addition, hybrid temporal-spatial positional embeddings were employed to facilitate learning. In various simulation settings, MACT demonstrated significant improvements in contour scanning and area scanning over the baseline model. In real-world testing, with only 50 demonstration trajectories, MACT surpassed the baseline model by achieving a 60-80% success rate on all scanning tasks. Our findings suggest that MACT is a promising model for adaptive scanning in surgical settings.

Model Based Position Control of Soft Hydraulic Actuators

In this article, we investigate the model based position control of soft hydraulic actuators arranged in an antagonistic pair. A dynamical model of the system is constructed by employing the port-Hamiltonian formulation. A control algorithm is designed with an energy shaping approach which accounts for the pressure dynamics of the fluid. A nonlinear observer is included to compensate the effect of unknown external forces. Simulations demonstrate the effectiveness of the proposed approach, and experiments achieve positioning accuracy of 0.043 mm with a standard deviation of 0.033 mm in the presence of constant external forces up to 1 N.

Soft Continuum Actuator Tip Position and Contact Force Prediction, Using Electrical Impedance Tomography and Recurrent Neural Networks

Enabling dexterous manipulation and safe human-robot interaction, soft robots are widely used in numerous surgical applications. One of the complications associated with using soft robots in surgical applications is reconstructing their shape and the external force exerted on them. Several sensor-based and model-based approaches have been proposed to address the issue. In this paper, a shape sensing technique based on Electrical Impedance Tomography (EIT) is proposed. The performance of this sensing technique in predicting the tip position and contact force of a soft bending actuator is highlighted by conducting a series of empirical tests. The predictions were performed based on a data-driven approach using a Long Short-Term Memory (LSTM) recurrent neural network. The tip position predictions indicate the importance of using EIT data along with pressure inputs. Changing the number of EIT channels, we evaluated the effect of the number of EIT inputs on the accuracy of the predictions. The least RMSE values for the tip position are 3.6 and 4.6 mm in Y and Z coordinates, respectively, which are 7.36% and 6.07% of the actuator's total range of motion. Contact force predictions were conducted in three different bending angles and by varying the number of EIT channels. The results of the predictions illustrated that increasing the number of channels contributes to higher accuracy of the force estimation. The mean errors of using 8 channels are 7.69%, 2.13%, and 2.96% of the total force range in three different bending angles.

Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning

In operating Rooms (ORs), activities are usually different from other typical working environments. In particular, surgeons are frequently exposed to multiple psycho-organizational constraints that may cause negative repercussions on their health and performance. This is commonly attributed to an increase in the associated Cognitive Workload (CWL) that results from dealing with unexpected and repetitive tasks, as well as large amounts of information and potentially risky cognitive overload. In this paper, a cascade of two machine learning approaches is suggested for the multimodal recognition of CWL in a number of four different surgical tasks. First, a model based on the concept of transfer learning is used to identify if a surgeon is experiencing any CWL. Secondly, a Convolutional Neural Network (CNN) uses this information to identify different types of CWL associated to each surgical task. The suggested multimodal approach consider adjacent signals from electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS) and pupil eye diameter. The concatenation of signals allows complex correlations in terms of time (temporal) and channel location (spatial). Data collection is performed by a Multi-sensing AI Environment for Surgical Task $\&$ Role Optimisation platform (MAESTRO) developed at HARMS Lab. To compare the performance of the proposed methodology, a number of state-of-art machine learning techniques have been implemented. The tests show that the proposed model has a precision of 93%.

Lumen Shape Reconstruction using a Soft Robotic Balloon Catheter and Electrical Impedance Tomography

Incorrectly sized balloon catheters can lead to increased post-surgical complications, yet even with preoperative imaging, correct selection remains a challenge. With limited feedback during surgery, it is difficult to verify correct deployment. We propose the use of integrated impedance measurements and Electrical Impedance Tomography (EIT) imaging to assess the deformation of the balloon and determine the size and shape of the surrounding lumen. Previous work using single impedance measurements, or pressure data and analytical models, whilst demonstrating high sizing accuracy, have assumed a circular cross section. Here we extend these methods by adding a multitude of electrodes to detect elliptical and occluded lumen and obtain EIT images to localise deformations. Using a 14 Fr (5.3 mm) catheter as an example, numerical simulations were performed to find the optimal electrode configuration of two rings of 8 electrodes spaced 10 mm apart. The simulations predicted that the maximum detectable aspect ratio decreased from 0.9 for a 14mm balloon to 0.5 at 30mm. The sizing and ellipticity detection results were verified experimentally. A prototype robotic balloon catheter was constructed to automatically inflate a compliant balloon while simultaneously recording EIT and pressure data. Data were collected in experiments replicating stenotic vessels with an elliptical and asymmetrical profile, and the widening of a lumen during angioplasty. After calibration, the system was able to correctly localise the occlusion and detect aspect ratios of 0.75. EIT images further localised the occlusion and visualised the dilation of the lumen during balloon inflation.

A cable-driven parallel manipulator with force sensing capabilities for high-accuracy tissue endomicroscopy

This paper introduces a new surgical end-effector probe, which allows to accurately apply a contact force on a tissue, while at the same time allowing for high resolution and highly repeatable probe movement. These are achieved by implementing a cable-driven parallel manipulator arrangement, which is deployed at the distal-end of a robotic instrument. The combination of the offered qualities can be advantageous in several ways, with possible applications including: large area endomicroscopy and multi-spectral imaging, micro-surgery, tissue palpation, safe energy-based and conventional tissue resection. To demonstrate the concept and its adaptability, the probe is integrated with a modified da Vinci robot instrument.