Dual-Modality Haptic Feedback Improves Dexterous Task Execution with Virtual EMG-Controlled Gripper

Upper-extremity amputees who use myoelectric prostheses currently lack the haptic sensory information needed to perform dexterous activities of daily living. While considerable research has focused on restoring this haptic information, these approaches often rely on single-modality feedback schemes which are necessary but insufficient for the feedforward and feedback control strategies employed by the central nervous system. Multi-modality feedback approaches have been gaining attention in several application domains, however, the utility for myoelectric prosthesis use remains unclear. In this study, we investigated the utility of dual-modality haptic feedback in a virtual EMG-controlled grasp-and-hold task with a brittle object and variable load force. We recruited N=20 non-amputee participants to perform the task in four conditions: no feedback, vibration feedback of incipient slip, squeezing feedback of grip force, and dual (vibration + squeezing) feedback of incipient slip and grip force. Results suggest that receiving any feedback is better than receiving none, however, dual-modality feedback is far superior to either single-modality feedback approach in terms of preventing the object from breaking or dropping, even after it started slipping. Control with dual-modality feedback was also seen as more intuitive than with either of the single-modality feedback approaches.

Haptic Shared Control Improves Neural Efficiency During Myoelectric Prosthesis Use

Clinical myoelectric prostheses lack the sensory feedback and sufficient dexterity required to complete activities of daily living efficiently and accurately. Providing haptic feedback of relevant environmental cues to the user or imbuing the prosthesis with autonomous control authority have been separately shown to improve prosthesis utility. Few studies, however, have investigated the effect of combining these two approaches in a shared control paradigm, and none have evaluated such an approach from the perspective of neural efficiency (the relationship between task performance and mental effort measured directly from the brain). In this work, we analyzed the neural efficiency of 30 non-amputee participants in a grasp-and-lift task of a brittle object. Here, a myoelectric prosthesis featuring vibrotactile feedback of grip force and autonomous control of grasping was compared with a standard myoelectric prosthesis with and without vibrotactile feedback. As a measure of mental effort, we captured the prefrontal cortex activity changes using functional near infrared spectroscopy during the experiment. Results showed that only the haptic shared control system enabled users to achieve high neural efficiency, and that vibrotactile feedback was important for grasping with the appropriate grip force. These results indicate that the haptic shared control system synergistically combines the benefits of haptic feedback and autonomous controllers, and is well-poised to inform such hybrid advancements in myoelectric prosthesis technology.

The Utility of Synthetic Reflexes and Haptic Feedback for Upper-Limb Prostheses in a Dexterous Task Without Direct Vision

Individuals who use myoelectric upper-limb prostheses often rely heavily on vision to complete their daily activities. They thus struggle in situations where vision is overloaded, such as multitasking, or unavailable, such as poor lighting conditions. Non-amputees can easily accomplish such tasks due to tactile reflexes and haptic sensation guiding their upper-limb motor coordination. Based on these principles, we developed and tested two novel prosthesis systems that incorporate autonomous controllers and provide the user with touch-location feedback through either vibration or distributed pressure. These capabilities were made possible by installing a custom contact-location sensor on thefingers of a commercial prosthetic hand, along with a custom pressure sensor on the thumb. We compared the performance of the two systems against a standard myoelectric prosthesis and a myoelectric prosthesis with only autonomous controllers in a difficult reach-to-pick-and-place task conducted without direct vision. Results from 40 non-amputee participants in this between-subjects study indicated that vibrotactile feedback combined with synthetic reflexes proved significantly more advantageous than the standard prosthesis in several of the task milestones. In addition, vibrotactile feedback and synthetic reflexes improved grasp placement compared to only synthetic reflexes or pressure feedback combined with synthetic reflexes. These results indicate that both autonomous controllers and haptic feedback facilitate success in dexterous tasks without vision, and that the type of haptic display matters.

A novel teleoperator testbed to understand the effects of master-slave dynamics on embodiment and kinesthetic perception *

With the rising popularity of telerobotic systems, the focus on transparency with regards to haptic perception is also increasing. Transparency, however, represents a theoretical ideal as most bilateral force-reflecting telerobots introduce dynamics (stiffness and damping) between the operator and the environment. To achieve true dexterity, it will be essential to understand how humans embody the dynamics of these telerobots and thereby distinguish them from the environment they are exploring. In this short manuscript, we introduce a novel single degree-of-freedom testbed designed to perform psychophysical and task performance assessments of kinesthetic perception during telerobotic exploration. The system is capable of being configured as a rigid mechanical teleoperator, a dynamic mechanical teleoperator, and an electromechanicaal teleoperator. We performed prefatory system identification and found that the system is capable of simulating telerobotic exploration necessary to understand the impact of master-slave dynamics on kinesthetic perception.

Preliminary investigation into how limb choice affects kinesthetic perception

We have a limited understanding of how we integrate haptic information in real-time from our upper limbs to perform complex bimanual tasks, an ability that humans routinely employ to perform tasks of varying levels of difficulty. In order to understand how information from both limbs is used to create a unified percept, it is important to study both the limbs separately first. Prevalent theories highlighting the role of central nervous system (CNS) in accounting for internal body dynamics seem to suggest that both upper limbs should be equally sensitive to external stimuli. However, there is empirical proof demonstrating a perceptual difference in our upper limbs for tasks like shape discrimination, prompting the need to study effects of limb choice on kinesthetic perception. In this manuscript, we start evaluating Just Noticeable Difference (JND) for stiffness for both forearms separately. Early results validate the need for a more thorough investigation of limb choice on kinesthetic perception.

Investigating the effects of exploration dynamics on stiffness perception

The utility of Human-in-the-loop telerobotic systems (HiLTS) is driven in part by the quality of feedback it can provide to the operator. While the dynamic interaction between robot and environment can often be sensed or modeled, the dynamic coupling of the human-robot interface is often overlooked. Enabling dexterous manipulation through HiLTS however, will require careful consideration of human haptic perception as it relates to the human's changing limb impedance at the human-robot interface. In this manuscript, we present results from a stiffness perception task run on a simple 1-DoF rotational kinesthetic device at three different angular velocities, based on participant's natural exploration strategy. We evaluated performance effects of exploration velocity as a proxy measurement for limb impedance and the results indicate the need to further investigate how the human body incorporates its knowledge of the body dynamics in kinesthetic perception under active exploration.

Towards an understanding of how humans perceive stiffness during bimanual exploration

In this paper, an experimental testbed and associated psychophysical paradigm are presented for understanding how people discriminate torsional stiffness using wrist rotation about their forearm. Featured in the testbed are two 1-DoF rotary kinesthetic haptic devices. An adaptive staircase was used to evaluate JNDs for a stiffness discrimination task where participants explored virtual torsion springs by rotating their forearms. The JNDs were evaluated across seven different conditions, under four different exploration modes: bimanual, unimanual, bimanual feedback for unimanual displacement, and unimanual feedback for bimanual displacement. The discrimination results will inform future investigation into understanding how stiffness percepts vary.

Towards an Understanding of the Role Operator Limb Dynamics Plays in Haptic Perception of Stiffness

Creating haptic interfaces capable of rendering the rich sensation needed for dexterous manipulation is crucial for the advancement of human-in-the-loop telerobotic systems (HiLTS). One limiting factor has been the absence of detailed knowledge of the effect of operator limb dynamics and haptic exploration dynamics on haptic perception. We propose to begin investigations of these effects with single-joint haptic exploration and feedback of physical and virtual environments. Here, we present our experimental apparatus, a 1-DoF rotational kinesthetic haptic device and electromyography (EMG) system, along with preliminary findings from our efforts to investigate the change in stiffness discrimination thresholds for differing exploration velocities. Result trends indicate a possible relationship between exploration velocity and discrimination thresholds, as well as a complex interaction between muscle activation, exploration velocity, and haptic feedback.

A novel testbed for investigating the impact of teleoperator dynamics on perceived environment dynamics

Human-in-the-loop telerobotic systems (HiLTS) are robotic tools designed to extend and in some circumstances improve the dexterous capabilities of the human operator in virtual and remote environments. Dexterous manipulation, however, depends on how well the telerobot is incorporated into the operator's sensorimotor control scheme. Empirical evidence suggests that haptic feedback can lead to improved dexterity. Unfortunately, haptic feedback can also introduce dynamics between the leader and follower of the telerobot that affect both stability and device performance. While concerted research effort has focused on masking these device dynamics or bypassing them altogether, it is not well understood how human operators incorporate these dynamics into their control scheme. We believe that to advance dexterous telerobotic manipulation, it is crucial to understand the process by which humans operators incorporate teleoperator dynamics and distinguish them from the dynamics of the environment. Key to this knowledge is an understanding of how advanced telerobotic architectures compare to the gold standard, the rigid mechanical teleoperators first introduced in the 1950's. In this manuscript, we present a teleoperator testbed that has reconfigurable transmissions between the leader and follower to change its dynamic behavior. The intent of this testbed is to investigate the effect of the teleoperator's dynamics on perception of and task performance in the remote/virtual environment. We describe the hardware and software components of the testbed and then demonstrate how the different teleoperator transmissions can lead to differences, sometimes significant, in the dynamics that would be felt by the operator when exploring the same environment.

Sensorimotor-inspired Tactile Feedback and Control Improve Consistency of Prosthesis Manipulation in the Absence of Direct Vision

The lack of haptically aware upper-limb prostheses forces amputees to rely largely on visual cues to complete activities of daily living. In contrast, able-bodied individuals inherently rely on conscious haptic perception and automatic tactile reflexes to govern volitional actions in situations that do not allow for constant visual attention. We therefore propose a myoelectric prosthesis system that reflects these concepts to aid manipulation performance without direct vision. To implement this design, we built two fabric-based tactile sensors that measure contact location along the palmar and dorsal sides of the prosthetic fingers and grasp pressure at the tip of the prosthetic thumb. Inspired by the natural sensorimotor system, we use the measurements from these sensors to provide vibrotactile feedback of contact location and implement a tactile grasp controller that uses automatic reflexes to prevent over-grasping and object slip. We compare this system to a standard myoelectric prosthesis in a challenging reach-to-pick-and-place task conducted without direct vision; 17 able-bodied adults took part in this single-session between-subjects study. Participants in the tactile group achieved more consistent high performance compared to participants in the standard group. These results indicate that the addition of contact-location feedback and reflex control increases the consistency with which objects can be grasped and moved without direct vision in upper-limb prosthetics.