ISIR
Abstract:Accurate estimation of interaction forces is crucial for achieving fine, dexterous control in robotic systems. Although tactile sensor arrays offer rich sensing capabilities, their effective use has been limited by challenges such as calibration complexities, nonlinearities, and deformation. In this paper, we tackle these issues by presenting a novel method for obtaining 3D force estimation using tactile sensor arrays. Unlike existing approaches that focus on specific or decoupled force components, our method estimates full 3D interaction forces across an array of distributed sensors, providing comprehensive real-time feedback. Through systematic data collection and model training, our approach overcomes the limitations of prior methods, achieving accurate and reliable tactile-based force estimation. Besides, we integrate this estimation in a real-time control loop, enabling implicit, stable force regulation that is critical for precise robotic manipulation. Experimental validation on the Allegro robot hand with uSkin sensors demonstrates the effectiveness of our approach in real-time control, and its ability to enhance the robot's adaptability and dexterity.
Abstract:Recent advances in large language models (LLMs) have led to significant progress in robotics, enabling embodied agents to better understand and execute open-ended tasks. However, existing approaches using LLMs face limitations in grounding their outputs within the physical environment and aligning with the capabilities of the robot. This challenge becomes even more pronounced with smaller language models, which are more computationally efficient but less robust in task planning and execution. In this paper, we present a novel modular architecture designed to enhance the robustness of LLM-driven robotics by addressing these grounding and alignment issues. We formalize the task planning problem within a goal-conditioned POMDP framework, identify key failure modes in LLM-driven planning, and propose targeted design principles to mitigate these issues. Our architecture introduces an ``expected outcomes'' module to prevent mischaracterization of subgoals and a feedback mechanism to enable real-time error recovery. Experimental results, both in simulation and on physical robots, demonstrate that our approach significantly improves task success rates for pick-and-place and manipulation tasks compared to both larger LLMs and standard baselines. Through hardware experiments, we also demonstrate how our architecture can be run efficiently and locally. This work highlights the potential of smaller, locally-executable LLMs in robotics and provides a scalable, efficient solution for robust task execution.
Abstract:The human hand is an immensely sophisticated tool adept at manipulating and grasping objects of unknown characteristics. Its capability lies in perceiving interaction dynamics through touch and adjusting contact force direction and magnitude to ensure successful manipulation. Despite advancements in control algorithms, sensing technologies, compliance integration, and ongoing research, precise finger force control for dexterous manipulation using tactile sensing remains relatively unexplored.In this work, we explore the challenges related to individual finger contact force control and propose a method for directing such forces perceived through tactile sensing. The proposed method is evaluated using an Allegro hand with Xela tactile sensors. Results are presented and discussed, alongside consideration for potential future improvements.
Abstract:Dexterous in-hand manipulation is a unique and valuable human skill requiring sophisticated sensorimotor interaction with the environment while respecting stability constraints. Satisfying these constraints with generated motions is essential for a robotic platform to achieve reliable in-hand manipulation skills. Explicitly modelling these constraints can be challenging, but they can be implicitly modelled and learned through experience or human demonstrations. We propose a learning and control approach based on dictionaries of motion primitives generated from human demonstrations. To achieve this, we defined an optimization process that combines motion primitives to generate robot fingertip trajectories for moving an object from an initial to a desired final pose. Based on our experiments, our approach allows a robotic hand to handle objects like humans, adhering to stability constraints without requiring explicit formalization. In other words, the proposed motion primitive dictionaries learn and implicitly embed the constraints crucial to the in-hand manipulation task.
Abstract:Recent advances in AI have led to significant results in robotic learning, including natural language-conditioned planning and efficient optimization of controllers using generative models. However, the interaction data remains the bottleneck for generalization. Getting data for grasping is a critical challenge, as this skill is required to complete many manipulation tasks. Quality-Diversity (QD) algorithms optimize a set of solutions to get diverse, high-performing solutions to a given problem. This paper investigates how QD can be combined with priors to speed up the generation of diverse grasps poses in simulation compared to standard 6-DoF grasp sampling schemes. Experiments conducted on 4 grippers with 2-to-5 fingers on standard objects show that QD outperforms commonly used methods by a large margin. Further experiments show that QD optimization automatically finds some efficient priors that are usually hard coded. The deployment of generated grasps on a 2-finger gripper and an Allegro hand shows that the diversity produced maintains sim-to-real transferability. We believe these results to be a significant step toward the generation of large datasets that can lead to robust and generalizing robotic grasping policies.
Abstract:Robotic capacities in object manipulation are incomparable to those of humans. Besides years of learning, humans rely heavily on the richness of information from physical interaction with the environment. In particular, tactile sensing is crucial in providing such rich feedback. Despite its potential contributions to robotic manipulation, tactile sensing is less exploited; mainly due to the complexity of the time series provided by tactile sensors. In this work, we propose a method for assessing grasp stability using tactile sensing. More specifically, we propose a methodology to extract task-relevant features and design efficient classifiers to detect object slippage with respect to individual fingertips. We compare two classification models: support vector machine and logistic regression. We use highly sensitive Uskin tactile sensors mounted on an Allegro hand to test and validate our method. Our results demonstrate that the proposed method is effective in slippage detection in an online fashion.