Touch Improves Learning Dexterous Grasping from Demonstration

Abstract

Robotic dexterous grasping with human-level performance remains an unsolved challenge. Imitation Learning (IL) offers a compelling paradigm, yet vision-only approaches are brittle under lighting changes and novel objects. Touch sensing addresses this gap by grounding manipulation in contact properties rather than visual appearance. We present a systematic study of touch sensing in dexterous, multi-fingered IL, evaluating three algorithms (ACT, Flow Matching, BESO) across transformer, Mamba, and xLSTM backbones, and three tactile representations (Vision-only, end-to-end CNN, pretrained Sparsh-X). Using VR teleoperation with a Digit 360-equipped robotic hand, we collect demonstrations and evaluate generalization to novel objects and robustness to lighting perturbations. We train 21 policy variants (7 algorithms-backbone combinations × 3 tactile representations) and evaluate each on the training and test objects, yielding 28 comparison pairs. In 25 of 28, tactile signals match or improve upon vision-only performance, with the largest gains on unseen objects and degraded lighting. Failure analysis reveals that touch primarily improves initial grasp alignment, reducing misplaced approaches. Notably, end-to-end CNN encoders degrade diffusion-based policies by corrupting visual subtasks, while pretrained encoders avoid this. These results provide practical guidance on learning algorithm, backbone, and tactile representation choices for imitation learning of dexterous grasping.

Overview Video

Method Overview

We present a systematic study evaluating multiple state-of-the-art imitation learning algorithms, backbone architectures, and tactile representations for dexterous grasping. Our approach uses VR teleoperation with a Digit 360-equipped multi-finger hand to collect demonstrations, then trains policies under identical conditions to isolate the contribution of tactile sensing from architectural choices. We evaluate three algorithms (ACT, Flow Matching, BESO) across three backbones (Transformer, Mamba, xLSTM) with three tactile representations (Vision-only, end-to-end CNN, pretrained Sparsh-X), yielding 21 policy variants tested across 28 comparison pairs.

System Overview: We study tactile sensing for dexterous grasping via systematic comparison of IL algorithms, backbones, and tactile representations. The VR teleoperation setup features the Digit 360-equipped multi-finger hand grasping an object, with a close-up of the tactile sensor reading. Across 28 evaluation configurations, touch matches or improves vision-only performance in 25 cases.

Hardware Platform and Data Collection: We develop a VR-based teleoperation system using the Meta Quest 3 to track human operator movements, collecting 180 demonstrations at 30 Hz with the xArm-7 robot arm mounting a Tilburg hand equipped with four Digit 360 tactile sensors. The dataset consists of two training objects (marked in purple) made of firm plastic, with ten grasps in nine different positions each. Object Set: We evaluate on 16 objects total — 2 training objects and 14 everyday household test objects made of different materials, testing generalization beyond the training distribution.

Teleoperation Setup: VR teleoperation system with Digit 360 tactile sensors for collecting dexterous grasping demonstrations.

Object Set: 16 objects used in the study, including 2 training objects and 14 test objects for evaluating generalization.

Results

We evaluate all 21 policy variants on both training and test objects, with each policy performing 144 grasps (16 objects × 9 positions). Our key finding: tactile signals match or improve upon vision-only performance in 25 out of 28 cases, with the benefit becoming more pronounced when generalizing to novel objects. The performance gap between training and test objects narrows when tactile signals are included, suggesting that contact-grounded observations provide appearance-invariant information that supports generalization beyond the demonstrated object set.

Grasping Performance: Performance comparison across different algorithms, backbones, and tactile representations on training objects.

Generalization to Novel Objects: Touch sensing provides the largest gains when generalizing to unseen objects.

Tactile Representation Comparison: While the performance is more similar on training objects with a 6% difference [≈83.14% Vision-only, ≈89.57% Tactile (CNN), ≈88.86% Tactile (Sparsh-X)], this gap grows to an 11% higher success rate when facing test objects [≈40.14% Vision-only, ≈50.14% Tactile (CNN), ≈51.14% Tactile (Sparsh-X)]. The CNN and Sparsh-X representations perform within 1% of each other on average, indicating that for this grasping task, both task-specific learned features and pretrained embeddings capture comparable contact information.

Algorithm and Backbone Analysis: We observe that xLSTM achieves a 7% and 10% higher success rate when compared to Mamba and Transformer using Flow Matching respectively, yet it performs 5% and 4% worse for the BESO algorithm. No single backbone emerges as universally advantageous across both learning algorithms, highlighting the importance of algorithm-backbone interaction.

Tactile Representation Comparison: Comparing vision-only, end-to-end CNN, and pretrained Sparsh-X tactile representations.

Backbone Comparison: Mean performance across training and test objects for Transformer, Mamba, and xLSTM backbones.

Environmental Robustness: We evaluate four lighting conditions: baseline, dimmed with purple side light, purple-only color perturbation, and fully darkened room. Average grasping performance decreases across all policies as lighting conditions worsen, confirming the expected sensitivity of vision-based control to appearance shifts. However, the Sparsh-X encoded tactile representation degrades considerably less than the other two, demonstrating that pretrained contact representations provide meaningful robustness when visual information becomes unreliable.

Failure Mode Analysis: The failure analysis reveals that the benefit of touch is not uniformly distributed across the grasping sequence. The largest improvement occurs at the initial approach stage: tactile variants reduce Misplaced grasp attempts from ≈44.3 to ≈23.3 (CNN) and ≈31.9 (Sparsh-X) compared to vision-only baselines. Once the robot establishes contact, differences between tactile and vision-only variants diminish to within one failure case across Failed lift and Dropping. This indicates that touch primarily provides the contact-geometry signal needed to approach the object with a viable grasp configuration, rather than aiding in-hand stabilization.

Robustness to Lighting Changes: Touch sensing significantly improves robustness under degraded lighting conditions, with pretrained Sparsh-X showing the most stable performance.

Failure Case Analysis: Mean accumulated failure occurrences across 1008 grasping attempts. Most failures occur due to hand misplacement, which is significantly reduced when using tactile signals (both CNN and Sparsh-X).

Acknowledgements

This work was partly supported by the project Genius-Robot (01IS24083) funded by the Federal Ministry of Research, Technology and Space (BMFTR), by BMFTR in the DAAD project 57616814 (SECAI, School of Embedded Composite AI), by BMFTR under the Robotics Institute Germany (RIG), by the German Research Foundation (DFG) under the Cluster of Excellence CARE: Climate-Neutral And Resource-Efficient Construction (EXC 3115, project number 533767731) and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) (448648559). The authors gratefully acknowledge the computing time provided on the high-performance computer HoreKa by the National High-Performance Computing Center at KIT and by Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for GCS Supercomputer JUPITER at Jülich Supercomputing Centre (JSC). We furthermore thank the ZIH at TU Dresden for providing computing resources. We acknowledge the use of Claude Opus and Qwen 3.7 to support code development. The outputs were used in Section V to generate supportive scripts. All outputs were validated by the authors of this paper.

Author Contributions (CRediT)

Conceptualization: NL, PV, KN, RC; Data curation: NL; Formal Analysis: NL, PV; Funding acquisition: RL, RC; Investigation: NL, PV; Methodology: NL, PV, JS; Project administration: RL, RC; Resources: RL, RC; Software: NL, PV, JS; Supervision: KN, RL, RC; Validation: NL, PV, JS; Visualization: NL; Writing – original draft: NL, PV, JS; Writing – review & editing: NL, PV, JS, RL, RC

Citation

@article{leukroth2026touch, title={Touch Improves Learning Dexterous Grasping from Demonstration}, author={Leukroth, Niklas and Vanjani, Pankhuri and Suliga, Jakub and Nakahara, Ken and Lioutikov, Rudolf and Calandra, Roberto}, journal={arXiv preprint}, year={2026} }