$\small \mathcal{D(R,O)}$ Grasp: A Unified Representation of Robot and Object Interaction for Cross-Embodiment Dexterous Grasping

Zhenyu Wei1,2*, Zhixuan Xu1*, Jingxiang Guo1, Yiwen Hou1, Chongkai Gao1, Zhehao Cai1, Jiayu Luo1,
Lin Shao1
1National University of Singapore, 2Shanghai Jiao Tong University
* denotes equal contribution
arXiv Paper Appendix Code Data

Abstract

Dexterous grasping is a fundamental yet challenging skill in robotic manipulation, requiring precise interaction between robotic hands and objects. In this paper, we present $\mathcal{D(R,O)}$ Grasp, a novel framework that models the interaction between the robotic hand in its grasping pose and the object, enabling broad generalization across various robot hands and object geometries. Our model takes the robot handā€™s description and object point cloud as inputs and efficiently predicts kinematically valid and stable grasps, demonstrating strong adapt ability to diverse robot embodiments and object geometries. Extensive experiments conducted in both simulated and real world environments validate the effectiveness of our approach, with significant improvements in success rate, grasp diversity, and inference speed across multiple robotic hands. Our method achieves an average success rate of 87.53% in simulation in less than one second, tested on three different dexterous robotic hands, and also performs successfully in real-world experiments using LeapHand. $\mathcal{D(R,O)}$ Grasp provides a robust solution for dexterous grasping in complex and varied environments.

Pipeline Overview

Overview of D(R,O) Grasp framework
Overview of $\mathcal{D(R,O)}$ Grasp: We first pretrain the robot encoder with the proposed configuration-invariant pretraining method. Then, we predict the $\mathcal{D(R,O)}$ representation between the robot and object point cloud. Finally, we extract joint values from the $\mathcal{D(R,O)}$ representation.

Simulation Grasps

Barrett

Allegro

ShadowHand

Grasp ID: 0

Partial Observation Grasps

Grasp

Observation

Real-world Demos

Apple Cover

Apple

Bag Cover

Bag

Brush Cover

Brush

Cookie Box Cover

Cookie Box

Cube Cover

Cube

Cup Cover

Cup

Dinosaur Cover

Dinosaur

Duck Cover

Duck

Tea Box Cover

Tea Box

Toilet Cleaner Cover

Toilet Cleaner

BibTeX

@article{wei2024dro,
    title={D(R, O) Grasp: A Unified Representation of Robot and Object Interaction for Cross-Embodiment Dexterous Grasping},
    author={Wei, Zhenyu and Xu, Zhixuan and Guo, Jingxiang and Hou, Yiwen and Gao, Chongkai and Cai, Zhehao and Luo, Jiayu and Shao, Lin},
    journal={arXiv preprint arXiv:2410.01702},
    year={2024}
}