Linking Touch, Language, and Vision:
- Learning Tactile Representation through Curriculum Guidance and Semantic Alignment
+Touch100k:
+ A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation
@@ -189,8 +187,16 @@Learning Tactile Representation through
Abstract
- Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots. Current research on tactile representation mainly focuses on the utilization of visual and tactile modalities, overlooking the language domain, even though language offers rich semantics and has been extensively explored. We thus propose TLV-Link, a pre-training method that links touch, language, and vision to learn a tactile representation for Gelsight sensor by capturing the relationship among these modalities. Specifically, we designate the vision encoder as the teacher model and the touch encoder as the student, guiding tactile representation through the visual modality using an improved curriculum learning approach. Simultaneously, we freeze the text encoder and employ contrastive learning to achieve semantic alignment between the tactile and language modalities. While TLV-Link extends our tactile representation learning beyond the alignment of visual and tactile modalities, we also need a high-quality dataset with aligned data pairs accompanied by language descriptions. We thus propose Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions
- in multiple granularities (i.e., sentence-level natural expressions with rich semantics, including contextual and dynamic relationships, and phrase-level descriptions capturing the key features of tactile sensations). Moreover, we evaluate our representation's performance across two task categories (namely, material property identification and robot grasping prediction), focusing on tactile representation and zero-shot touch understanding. The experimental evaluation showcases the effectiveness of our representation. By enabling TLV-Link to achieve substantial improvements and establish a new state-of-the-art in touch-centric multimodal representation learning, Touch100k demonstrates its value as a valuable resource for research.
+ Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots.
+ Despite its significance, current tactile research mainly focuses on visual and tactile modalities, overlooking the language domain.
+ Inspired by this insight, we construct Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions
+ in multiple granularities (i.e., sentence-level natural expressions with rich semantics, including contextual and dynamic relationships,
+ and phrase-level descriptions capturing the key features of tactile sensations). Based on the dataset, we propose a pre-training method,
+ Touch-Language-Vision Representation Learning through Curriculum Linking (TLV-Link, for short), inspired by the concept of curriculum learning. TLV-Link aims
+ to learn a tactile representation for the GelSight sensor and capture the relationship between tactile, language, and visual modalities.
+ We evaluate our model's performance across two task categories (namely, material property identification and robot grasping prediction),
+ focusing on tactile representation and zero-shot touch understanding. The experimental results demonstrate that TLV-Link achieves significant advancements,
+ establishing a new state-of-the-art performance in touch-centric multimodal representation learning. Additionally, the results validate the efficacy of the constructed Touch100k dataset.
- Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots. Current research on tactile representation mainly focuses on the utilization of visual and tactile modalities, overlooking the language domain, even though language offers rich semantics and has been extensively explored. We thus propose TLV-Link, a pre-training method that links touch, language, and vision to learn a tactile representation for Gelsight sensor by capturing the relationship among these modalities. Specifically, we designate the vision encoder as the teacher model and the touch encoder as the student, guiding tactile representation through the visual modality using an improved curriculum learning approach. Simultaneously, we freeze the text encoder and employ contrastive learning to achieve semantic alignment between the tactile and language modalities. While TLV-Link extends our tactile representation learning beyond the alignment of visual and tactile modalities, we also need a high-quality dataset with aligned data pairs accompanied by language descriptions. We thus propose Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions - in multiple granularities (i.e., sentence-level natural expressions with rich semantics, including contextual and dynamic relationships, and phrase-level descriptions capturing the key features of tactile sensations). Moreover, we evaluate our representation's performance across two task categories (namely, material property identification and robot grasping prediction), focusing on tactile representation and zero-shot touch understanding. The experimental evaluation showcases the effectiveness of our representation. By enabling TLV-Link to achieve substantial improvements and establish a new state-of-the-art in touch-centric multimodal representation learning, Touch100k demonstrates its value as a valuable resource for research. + Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots. + Despite its significance, current tactile research mainly focuses on visual and tactile modalities, overlooking the language domain. + Inspired by this insight, we construct Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions + in multiple granularities (i.e., sentence-level natural expressions with rich semantics, including contextual and dynamic relationships, + and phrase-level descriptions capturing the key features of tactile sensations). Based on the dataset, we propose a pre-training method, + Touch-Language-Vision Representation Learning through Curriculum Linking (TLV-Link, for short), inspired by the concept of curriculum learning. TLV-Link aims + to learn a tactile representation for the GelSight sensor and capture the relationship between tactile, language, and visual modalities. + We evaluate our model's performance across two task categories (namely, material property identification and robot grasping prediction), + focusing on tactile representation and zero-shot touch understanding. The experimental results demonstrate that TLV-Link achieves significant advancements, + establishing a new state-of-the-art performance in touch-centric multimodal representation learning. Additionally, the results validate the efficacy of the constructed Touch100k dataset.