Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
@xiaoen0
xiaoen0 committed Dec 9, 2024
1 parent 0066b67 commit 8f802cc
Showing 1 changed file with 20 additions and 14 deletions.
34 changes: 20 additions & 14 deletions index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@

<head>
<meta charset="utf-8">
<meta name="description" content="Linking Touch, Language, and Vision: Learning Tactile Representation through Curriculum Guidance and Semantic Alignment">
<meta name="description" content="Touch100k: A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation">
<meta name="keywords" content="Touch-Centric Multimodal">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>TLV-Link</title>
<title>Touch100k</title>

<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/bulma.min.css">
Expand Down Expand Up @@ -103,8 +103,8 @@
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title"><img id="logo" width="6%" src="images/logo.jpg" style="vertical-align: bottom; margin-right: 10px;">Linking Touch, Language, and Vision:</h1>
<h1 class="title is-2 publication-title">Learning Tactile Representation through Curriculum Guidance and Semantic Alignment</h1>
<h1 class="title is-1 publication-title"><img id="logo" width="6%" src="images/logo.jpg" style="vertical-align: bottom; margin-right: 10px;">Touch100k:</h1>
<h1 class="title is-2 publication-title">A Large-Scale Touch-Language-Vision Dataset for Touch-Centric Multimodal Representation</h1>
<div class="is-size-5 publication-authors">
<br>
<span class="author-block">
Expand Down Expand Up @@ -154,26 +154,24 @@ <h1 class="title is-2 publication-title">Learning Tactile Representation through

<div class="column has-text-centered">
<div class="publication-links">

<span class="link-block">
<a href="https://github.com/cocacola-lab/TLV-Link" target="_blank"
<a href="https://drive.google.com/drive/folders/1QOvbkIZtpJpz4Ry_Zg3ouXX-zLJNqu9m?usp=sharing" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
<i class="fas fa-database"></i>
</span>
<span>Code</span>
<span>Dataset</span>
</a>
</span>
<span class="link-block">
<a href="https://drive.google.com/drive/folders/1QOvbkIZtpJpz4Ry_Zg3ouXX-zLJNqu9m?usp=sharing" target="_blank"
<a href="https://github.com/cocacola-lab/TLV-Link" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-database"></i>
<i class="fab fa-github"></i>
</span>
<span>Dataset</span>
<span>Code</span>
</a>
</span>

</div>
</div>
</div>
Expand All @@ -189,8 +187,16 @@ <h1 class="title is-2 publication-title">Learning Tactile Representation through
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots. Current research on tactile representation mainly focuses on the utilization of visual and tactile modalities, overlooking the language domain, even though language offers rich semantics and has been extensively explored. We thus propose TLV-Link, a pre-training method that links touch, language, and vision to learn a tactile representation for Gelsight sensor by capturing the relationship among these modalities. Specifically, we designate the vision encoder as the teacher model and the touch encoder as the student, guiding tactile representation through the visual modality using an improved curriculum learning approach. Simultaneously, we freeze the text encoder and employ contrastive learning to achieve semantic alignment between the tactile and language modalities. While TLV-Link extends our tactile representation learning beyond the alignment of visual and tactile modalities, we also need a high-quality dataset with aligned data pairs accompanied by language descriptions. We thus propose Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions
in multiple granularities (i.e., <em>sentence-level</em> natural expressions with rich semantics, including contextual and dynamic relationships, and <em>phrase-level</em> descriptions capturing the key features of tactile sensations). Moreover, we evaluate our representation's performance across two task categories (namely, material property identification and robot grasping prediction), focusing on tactile representation and zero-shot touch understanding. The experimental evaluation showcases the effectiveness of our representation. By enabling TLV-Link to achieve substantial improvements and establish a new state-of-the-art in touch-centric multimodal representation learning, Touch100k demonstrates its value as a valuable resource for research.
Touch holds a pivotal position in enhancing the perceptual and interactive capabilities of both humans and robots.
Despite its significance, current tactile research mainly focuses on visual and tactile modalities, overlooking the language domain.
Inspired by this insight, we construct Touch100k, a paired touch-language-vision dataset at the scale of 100k, featuring tactile sensation descriptions
in multiple granularities (i.e., <em>sentence-level</em> natural expressions with rich semantics, including contextual and dynamic relationships,
and <em>phrase-level</em> descriptions capturing the key features of tactile sensations). Based on the dataset, we propose a pre-training method,
Touch-Language-Vision Representation Learning through Curriculum Linking (TLV-Link, for short), inspired by the concept of curriculum learning. TLV-Link aims
to learn a tactile representation for the GelSight sensor and capture the relationship between tactile, language, and visual modalities.
We evaluate our model's performance across two task categories (namely, material property identification and robot grasping prediction),
focusing on tactile representation and zero-shot touch understanding. The experimental results demonstrate that TLV-Link achieves significant advancements,
establishing a new state-of-the-art performance in touch-centric multimodal representation learning. Additionally, the results validate the efficacy of the constructed Touch100k dataset.
</p>
</div>
</div>
Expand Down

0 comments on commit 8f802cc

Please sign in to comment.