Back to News & Insights
Frontier Observations

Haptics' "foundation model moment": from fragmented sensors to a shared latent space

Haptics' "foundation model moment": from fragmented sensors to a shared latent space

2026-09-08
Chief News Officer, TOBI Technology
Frontier Observations
Camera calibration and geometric quality inspection laboratory scenario

Introduction

The reason machine vision could enter the era of foundation models is that the output of cameras worldwide is the same thing — a two-dimensional RGB array. Tactile sensing has long lacked this premise: vision-based tactile sensors (GelSight-type) observe elastomer deformation, magnetic tactile sensors read magnetic field changes, resistive arrays measure pressure distributions, and taxel arrays output temporal vectors — the gap between them is closer to "camera vs. microphone vs. pressure plate" than to "iPhone camera vs. Samsung camera." For this reason, tactile policies must be retrained whenever the sensor changes, and tactile data cannot be aggregated across laboratories. In late June 2026, two back-to-back papers broke this deadlock with nearly the same approach: equipping each sensor type with a lightweight encoder and projecting all modalities into the same shared latent space. Around the same time, the first general-purpose tactile foundation policy and a million-hour-scale tactile dataset appeared in succession — tactile sensing has finally arrived at its own foundation model moment.

Technical Highlights

HTT: A unified backbone for heterogeneous tactile sensing. Heterogeneous Tactile Transformer (arXiv 2606.29948, 2026-06-29, National University of Singapore and Carnegie Mellon University, among others) constructed the Heterogeneous Paired Tactile (HPT) dataset, containing 1.6 million synchronized paired tactile frames, covering four commercial sensors: GelSight Mini and 9DTact (optical), Xela and TAC-02 (array-based). "Paired" means two sensors mounted on the same gripper simultaneously observing the same contact event — this is the key supervision signal required for cross-modal alignment. Architecturally, each sensor type is equipped with a dedicated encoder (a lightweight ViT for optical, a CNN for array-based), followed by a shared Transformer backbone; the training objective is per-modality masked reconstruction (preserving sensor details) + cross-modal alignment (pulling together embeddings of different sensors for the same event). Results: overall object classification accuracy 66.20%, 3D force estimation MAE 0.636, slip detection macro-F1 56.35; on a real robot, toy screw insertion success rate 95%, tofu grasping 55%, significantly outperforming baselines using only joint angles and force/torque.

TactX: zero-shot transfer across physical principles. TactX (arXiv 2606.31236, 2026-06-30, UC San Diego, Seoul National University, Amazon, et al.) tackles a harder version: unifying three sensors with completely different physical principles — Daimon (vision-based tactile), eFlesh (magnetic), and FlexiTac (resistive). Each sensor has its own encoder and decoder, but all are projected into the same 16-dimensional shared latent space; a triple objective is trained jointly: contrastive alignment (pulling paired contact events closer), self-reconstruction (latent variables can reconstruct their own signal), and cross-reconstruction (latent variables from sensor A can be decoded by sensor B's decoder). The third is the key — it forces the latent variables to retain "contact semantics" rather than "sensor identity".

FTP-1: a general-purpose tactile foundation policy. Proposed by Sharpa in collaboration with Tsinghua University, UC Berkeley, Shanghai Jiao Tong University, ETH Zurich, and others, and pretrained on approximately 3,000 hours of data drawn from 26 data sources and 21 types of tactile sensors (7 image-based, 5 array-based, and 9 state-based). The core is the Morphology-aware Tactile Token Space (MTTS), which maps arbitrary tactile input to semantically consistent tokens across 24 functional hand regions; it is paired with a dedicated tactile Transformer expert module to avoid interfering with existing vision-language knowledge. The result is transferability across embodied platforms, with a success-rate improvement of more than 31 percentage points on sensors unseen during pretraining.

The cost barrier has been broken. Tactile sensor prices have fallen by two orders of magnitude over the past decade: BioTac-class sensors at $5,000–$10,000 → GelSight Mini at approximately $499 and DIGIT, developed in collaboration with Meta AI, at approximately $350 → the open-source GelSlim, self-assembled for approximately $122. This price decline has directly unlocked the scale of data collection.

Data-side synchronization is scaling up. In April 2026, Daimon Robotics (DAIMON) jointly released the Daimon-Infinity omni-modal dataset with dozens of institutions worldwide. The dataset reaches a scale of one million hours, covering more than 80 real-world scenarios and over 2,000 human skills, of which 10,000 hours are open-sourced. Its vision-tactile approach converts tactile information into image format, thereby naturally integrating with existing vision AI frameworks, and on this basis proposes extending VLA into VTLA (Vision-Tactile-Language-Action).

Key Breakthroughs

  • •Cross-sensor zero-shot transfer has been empirically validated: HTT and TactX, two independent approaches, simultaneously validate that tactile data can be aggregated across hardware in the same way as vision data — this is the data prerequisite for tactile scaling.
  • •Tactile sensing now has, for the first time, a "foundation model + large-scale dataset" combination: FTP-1 provides a reusable policy, while Daimon-Infinity provides the corpus; the two are complementary.
  • •Engineering methods mature in step: FARM (ICRA 2026) uses a modified UMI handheld gripper integrated with a GelSight Mini to collect human demonstrations; the policy simultaneously predicts pose, gripper width, and gripping force, outperforming baselines across three task categories—high-force, low-force, and dynamic-force—Force is explicitly incorporated into the action space rather than being a byproduct of grasping commands.
  • •Tactile data is beginning to be demonstrated as directly improving production-line metrics: Public case studies show that introducing reinforcement learning in industrial machine tool loading/unloading tasks increased throughput by 42%, raised handover task success rate from 80% to 98%, and improved dual-arm bin transport success rate from 78% to 99%. Residual failures mainly stem from contact details—tactile sensing is the key to filling this gap.

Impact on the Industry

First,Tactile sensing goes from "optional accessory" to "data asset". When tactile input can be processed in the same latent space as vision (VTLA), the value of multimodal data acquisition devices is elevated from "one additional sensor" to "one additional data modality as an asset." Device manufacturers should incorporate tactile channels into standardized designs and record timestamps and calibration parameters in sync.

Second,A unified cross-sensor latent space lowers the barrier to data aggregation, meaning that parties with multi-model sensor collection capability have higher data reusability and bargaining power. It is recommended to explicitly annotate sensor model, sampling rate, and pairing relationships in the data card to facilitate future integration into shared latent spaces.

Third,Force/tactile annotation will become a new quality dimension. Industry datasets have traditionally been annotated primarily for vision and trajectories; the coverage of tactile, force/torque, and failure labels is becoming a differentiating metric (RoboMIND 2.0 already includes tactile and failure demonstration labels). For data collectors, this is both a technical barrier and a source of premium pricing.

Image Suggestions

Figure 1 (placement: after the introduction)

  • •Image description: A side-by-side physical illustration and output format comparison of four/three types of tactile sensors: vision-based tactile (elastomer deformation images), magnetic (magnetic field vector changes), resistive (pressure heatmaps), and taxel arrays (time-series curves). Below each, small text notes "Different measured physical quantities → cannot be directly aggregated," and a red X on the right indicates "Traditional approach: one-to-one pairing and retraining."
  • •Figure caption: Figure 1 | Heterogeneity of tactile sensors: differing measurement principles prevent direct data aggregation (GelSight / Xela / 9DTact / TAC-02 / Daimon / eFlesh / FlexiTac, etc.).

Figure 2 (placement: after the TactX paragraph in "Technical Highlights")

  • •Image description: Shared latent space architecture diagram. On the left, three sensor input blocks of different shapes (image, magnetic field, resistance) each pass through dedicated encoders (small colored squares) and are projected into the same 16-dimensional / unified latent space sphere in the center; from the sphere, three arrows point respectively to three training objectives — "contrastive alignment," "self-reconstruction," and "cross-reconstruction"; on the right is the output policy head.
  • •Figure caption: Figure 2 | Shared tactile latent space of HTT / TactX: dedicated encoders + unified backbone + triple alignment objective (architecture schematic; dimensions per the original paper).

Figure 3 (placement: after the last item in "Key Breakthroughs")

  • •Image description: Price-decline staircase chart (logarithmic scale): BioTac-class $5,000–$10,000 → GelSight Mini $499 / DIGIT ≈$350 → open-source GelSlim self-assembled for approximately $122. On the right, an arrow showing growth in "number of sensors × collection hours" illustrates how cost reduction unlocks scale.
  • •Figure caption: Figure 3 | Tactile sensor prices have fallen by two orders of magnitude over a decade, unlocking the preconditions for collection scale (prices are public quote ranges, varying by channel and order volume).

Source and Date

  • •HTT / Heterogeneous Paired Tactile (HPT) dataset: arXiv 2606.29948, 2026-06-29, NUS and CMU, etc.
  • •TactX: arXiv 2606.31236, 2026-06-30, UCSD, SNU, Amazon, et al.
  • •FTP-1: Sharpa, in collaboration with Tsinghua University, UC Berkeley, Shanghai Jiao Tong University, and ETH Zurich (2026, publicly released).
  • •Daimon-Infinity: Daimon Robotics with multiple institutions, released 2026-04, partially open-sourced (10,000 hours).
  • •FARM / Tactile-Conditioned Diffusion Policy: ICRA 2026, TU Darmstadt et al.
  • •Sensor price ranges: compiled from public reports (2026); industrial production-line reinforcement learning improvement data sourced from a public Humanoid report (2026).

Annotation Notes: The metrics for HTT/TactX/FTP-1 are all self-reported results from the original papers; evaluation protocols and hardware configurations differ,Cross-work horizontal comparison should be done with caution; price ranges fluctuate by channel and order volume,Reference value only. The industrial lifting data is on the enterprise case basis, not a third-party audit result (To be verified).

Contact Information · WeChat QR Code · Requirements Form

Hunan TOBI Technology Co., Ltd. (TOBI) · Embodied AI Partnerships +86 137 5502 0164

WeChat QR Code

Fill out the requirements form