Back to News & Insights
Vision Algorithm Special Topic

Multi-view and tactile fusion: extending "seeing" to "feeling"

The capability boundaries of pure vision solutions in embodied manipulation are already fairly clear: they can reliably solve "where is the object, what is it, and roughly how to grasp it," but for contact-state judgments such as "is it grasped firmly, is the force appropriate, and is there slippage," vision can only infer indirectly and is often disrupted by occlusion. Tactile and force sensing provide what vision…

2026-09-08
TOBI Technology Vision Algorithm Team
Vision Algorithm Special Topic
Data annotation platform and quality control process

Abstract

The capability boundaries of vision-only approaches in embodied manipulation are now fairly clear: they can reliably solve "where is the object, what is it, and roughly how to grasp it," but for contact-state judgments such as "is it grasped firmly, is the force appropriate, is there slipping," vision can only infer indirectly and is often disrupted by occlusion. Tactile and force sensing provide direct contact information that vision cannot replace, while multi-view vision provides spatial completeness lost from a single viewpoint. The fusion of the two is not a simple stacking of modalities; it requires connecting "pre-contact position prediction" with "post-contact state feedback" into a closed loop under a unified time base and spatial coordinate system. Key advances from 2025 to 2026 include: an order-of-magnitude drop in tactile sensor costs (from the thousands-of-dollars range down to hundreds or even the hundred-dollar range), the emergence of shared tactile-visual representation learning (HTT, TactX), the opening of large-scale tactile datasets (FTP-1, Daimon-Infinity at the million-hour scale), and methods that incorporate force/tactile information directly into the action space (FARM, ICRA 2026). This article surveys the technical pathways for fusion, compares representative works, and offers implementation recommendations.

I. Problem Background: Three Blind Spots of Vision

Blind spot one: the contact area under occlusion is invisible. When a gripper grasps an object, the contact region is precisely occluded by the gripper and the object itself. Wrist-mounted cameras used for grasping often cannot clearly see contact details due to distance and viewing angle. This means that at the moment when feedback is most needed, visual information is at its scarcest.

Blind spot 2: Inability to directly observe mechanical quantities. Vision can estimate object deformation (such as the degree to which a soft object is compressed), but it cannot directly measure contact force, friction, or slippage. These quantities are precisely what determine the success or failure of a grasp—too loose and the object slips; too tight and the object is damaged.

Blind spot three: lack of priors on material and stiffness. The same grasping force is suitable for metal parts but would damage tofu. Material properties cannot be inferred from a single-frame image; they must be learned through interaction, and tactile sensing is a direct product of that interaction.

The three technical routes of tactile sensing each have their own positioning:Vision-based tactile sensing(such as GelSight-type sensors, which infer contact geometry and force through optical imaging of elastomer deformation) offers high spatial resolution and rich information;Capacitive/resistive arraysEasy to integrate, with strong durability;Magnetic / opticalIt has advantages in flexibility and cost. In recent years, costs have dropped significantly — from the tens of thousands to tens of thousands of dollars range down to the hundreds of dollars range, and DIY solutions can even reach the hundred-dollar scale, which directly changes the scale at which tactile data can be collected.

II. Core Technical Path

2.1 Representation Learning for Touch: A Latent Space Shared with Vision

Tactile data is high-dimensional (image-based vision tactile or array-based signals) and lacks annotations, so self-supervised representation learning has become the mainstream approach. The core question is:What should tactile representations be aligned with?

One approach is tactile autoencoding (reconstructing tactile signals), which has the advantage of not requiring paired data, but the drawback that the learned representations may not be task-relevant. A more promising approach isCross-modal alignment: aligning tactile signals with vision and language into a shared latent space. Representative works include HTT (arXiv 2606.29948, 2026-06-29, NUS + CMU, based on HPT with approximately 1.6 million paired frames across 4 sensor types; classification 66.20%, force MAE 0.636, slip F1 56.35; toy screw task 95%, tofu task 55%) and TactX (arXiv 2606.31236, 2026-06-30, UCSD + SNU + Amazon, 16-dimensional shared latent space, 3 training objectives).

The practical value of cross-modal alignment is that it allows language instructions to directly "retrieve" tactile patterns (for example, which tactile signal corresponds to "grip gently"), thereby making tactile information consumable by VLMs/VLAs.

2.2 Force/Tactile Input into the Action Space

The traditional approach feeds tactile sensing as an additional observation channel into the policy; an alternative idea isdefining force/tactile on the action space side—that is, the policy outputs not only position but also desired contact force, which is realized by the underlying controller. The representative work FARM (ICRA 2026, combining GelSight Mini + UMI) validated the feasibility of this path. Its advantages are: the policy's expression is more natural (human manipulation intent inherently includes force), and the response frequency of force control can be guaranteed by the underlying controller, independent of the visual inference frequency.

2.3 Multi-View Fusion: Geometric Complementarity and Occlusion Resolution

The value of multi-view setups (head-mounted binocular + wrist camera + third-person) lies in complementarity: head-mounted/first-person views align with human manipulation intuition and the field of view moves with attention; wrist cameras provide close-up views of the contact region; third-person views provide global spatial relationships and occlusion resolution. Fusion methods have evolved from early feature concatenation to transformer-based cross-view attention, and further to explicitly incorporating multi-view geometric constraints (epipolar geometry, multi-view consistency) into training.

An important engineering judgment is:the marginal returns of adding more viewpoints diminish rapidly.. The improvement from 1 channel to 2–3 channels is significant; beyond 6 channels, the gains mainly come from special occlusion conditions, while costs (bandwidth, synchronization, calibration complexity) rise linearly. Therefore, the number of viewpoints should be determined based on task occlusion characteristics, not simply "more is better."

2.4 Time Synchronization: A Prerequisite for Fusion, Not a Detail

All multimodal fusion rests on one prerequisite: the modalities share a unified time base. The sampling rates of cameras and tactile/force sensors differ enormously (vision at 30—60 Hz, tactile up to several hundred Hz, torque up to kHz). Without hardware-level timestamping and trigger alignment, the quality of fusion has a structural ceiling. This is a link where "getting it right is invisible, but getting it wrong loses everything."

III. Comparison of Representative Work

WorkModalityCore methodPublished metricsTime
HTTVision + Tactile (4 sensor types)HPT-based cross-modal alignment, approximately 1.6 million paired framesClassification 66.20%; force MAE 0.636; slip F1 56.35; toy screw 95%, tofu 55%2026-06-29
TactXVision + tactile16-dimensional shared latent space, 3 training objectivesThe paper reports gains in cross-modal retrieval and transfer2026-06-30
FTP-1Tactile + Force + VisionLarge-scale tactile pretraining, approximately 3,000 hours, 26 sources, 21 sensor types; MTTS 24 tokens+31 percentage points relative to baseline2025—2026
FARMGelSight Mini + UMIForce defined in action spaceThe paper reports improvements on manipulation tasksICRA 2026
Daimon-Infinity (Daimeng)Tactile-centricMillion-hour-scale data, 80+ scenarios, 2000+ skills, 10,000 hours releasedData Scale as the Primary Contribution2026-04
First-person works such as EgoScaleFirst-person visionLarge-scale first-person pretrainingLog-linear scaling lawICML 2026

Note: All metrics come from their respective papers or official releases,Tasks, protocols, and sensor configurations differ; absolute values are not directly comparable; some works are preprints, and conclusions are subject to the final versions.

IV. Challenges and Trends

Challenge one: sensor heterogeneity. Tactile sensors from different vendors and based on different principles produce outputs with completely different formats and physical meanings, making tactile data difficult to mix and train across sources the way images can. This is a structural disadvantage of the tactile field compared to vision.

Challenge Two: The Scale Gap in Tactile Data. Vision has internet-scale data; tactile does not. Million-hour-scale tactile data (such as Daimon-Infinity) already represents top-tier industry scale, but it remains orders of magnitude behind vision data.

Challenge three: degradation of tactile signals and calibration drift. Elastomer aging and temperature drift cause tactile sensor responses to change over time, requiring recalibration or domain adaptation. This is a substantive issue in long-term deployment.

Trend One: Declining Costs Drive a Leap in Data Scale. Sensor prices have fallen from the thousands-of-dollars range to hundreds or even hundreds-of-dollars range (DIY solutions are on the order of a hundred dollars), enabling large-scale, multi-source tactile data collection.

Trend 2: Tactile sensing shifts from an "optional modality" to an essential modality for precision manipulation. On tasks involving deformable objects, fragile items, and precision assembly, the success-rate ceiling of pure vision solutions has already become apparent.

Trend Three: Alignment Between Tactile and Language Becomes the Competitive Focus. Whoever can translate tactile signals into instructable, retrievable semantics controls the entry point for integrating tactile sensing into VLAs.

V. Implementation Recommendations

  1. 1.First determine whether the task truly requires tactile sensing. For pick-and-place of rigid objects, pure vision suffices; tactile sensing is only necessary for compliant, fragile, precision insertion, or force-graded tasks. Avoid increasing system complexity for the sake of being "technologically advanced."
  2. 2.Treat time synchronization as the top priority. Unify hardware timestamps and trigger signals, and record them in the data card. This is a prerequisite for all fusion work and cannot be remedied after the fact.
  3. 3.Consider long-term stability when selecting tactile sensors. Prioritize models with replaceable elastomers, well-defined calibration procedures, and temperature-drift compensation; include the "recalibration interval" as one of the selection criteria.
  4. 4.Design a unified intermediate representation for tactile data. Even when physical meanings differ across sensors, define a task-relevant intermediate layer (such as contact force direction, slip probability, contact area) so that data from different sources can be mixed.
  5. 5.Collect vision and tactile data simultaneously, rather than tactile alone. The value of cross-modal alignment methods depends on paired data; the reuse value of single-modality tactile data is significantly lower.
  6. 6.Incorporate force/tactile data into the annotation system. At minimum, annotate: contact time, contact type (point/line/surface), whether slipping occurred, and whether damage was caused. These fields have very low cost but extremely high downstream value.

VI. Actionable Implications for TOBI Technology

  • •Hardware Side: multi-camera + scalable tactile interfaces represent a reasonable form for next-generation embodied data acquisition. The key is not whether tactile sensing is standard in the current configuration, but whethera unified time base and trigger interface are reserved, so that when the customer later adds tactile sensors, the synchronization architecture does not need to be redesigned. Microsecond-level synchronization capability here expands from a "vision advantage" to a "multimodal advantage".
  • •Data Side: Very few public datasets simultaneously provide vision, pose, and tactile/force pairings (RoboMIND 2.0 is one of the few datasets that includes tactile data, force, and failure labels, with approximately 310,000 trajectories, 1,000+ hours, 739 tasks, and 6 embodiments).Multimodal paired data is a clearly scarce resource, which is of greater strategic value than simply increasing visual hours.
  • •Ecosystem Side: define and publish the temporal reference and pairing format for multimodal data. During the window where multimodal fusion has become a necessity but standards are still missing, participating in standard-setting is a low-cost, high-return ecosystem positioning move.

Source and Date

HTT (arXiv 2606.29948, 2026-06-29, NUS + CMU, HPT approximately 1.6 million paired frames, 4 sensor types; classification 66.20%, force MAE 0.636, slip F1 56.35; toy screw 95%, tofu 55%); TactX (arXiv 2606.31236, 2026-06-30, UCSD + Seoul National University + Amazon, 16-dimensional shared latent space, 3 objectives); FTP-1 (Sharpa + Tsinghua University + UC Berkeley + Shanghai Jiao Tong University + ETH, approximately 3,000 hours, 26 sources, 21 sensor types, MTTS 24 tokens, +31 percentage points relative to baseline); FARM (ICRA 2026, GelSight Mini + UMI, force defined in action space); Daimon-Infinity (2026-04, million-hour-scale tactile data, 80+ scenes, 2,000+ skills, 10,000 hours released); RoboMIND 2.0 (approximately 310,000 trajectories, 1,000+ hours, 739 tasks, 6 embodiments, including tactile data, force, and failure labels); EgoScale (ICML 2026, 21,000 hours of first-person pretraining + 50 hours of motion capture + 4 hours of teleoperation, <0.1% teleoperation ratio, 22 degrees of freedom, log-linear scaling); tactile sensor price ranges (thousands to tens of thousands of USD → hundreds of USD → DIY at approximately the hundred-USD level, compiled from public and market information, 2025–2026).

Annotation: The price range isCompiled from market information, because variations across models, batches, and regions are significant,Specific values to be verified; cross-work metricsNot directly comparable; judgments regarding "diminishing marginal returns from viewpoints" and "scarcity of multimodally paired data" belong toEngineering Experience InferenceandForward-Looking Assessment.