First-Person Multimodal Data & Simulation Platform: The Two Pillars of Data Supply

Introduction

The data supply for embodied AI is taking the form of two mutually supporting pillars: one is first-person (egocentric) multimodal data from the real world, which provides physical realism and scalability potential; the other is simulation platforms and evaluation benchmarks, which provide controllability, reproducibility, and the massive number of environments required for reinforcement learning. The shift in 2026 is that these two pillars are beginning to interlock: first-person data provides real scenes and task distributions for simulation, simulation amplifies first-person data into interactive environments, and evaluation benchmarks provide a common yardstick for both. For data providers, understanding this interlocking relationship matters more than simply competing on data volume.

Technical Highlights

First-person data: from video to computable structure. Early ego data (such as Ego4D and similar) consisted mainly of unlabeled long-duration video, with limited usability. The key advance in 2026 is converting ego video into structured training signals: EgoTSR (ICML 2026) builds a first-person dataset of approximately 46 million samples, uses a three-stage curriculum to train models to judge task progress, achieving 92.4% long-horizon logical reasoning and 88.2% fine-grained perception, and uses bidirectional frame pairs to eliminate "temporal order bias"; EgoScale (ICML 2026) pretrains on 21,000 hours of first-person human data, achieving dexterous manipulation with only 4 hours of teleoperation fine-tuning; VITRA (Microsoft Research Asia) automatically converts unstructured human video into VLA training data in the same format as robot data (3D hand reconstruction → atomic action segmentation → language instruction and trajectory generation).

Multimodal expansion: haptics added in first-person perspective. In April 2026, Daimon Robotics jointly released the Daimon-Infinity full-modal dataset with multiple institutions, with a scale on the order of a million hours, covering more than 80 real-world scenarios and over 2,000 human skills, of which 10,000 hours are open-sourced. Its vision-tactile approach converts tactile signals into image format, enabling integration with existing vision frameworks, and proposes extending VLA to VTLA (Vision-Tactile-Language-Action).

Simulation platforms: a five-player landscape. Mainstream platforms each have their positioning: NVIDIA Isaac (Isaac Sim is built on Omniverse, supporting synthetic data generation and software-in-the-loop testing; Isaac Gym focuses on GPU-accelerated physics simulation, capable of training thousands of instances simultaneously on a single GPU), MuJoCo (generalized-coordinate simulation, robust contact optimization, supports particles/cloth/ropes/soft bodies), PyBullet, SAPIEN, Genesis. Companion datasets include BEHAVIOR-1K (1000 everyday activities), CALVIN (long-horizon language-conditioned tasks), LIBERO (130 language-conditioned tasks), ManiSkill (GPU parallelization).

Evaluation benchmarks: from success rate to multiple dimensions. RoboDojo was initiated by the Multimedia Laboratory of the University of Hong Kong in collaboration with UC Berkeley, Tsinghua University, and nearly 20 other institutions. It comprises 42 simulation tasks + 18 real-robot tasks, covering five dimensions: generalization, memory, fine manipulation, long-horizon execution, and open semantic understanding. In the first batch of leaderboards released in July 2026, the top simulation models achieved an average success rate of 8.80%, while the best real-robot result was 12.8%; human experts achieved 76.03% and 100% respectively. Tencent, together with industry partners, established RxBrain-Bench for evaluating world cognition and goal-image generation. In addition, on the 3D geometry side there are classic benchmarks such as DTU/CO3Dv2, and on the spatial reasoning side there is SpinBench (ICLR 2026).

Real to sim to real (real2sim2real). NVIDIA proposes extracting interactive objects via a smartphone-capture plus 3D-scanning pipeline, then infinitely augmenting them in simulation using "digital cousins," forming a closed loop from real scenes to trainable environments.

Key Breakthroughs

• •Ego videos now have automatic structuring capability: 3D hand reconstruction + atomic action segmentation + language generation, enabling unlabeled human video to be batch-converted into training data.
• •"Task progress" becomes a learnable continuous signal: works such as EgoTSR enable models to answer "which step has been completed," directly addressing the weakness in long-horizon tasks.
• •Tactile Sensing Enters the First-Person Data Ecosystem, the VTLA concept incorporates touch and vision into a unified processing framework.
• •The boundary between simulation and reality has been bridged: real2sim2real allows real-world data collection to be directly transformed into interactive environments, amplifying the value of each individual data point.

Impact on the Industry

First,Automatic structuring capability increases the leverage on "input quality". 3D hand reconstruction and pose tracking depend on precise camera pose and timestamps; synchronization and calibration noise in the input directly determine the usable rate of pre-annotations — this metric is replacing "annotation throughput" as the core operational metric.

Second,The diversity of simulation platforms requires data to have well-structured metadata. If the same batch of real-world data is to be reproduced across multiple platforms such as Isaac, MuJoCo, and Genesis, calibration parameters, time reference, and object geometry must be provided. Datasets with complete metadata have significantly lower adaptation cost.

Third,The multi-dimensionalization of evaluation benchmarks has changed the value function of data: Generalization, memory, fine manipulation, long-horizon execution, and open semantics — the five dimensions have different data requirements; a dataset should annotate its own coverage by dimension rather than only reporting total volume.

Fourth,Compliance and traceability become procurement prerequisites: first-person capture inherently involves faces, license plates, and commercial spaces; de-identification and authorization chains must be delivered together with the data.

Image Suggestions

Figure 1 (placement: after the introduction)

• •Scene description: A diagram of the interlocking of the two pillars. Left pillar "First-person real data": capture device → Ego multimodal stream → automatic structuring; right pillar "Simulation and evaluation": simulation platform → task environment → evaluation benchmark. The two pillars intersect at two connection points: ① real data provides scenes and task distributions for simulation (real2sim); ② simulation amplifies data into interactive environments and feeds back to policies (sim2real). At the bottom is a shared "policy training" platform.
• •Figure caption: Figure 1 | The interlocking relationship between first-person data and simulation platforms: the real2sim2real closed loop.

Figure 2 (placement: after the Ego structuring section in "Technical Highlights")

• •Scene description: An automatic structuring pipeline for Ego video. Four steps in series: raw first-person video → 3D hand reconstruction (hand keypoints/mesh) → atomic action segmentation (segmenting continuous motion into atomic segments according to kinematic principles) → language instruction and trajectory generation → training samples consistent with robot data formats. Each step is accompanied by a small icon, steps are connected by arrows, and "depends on camera pose and timestamp accuracy" is noted below "3D hand reconstruction."
• •Figure caption: Figure 2 | Automated conversion pipeline from unstructured human video to VLA training samples (drawn based on public methods like VITRA).

Figure 3 (placement: after the third item in "Impact on the Industry")

• •Image description: a five-axis radar chart with axes for generalization, memory, fine manipulation, long-horizon execution, and open semantic understanding. Two polygons are overlaid: one for "current model average performance" (lower on all axes) and one for "human expert" (close to the outer ring on all axes). An annotation bubble is placed next to the "memory" axis, marking the score gap between the top and second-best entries on the leaderboard.
• •Figure caption: Figure 3 | The multi-dimensionalization of evaluation benchmarks: the gap between models and humans is concentrated in memory and long-horizon dimensions (illustrative, not to scale).

Source and Date

• •EgoTSR (ICML 2026): approximately 46 million samples, 92.4% on long-horizon logical reasoning, 88.2% on fine-grained perception (metrics self-reported by the original paper,evaluation protocol to be verified).
• •EgoScale (ICML 2026, NVIDIA); VITRA (public technical introduction from Microsoft Research Asia).
• •Daimon-Infinity (Daimon Robotics jointly with multiple institutions, 2026-04, partially open-sourced with 10,000 hours).
• •Simulation platforms and datasets: compiled from the Chinese Association for Artificial Intelligence "Embodied AI (2026 Edition)" white paper (Isaac / MuJoCo / PyBullet / SAPIEN / Genesis; BEHAVIOR-1K / CALVIN / LIBERO / ManiSkill).
• •RoboDojo leaderboard (first batch 2026-07 / updated 2026-08-25); RxBrain-Bench (Tencent with industry partners, 2026); SpinBench (ICLR 2026).
• •NVIDIA real2sim2real and digital cousins solution (public talk, 2026).

Annotation Notes: metrics such as those for EgoTSR are self-reported by the original papers and are not directly comparable across works; simulation platform capability comparisons follow the white paper's compilation; the radar chart isIllustrative representation, and does not represent the exact values of any specific leaderboard.