场景与佩戴者问答Scene & wearer QA
区分谁在操作、看到了什么、哪些问题无法回答。Identify who acts, what is visible and which questions cannot be answered.
问答 + 证据时间段QA + evidence spans
评估这个任务 →Scope this task →EGOCENTRIC DATA · VLM
让第一人称采集，
进入你的 VLM 管线。Bring egocentric capture
into your VLM pipeline.
从真实采集、时间对齐到语义标注。先核对 E2 / E6 样例与字段，再按目标任务评估训练、评测与交付范围。From real capture and time alignment to semantic annotation. Inspect E2 / E6 samples and fields, then scope training, evaluation and delivery for your task.
公开样例可核对 · 来源分层可追溯 · 适配范围按项目确认Inspectable samples · Traceable provenance · Project-scoped adaptation
TOBI · VLM DATA ENGINEERING
你的模型，需要理解什么？What does your model need to understand?
以下是项目需求入口。现有样例展示基础观测与部分标注；完整任务集、适配与效果评测需单独确认。These are project-scoping paths. Current samples show observations and selected annotations; full task datasets, adaptation and model evaluation require confirmation.
时间定位与步骤理解Temporal grounding
定位动作起止，核对前后状态与步骤顺序。Locate action boundaries, before/after states and step order.
时间区间 + 步骤标签Time spans + step labels
评估这个任务 →Scope this task →手物交互与空间关系Hand–object understanding
核对手、物体、遮挡及坐标定义。Inspect hands, objects, occlusion and coordinate conventions.
可选关键点 + 有效掩码Optional keypoints + validity
评估这个任务 →Scope this task →长视频事件检索Video event retrieval
按事件查找片段；跨会话记忆需新增采集设计。Retrieve clips by event; cross-session memory needs additional capture design.
事件索引 + 查询证据Event index + query evidence
评估这个任务 →Scope this task →任务进度与失败理解Progress & failure
为成功、失败与纠错设计可复核监督。Design reviewable supervision for success, failure and recovery.
状态变化 + 失败原因State transitions + failure labels
评估这个任务 →Scope this task →TOBI · VLM DATA ENGINEERING
每一步，都有明确输出。A concrete output at every stage.
- 01
任务定义Define the task
采集协议 / 任务矩阵Capture protocol / task matrix
先确定问题、场景覆盖、训练与测试划分，以及验收方式。Agree on the task, scene coverage, train/test split and acceptance criteria.
- 02
真实采集Capture
原片 / 传感器 / 标定Source / sensors / calibration
保留原始母版，记录设备、会话、标定版本及可用授权范围。Preserve source media with device, session, calibration and permitted-use records.
- 03
时空对齐Align
时间索引 / 有效区间Time indexes / valid intervals
区分时钟偏移与漂移；核对丢帧、缺口、单位和坐标系。Separate clock offset from drift; check dropped frames, gaps, units and coordinate frames.
- 04
语义监督Annotate
叙述 / 证据 / 审核记录Narration / evidence / review
根据任务评估问答、动作区间和步骤标签；标明人工或模型来源。Scope QA, action spans and step labels by task; record human or model provenance.
- 05
空间辅助Enrich
可选手点 / 空间关系Optional hands / spatial relations
按项目评估，保留方法与有效性。设备算法与离线估计都不自动等于真值。Evaluate per project, preserving method and validity. Device algorithms and offline estimates are not automatically ground truth.
- 06
模型适配Adapt
样本结构 / 加载配方Sample schema / loading recipe
约定模型版本、抽帧、分辨率和预算；通过实际加载验证后再确认适配。Agree on model version, sampling, resolution and budget; confirm compatibility only after a loading test.
- 07
独立评测Evaluate
冻结测试集 / 失败分析Frozen test set / failure analysis
按人员、场地和会话隔离数据。质检报告与模型效果分别验收。Separate people, locations and sessions across splits. Validate data quality separately from model performance.
- 08
版本交付Deliver
清单 / 质检 / 变更记录Manifest / QC / changelog
按确认范围交付文件、来源记录与质量报告；新增适配和评测需单独确认。Deliver agreed files, provenance and QC. Additional adaptation and evaluation require a scoped agreement.
交付从任务与字段评审开始。下面的真实样例可核对当前输出；新增监督、模型适配和独立评测不属于默认已交付项。Delivery starts with task and field review. Samples below document current outputs; new supervision, model adapters and independent evaluation are not included by default.
来源说明：原始传感器、设备内置算法、离线估计和人工标注分别记录。OpenXR / SLAM 属于算法输出，不因设备写入而成为地面真值。所有频率、偏移和有效率数字仅描述对应公开样例。Provenance distinguishes raw sensors, device algorithms, offline estimates and human annotation. Device-written OpenXR / SLAM is not ground truth. Rates, offsets and validity figures describe the specific public samples only.
交互样例Interactive sample
标注工作台
Annotation workbench
点步骤或用左右键：两路同步跳转并播放，只响原始片。播放器 0 秒 = 记录 22 秒。框、掩码、6DoF、接触力、关节置信度为空。
Click a step or use the arrows: both players jump and play together; only raw stays audible. Player 0 s = record 22 s. Boxes, masks, 6DoF, contact force and joint confidence stay empty.
-
E6 · 22.00–23.17 · RGB 1320–1390
握持手柄
Holding controllers
双手在桌面操作区，握持深色手柄。左手追踪还没稳定。
Both hands at the desk, gripping dark controllers. Left tracking is not stable yet.
- 左手有效Left active3%
- 右手有效Right active97%
- 头路径Head path0.19m
- 陀螺 RMSGyro RMS0.15rad/s
- 右拇指–食指R thumb–index45mm
拇指到食指是关节几何距离，不是接触力。框 / 掩码 / 6DoF / 关节置信度为空。
Thumb–index is joint geometry, not contact force. Boxes / masks / 6DoF / joint confidence stay empty.
-
E6 · 23.17–25.50
看向桌面终端
Facing the terminal
左右手同时有效，仍握持深色手柄，看向桌面终端。
Both hands stay active, still gripping the controllers and facing the terminal.
- 双手同时Both active92%
- 左腕路径L wrist path1.46m
- 右腕路径R wrist path1.52m
- 陀螺 RMSGyro RMS0.67rad/s
- 拇指–食指Thumb–index73 / 56mm
拇指–食指：左 73 mm，右 56 mm。不是接触力。
Thumb–index: left 73 mm, right 56 mm. Not contact force.
-
E6 · 25.50–27.52
纸卷和布
Paper roll and cloth
转到纸卷和布。右手更稳，左手时有时无。物体没有框。
Hands move to a paper roll and cloth. Right is steadier; left comes and goes. No object boxes.
- 左手有效Left active47%
- 右手有效Right active98%
- 头路径Head path0.53m
- 右腕路径R wrist path1.28m
-
E6 · 27.52–29.50
继续纸卷和布
Still on paper and cloth
左右手同时有效，继续操作纸卷和布。头几乎不位移。
Both hands stay active on the paper and cloth. Head barely nets displacement.
- 双手同时Both active86%
- 头路径Head path0.66m
- 头净位移Head net0.03m
- 左腕路径L wrist path1.70m
-
E6 · 29.50–31.20
捏持小零件
Pinching a small part
双手捏持小零件。拇指到食指更近。没有物体框。
Both hands hold a small part. Thumb–index is closer. No object box.
- 双手同时Both active85%
- 右拇指–食指R thumb–index24mm
24 mm 是关节几何，不是夹持力。
24 mm is joint geometry, not grip force.
-
E6 · 31.20–结束E6 · 31.20–end
左手持手机
Left hand holds a phone
左手持手机看屏幕。右手有效 0%。手柄位姿文件仍是一行零。
Left holds a phone and looks at the screen. Right active 0%. Controller-pose file is one zero row.
- 左手有效Left active100%
- 右手有效Right active0%
- 头路径Head path0.05m
- 左腕路径L wrist path0.03m
屏幕文字未做检测级打码。
Screen text is not detector-masked.
E2 · 20260917_133640
双目上的后处理
Estimates on stereo
11 分 22 秒。设备只有左右目、IMU、磁力计和 16 kHz 立体声。骨架、马赛克和右上视差不是设备输出。一起播放时只响你按下的那一路。
11 min 22 s. Device files are the two eyes, IMU, mag and 16 kHz stereo. Skeleton, mosaic and corner disparity are estimates. Only the player you press stays audible.
分级数据Graded data
完整管线：真实佩戴记录 → 按深度加工成可训资产
One pipeline: worn capture → graded assets for training
用 TOBI 已落地的命名：device → indexes → human → estimate。对应采购时可按深度选档，不空喊未交付能力。样例用于核对字段；具体交付以项目清单为准。
Named with what TOBI already ships: device → indexes → human → estimate. Buyers pick depth; we do not claim undelivered layers. Use samples to inspect fields; the agreed manifest defines delivery.
D1 · DEVICE
原始采集数据
Source data
视频与 IMU / 位姿。原生帧率原片，标定随序列号。
Video plus IMU / pose. Native-Hz source; calibration by serial.
- E2 双目 / E6 多路 RGB
- IMU · 音频偏移
- 相机内外参
- E2 stereo / E6 multi RGB
- IMU · audio offset
- Intrinsics / extrinsics
D2 · INDEXES
基础对齐
Aligned base
标准化结构、多模态时间对齐、validity 质检。
Standard structure, multimodal time sync, validity QA.
- time_sync.json
- rgb / stereo 索引
- 缺列留空
- time_sync.json
- rgb / stereo indexes
- Empty columns stay empty
D3 · HUMAN
语义数据
Semantic data
让模型读佩戴者句子与关键帧：场景、物体、动作时间段。
Wearer sentences and keyframes: scene, objects, action spans.
- narration.jsonl
- 关键帧六步 / 物体时间段
- 人工核对，非检测器改写
- narration.jsonl
- Six keyframes / object spans
- Human check, not detector rewrite
D4 · ESTIMATE
行为估计（可选）
Behavior estimate (optional)
手点、轨迹辅助、隐私打码。带 method，不当设备来源数据，不作机器人动作。
Hand points, track aids, privacy mosaic. Method-tagged; not device truth; not robot actions.
- Vision 21 · estimate
- 马赛克 · 弱 OCR
- SGBM 距离样例（非深度真值）
- Vision 21 · estimate
- Mosaic · weak OCR
- SGBM distance sample (not depth truth)
D1
原片Source
双目 / 多路视频 · IMU · 标定
Stereo / multi video · IMU · calibration
清洗对齐
质检clean · sync
QA
质检clean · sync
QA
D2
对齐Aligned
标准化结构 · 多模态同步 · 场景索引
Standard structure · multimodal sync · scene index
叙述挂接
人工复核attach narration
human review
人工复核attach narration
human review
D3
语义Semantic
任务语义 · 物体 / 动作 · 结构化句子
Task semantics · object / action · structured sentences
估计提取
写 methodestimate extract
write method
写 methodestimate extract
write method
D4
估计Estimate
手部关键点 · 轨迹辅助 · 行为理解辅助
Hand keypoints · track aids · behavior helpers
VLM
模型训练主推
Primary for training
原片、叙述与时间索引是样本基础；训练前需完成任务监督、抽帧与加载验证。
Source, narration and time indexes form a sample foundation; task supervision, sampling and loading still need validation.
交付选档Delivery pick按 D2 / D3 / D4 交付
Deliver D2 / D3 / D4
文件、结构化结果、索引与 validity 报告一并给出。
Files, structured results, indexes and validity reports together.
STOCK现货小时资产
Hour stock assets
按场景加购小时；默认未标注原片，可加叙述 / 估计档。
Buy hours by scene; default unlabeled source, optional narration / estimate.
07 · 交付样例07 · Delivery samples
原片、对齐、语义与手部估计，按需求加工交付
Source, alignment, semantics and hand estimates — delivered by depth
主推 VLM 包落在 device / human / indexes；estimate 可选。MCAP 与 LeRobot 目录已预留，当前未导出。下面四类是真实交付清单，不是口号。
Primary VLM packs land as device / human / indexes; estimate optional. MCAP and LeRobot folders are reserved, not exported yet. The four cards below are the real delivery checklist.
TOBI DATA PACK
拓比交付包 · 按项目清单确认交付字段
TOBI pack · delivery fields defined by the project manifest
根据项目选深度。默认 D2；VLM 主推加 D3；手部辅助加 D4。
Pick depth per project. Default D2; add D3 for VLM; add D4 for hand aids.
D2
D2 · INDEXES
基础数据 · 标准化采集对齐
Base · standardized aligned capture
time_sync、索引、validity。适合先对接管线。
time_sync, indexes, validity. First pipeline hook.
D3
D3 · HUMAN
语义数据 · 场景 / 叙述 / 关键帧
Semantic · scene / narration / keyframes
VLM 主推档：人工句子可核对。
Primary VLM tier: checkable human sentences.
D4
D4 · ESTIMATE
行为估计 · 面向策略辅助
Behavior estimate · policy helpers
手点与打码可选；不写成机器人动作或深度真值。
Optional hands / mosaic; not robot actions or depth truth.
01
数据文件
Data files
- 视频 / 双目或多路原片
- IMU / 传感器流
- 时间同步与音频偏移
- Video / stereo or multi-cam source
- IMU / sensor streams
- Time sync and audio offset
02
结构化结果
Structured results
- 叙述 / 关键帧 / 物体时间段
- 手部轨迹（设备或 Vision 估计）
- 动作时间段（人工核对）
- Narration / keyframes / object spans
- Hand tracks (device or Vision estimate)
- Action spans (human-checked)
03
数据索引
Data index
- Episode / 任务时间戳
- 场景与文件关联
- RGB / 手部 / 同步索引
- Episode / task timestamps
- Scene and file links
- RGB / hand / sync indexes
04
质量报告
Quality report
- 数据量与有效率
- 异常与门禁
- QA / validity 验收
- Volume and yield
- Anomalies and gates
- QA / validity acceptance
device/原始流与标定
训练读这里Raw streams & calibration
Train here
训练读这里Raw streams & calibration
Train here
estimate/离线算法
带 methodOffline algorithms
with method
带 methodOffline algorithms
with method
human/叙述与关键帧Narration & keyframes
indexes/时间同步
validityTime sync
validity
validityTime sync
validity
preview/低帧率 mp4
仅供人看Low-Hz mp4
humans only
仅供人看Low-Hz mp4
humans only
用于模型训练Model training · VLM / 场景描述
用于数据有效性检查Evaluation · validity · 工作台核对
用于数据研究Research · 字段与时间轴审计
VLM 最小集：device/ 原片、标定与适用的音频偏移 · human/narration.jsonl（及 E6 关键帧）· indexes/time_sync.json · validity。预览 mp4 与未门控的估计距离不进训练列，公开模型分数也不写入。
Minimum VLM set: device/ source, calibration, and audio offsets where applicable · human/narration.jsonl (plus E6 keyframes) · indexes/time_sync.json · validity. Do not put preview mp4, ungated estimated distance, or public model scores into a training column.
episode_bundle.v1.json embodied_layers.v1.json 交付分级说明Delivery tier note
TOBI · VLM DATA ENGINEERING
先约定怎么验收，再扩大采集。Agree on acceptance before scaling.
文件与同步质量Files & synchronization
核对 schema、时间单调性、文件校验和、丢帧、缺失区间和同步残差。阈值、样本数与验收分母在项目中约定。Check schema, monotonic timestamps, checksums, frame drops, gaps and sync residuals. Agree on thresholds, sample size and denominators.
独立任务评测Independent task evaluation
问答看正确性与证据；时间定位看区间重叠；空间定位看坐标误差；失败识别看漏检。固定模型、提示、抽帧及数据划分后再报告结果。Assess answer evidence, temporal overlap, spatial error and missed failures. Freeze model, prompts, sampling and data splits before reporting results.
公开 validity 报告用于数据检查，不代表模型效果。当前不提供本页数据上的模型性能承诺。Public validity reports describe data checks, not model performance. No model-performance claim is made for these samples.
TOBI · VLM DATA ENGINEERING
对接前，先明确边界。Clarify the scope before integration.
这些数据可以直接训练 VLM 吗？Can I train a VLM directly?
可作为构建样本的基础。需要按目标任务完成监督标注、数据划分、抽帧、输入格式和加载验证；少量展示样例不等于完整训练集。They are a foundation for samples. Task labels, splits, sampling, input formatting and loading require validation; a small demo is not a complete training set.
VLM 都需要音频吗？Does every VLM need audio?
不需要。纯视觉语言任务使用图像／视频与文本；音频按目标模型能力另设输入或转录分支，并保留偏移与来源。No. Visual-language tasks use images/video and text. Audio requires a supported model input or a transcription branch with offset and provenance.
是否已经适配 LeRobot 或指定模型？Are LeRobot or model adapters verified?
当前公开页的 MCAP／LeRobot 目录为预留，并非已验证导出。适配须确认版本、字段、实际加载与任务范围后再约定。The public MCAP/LeRobot directories are reserved, not validated exports. Confirm version, fields, actual loading and task scope before committing.
可以作为机器人动作训练包吗？Is this a robot action training pack?
当前公开包未提供面向目标机器人的已验证动作监督。人体观测可用于表征或研究，但不能直接当作机器人控制动作；接触力是否需要取决于任务。The public pack does not provide verified action supervision for a target robot. Human observations may support representation research but are not robot control actions; force requirements depend on the task.
TOBI · VLM DATA ENGINEERING
把你的任务，变成可交付的数据需求。Turn your task into a scoped data project.
从一段样例，开始技术评审。Start with one sample and a technical review.
告诉我们目标任务、现有模型或缺失字段。先确认样例与交付范围，再讨论小规模 PoC 和批量采集。Share your task, model or missing fields. Confirm sample and delivery scope before discussing a small PoC and volume capture.
- 字段与来源核对Review fields and provenance
- 适配与验收范围确认Agree on adaptation and acceptance
- 试采、质检与版本交付Pilot capture, QC and versioned delivery
TOBI · VLM DATA ENGINEERING
可追溯的技术依据Traceable technical references
方法参考，非 TOBI 数据上的性能证明。Method references, not performance evidence on TOBI data.
方法、来源与样例限制Methods, provenance and sample limitations
技术方案Method
手部识别：先写能核对的 21 点
Hands: write 21 points you can check
当前公开样例未包含 MANO 重建结果。E2 用 Apple Vision 在原片上写 21 点，每个点带置信度。E6 的手仍是设备 OpenXR，不用这张图替换。
The current public sample does not include MANO reconstruction. On E2, Apple Vision writes 21 points on the source frame, each with a confidence. E6 hands stay device OpenXR. This picture does not replace them.
-
01
现在
Now
原片上的 21 点进入 estimate，并写明 method。低于 0.5 的点标成无效，不删掉。
The 21 points go into estimate with a method field. Points below 0.5 are marked invalid, not deleted.
-
02
后续可评估项
Potential follow-up evaluation
WiLoR / HaWoR 可作为后续方法候选，需验证输入、尺度与误差；当前未交付这些结果。
WiLoR / HaWoR are potential methods requiring input, scale and error validation; their results are not currently delivered.
-
03
不做
Not done
不把两目视差写成米。不把 E6 的设备关节换成视觉网格。不把对齐后的论文毫米数写成这段的精度。
Do not turn two-eye disparity into meters. Do not replace E6 device joints with a vision mesh. Do not treat a paper’s aligned millimeter score as this take’s accuracy.
18/21左手有效点Left joints kept
15/21右手有效点Right joints kept
0.5置信度门限Confidence gate
estimate不是设备输出Not a device output
这一帧来自已公开的 E2 样例左目，文件名序号 0120，曝光时刻没有重新读取。另一段原片（uid 532f0cf9a78d，不是 20260917）已按同样方法在左右目各跑 613 秒：左目 593 秒有手，右目 588 秒有手。那份 1 Hz 坐标不替代 30 Hz 原片。
This frame is the published E2 left eye, filename index 0120. The exposure time was not re-read. A different take (uid 532f0cf9a78d, not 20260917) was run the same way for 613 seconds on each eye: a hand is present on 593 left seconds and 588 right seconds. That 1 Hz file does not replace the 30 Hz source.
信息架构Architecture
device · estimate · human
同一段采集拆成三层。缺的列留空。训练读原片原生帧率，不读 4 Hz / 10 Hz 预览。
One capture splits into three namespaces. Missing columns stay empty. Train at native source Hz, not the 4 Hz / 10 Hz preview.
device
设备写过的
From the device
只收采集设备或出厂标定写进文件的流。
Only streams written by the capture device or factory calibration.
- RGB / 灰度 / IMU / 磁力计 / 音频
- 头部位姿 · OpenXR 手
- 相机内外参 · 时间同步
- RGB / grayscale / IMU / magnetometer / audio
- Head pose · OpenXR hands
- Camera intrinsics & extrinsics · time sync
estimate
离线算法
Offline algorithms
必须带 method，并写 not_device_output。
Must carry method and not_device_output.
- Vision 21 点手 · SGBM 距离
- 马赛克 · OCR
- MediaPipe / RTMPose / HaMeR / FoundationStereo 槽位保留，未在原片重跑
- Vision 21-point hands · SGBM distance
- Mosaic · OCR
- MediaPipe / RTMPose / HaMeR / FoundationStereo slots reserved, not re-run on source
human
人工核对
Human review
句子不改写成检测器输出。
Sentences are never rewritten as detector output.
- 叙述 · 六步关键帧句
- 物体时间段
- 接受标记
- Narration · six keyframe sentences
- Object time spans
- Accept flags
embodied_layers.v1.json E2 validity E6 validity vision_pipeline_v1.json
层说明Layer notes
IMU · 距离 · 检测
IMU · Distance · Detection
设备来源数据Device不是从画面反推Not from the picture
IMU 数据
IMU data
设备陀螺和加速度计给出角速度和线加速度。E2 陀螺和加速度约 799 Hz，磁力计 200 Hz，加速度模长抽样均值 9.83 m/s²。E6 陀螺和加速度约 1015 Hz。E2 加速度比左目曝光起点早 124 毫秒。距离片底部曲线来自 gyro.csv。
E2 这 682 秒按 5 秒窗口角速度均方根切开。中位数 0.352 rad/s。高于 1.8 倍且持续至少 10 秒的画成橙色。这是头戴角速度，不是手部动作，也不是位姿。最长日常段 150–505 秒；本段最高 595–605 秒，均方根 2.188 rad/s。积分航向会漂移，不是 SLAM，E2 也没有头部位姿文件。
Device gyro and accelerometer record angular velocity and linear acceleration. E2 gyro/accel ≈ 799 Hz, magnetometer 200 Hz, accel magnitude subsample averages 9.83 m/s². E6 ≈ 1015 Hz. E2 accel starts 124 ms before left-eye exposure. The curve under the distance clip is gyro.csv.
The 682 s are split by 5-second gyro RMS (median 0.352 rad/s). Spans above 1.8× that median and ≥10 s are orange. That is head-worn rate, not a hand verb or pose. Longest usual stretch 150–505 s; peak 595–605 s at RMS 2.188. Integrated heading drifts — not SLAM; E2 has no head-pose file.
橙：角速度升高。灰：其余。40–48 秒样例落在第二段橙色（30–100 秒，RMS 1.818）。 motion_spans.jsonl
Orange: elevated rate. Gray: the rest. The 40–48 s sample sits in the second orange span (30–100 s, RMS 1.818). motion_spans.jsonl
视频估计Video estimate不是深度相机Not a depth camera
三维距离
Distance from stereo
两段都没有深度相机文件。这 8 秒取自 E2 40.0–48.0 秒，4 Hz，只供核对。左为 Camera0 原片 1600×1200（未校正），屏幕与远处的人打了马赛克，佩戴者手不打。右为校正后 Z = f·B / d（四元数 xyzw）：焦距约 253 px，基线约 66 mm。SGBM 后左右一致 + 置信门控：有效像素约 40%，中位距离约 1.34 m，有效区置信中位约 0.90，裁在 0.25–6 m。黑块不填洞。当前样例未采用 WLS 后处理。
estimate 层输出。径向只用前三个系数，未经棋盘格复核。音频 16 kHz 立体声，比左目曝光晚 97 ms。训练读 30 Hz 原片。FoundationStereo（CVPR 2025，第三方）槽位已留，当前未运行。E6 RGB 基线约 60 mm。标称 SLAM 5 mm 需按任务验证，本段无地面真值。
Neither episode has a depth-camera file. These 8 s are E2 40.0–48.0 at 4 Hz for checking. Left is original Camera0 1600×1200 (not rectified); screen and distant people mosaicked; wearer hands kept. Right is rectified Z = f·B / d (xyzw): ~253 px focal, ~66 mm baseline. After SGBM: LR check + confidence gate → ~40% valid, median ~1.34 m, confidence median ~0.90, clipped 0.25–6 m. Black not filled. This sample does not use WLS post-processing.
Estimate-layer only. Three radial coeffs, no checkerboard check. Audio 16 kHz stereo, +97 ms. Train at 30 Hz source. FoundationStereo reserved, not run for this sample. E6 RGB baseline ~60 mm. Nominal 5 mm SLAM still needs per-task verification; no ground truth here.
回到样例Back to sample depth_sample.json stereo_distance_sample.json E2 全长样例Full E2 sample
目标检测与分割
Detection, not yet segmentation
E2 标注片已画：Vision 21 点手、人脸、远处人体、亮屏幕、蓝色墙标、置信度 ≥ 0.40 的文字。707 秒里：右手 525、左手 440、双手 410、人体 221、人脸 77、蓝标 32、亮屏 642。无类别名。OCR 常误读，弱结果不当招牌。手部包装 hand_2d_vision21.json（estimate，非 OpenXR，关节置信空，HaMeR 未跑）。
无实例掩码。E6 物体只有人工时间段；E6 手为设备 OpenXR 26 关节 hand_openxr_device.json（文件在 estimate 目录，命名空间仍是 device）。类别框/掩码另开文件，不改写现有框。多目三角化、4 目以上离群剔除、关节 Butterworth 不用于这两段：E2 只有两目，且 707 行只有计数、没有关节点坐标。E6 手部索引按 RGB 帧对齐，最大间隔约 19.6 秒，不能平滑补洞。
E2 annotated preview draws Vision 21-point hands, faces, distant people, bright screens, blue wall marks, and text at confidence ≥0.40. Across 707 s: right 525, left 440, both 410, person 221, face 77, blue mark 32, screen 642. No class names. OCR often misreads; weak hits are not treated as signage. Hand package hand_2d_vision21.json (estimate, not OpenXR; joint confidence empty; HaMeR not run).
No instance masks. E6 objects are human time spans only. E6 hands are device OpenXR 26 joints: hand_openxr_device.json (folder is historical; namespace stays device). Class boxes or masks go in a separate file; existing boxes are not rewritten. Multi-camera triangulation, 4-plus-view outlier rejection, and joint Butterworth are not used on these takes: E2 has two eyes, and the 707 rows are counts without joint coordinates. The E6 hand index follows the RGB clock and has a gap of about 19.6 s, so the hole is not filled by smoothing.
具身智能自动标注
Automatic annotation
一段采集先对齐时钟，再按算法各写一份。已落盘：E6 设备关节与头部位姿；E2 手部 21 点、打码、陀螺窗口，以及页首 8 秒距离样例。场景句子仍是人工核对。预览片只用来看，训练读原始视频。
Align the clock first, then each algorithm writes its own file. On disk: E6 device joints and head pose; E2 21-point hands, mosaic, gyro windows, and the 8 s distance sample above. Scene sentences stay human. Preview is for looking; training reads source video.
训练门禁Training gate
清理：这些不进训练列
Drop what should not train
源文件不删。下面这些不写入训练列，也不用别的数填上。
Source files stay. These do not enter a training column, and nothing is invented to fill them.
- 用 4 Hz / 10 Hz 预览代替 E2 30 Hz、E6 ~60 Hz 原片。
- 把后处理估计写成设备来源数据。
- 文字置信度 < 0.40，或不足两个字母/数字。
- 盖住佩戴者双手的人体框。
- 深度、物体 6DoF、关节置信度缺失时补假值。
- E6 里 active ≠ 1 的关节当作可见。
- 没有头部位姿却套用工装时间对齐（这两段都没有套用）。
- 把人工句子写成检测器输出。
- 双目匹配失败像素插值补洞。
- 用两目去跑至少三目的三角化，或在没有左右对应关节点时写出 3D 手。
- 把手部索引里的时间空洞用平滑填上，再当成识别轨迹。
- Train on 4 Hz / 10 Hz preview instead of native source Hz.
- Write a later estimate as a device measurement.
- Drop text below confidence 0.40, or with fewer than two letters or digits.
- Person boxes that cover the wearer’s hands.
- Invent depth, object 6DoF, or joint confidence when absent.
- Treat E6 joints with active ≠ 1 as visible.
- Apply factory time alignment without head pose to check.
- Treat a human sentence as detector output.
- Fill stereo match failures with interpolation.
- Run a three-or-more-camera triangulation on two eyes, or write a 3D hand without left-right joint correspondence.
- Smooth across a gap in the hand index and call the result a recognition track.
对照Compare
按输出格式拆
By output format
预览 mp4 用来核对。训练读原始视频和索引。
Preview mp4 is for checking. Training reads source video and indexes.
| 格式 | E6 | E2 |
|---|---|---|
| 原始视频 | rgb / tracking / ctrl，60 Hz 量级 | Camera0、Camera1，1600×1200，30 Hz |
| 预览 | 10 Hz，从 22 秒起，带关节投影和声音 | 4 Hz 标注片，带声音；另有无标注并排片 |
| 时间索引 | time_sync.json · rgb_index.jsonl | time_sync.json · stereo_index.jsonl |
| 手 | hands_flags.jsonl + hand_openxr_device.json | hand_2d_vision21.json + e2-findings.jsonl |
| 句子 | narration.jsonl · e6_keyframe_step.v1.json | narration.jsonl |
| 物体 | object_spans.jsonl | 没有类别框 |
| 有效性 | e6 validity | e2 validity |
| Format | E6 | E2 |
|---|---|---|
| Source video | rgb / tracking / ctrl, about 60 Hz | Camera0 and Camera1, 1600×1200, 30 Hz |
| Preview | 10 Hz from record 22 s, with joint projection and audio | 4 Hz annotated preview with audio, plus an unannotated side-by-side |
| Time index | time_sync.json · rgb_index.jsonl | time_sync.json · stereo_index.jsonl |
| Hands | hands_flags.jsonl + hand_openxr_device.json | hand_2d_vision21.json + e2-findings.jsonl |
| Sentences | narration.jsonl · e6_keyframe_step.v1.json | narration.jsonl |
| Objects | object_spans.jsonl | No class boxes |
| Validity | e6 validity | e2 validity |
按算法需求拆
By algorithm need
客户要哪一种，就下哪一种文件，不要把估计写成设备来源数据。
Take the file that matches the need. Do not write an estimate as a device measurement.
| 要训练的输入 | 用 E6 | 用 E2 |
|---|---|---|
| 手怎么动 | 设备 OpenXR 26 关节，约 19 Hz | Vision 21 点，估计 |
| 头在哪里 | head_pose.csv，60 Hz | 无头部位姿；陀螺积分会漂移 |
| 场景一句 | 四句人工叙述 | 一句人工叙述 |
| 文字和隐私 | 本页预览无检测级打码 | 人脸、远处人体、文字 ≥0.40、亮屏、蓝标 |
| 远近 | 预览无视差；SLAM 5 mm 标称 | SGBM + LR + 置信；FoundationStereo 未跑 |
| 声音 | 44.1 kHz 单声道，+288 ms | 16 kHz 立体声，+97 ms |
| Input to train | Use E6 | Use E2 |
|---|---|---|
| How the hands move | Device OpenXR 26 joints, about 19 Hz | Vision 21-point estimate |
| Where the head is | head_pose.csv, 60 Hz | No head pose; integrated gyro drifts |
| One scene sentence | Four human sentences | One human sentence |
| Text and privacy | This preview is not detector-masked | Faces, distant people, text at 0.40+, bright screens, blue wall marks |
| Near and far | No disparity on the preview; the 5 mm SLAM figure is nominal | SGBM + left-right check + confidence gate; FoundationStereo not run |
| Audio | 44.1 kHz mono, +288 ms | 16 kHz stereo, +97 ms |
两段都没有接触力、物体 6DoF、机器人末端动作。E6 的 grayscale tracking 与朝下 ctrl 在观测里；ctrl 不是手柄位姿（手柄文件一行零）。E2 没有这些灰度相机。
Neither episode has contact force, object 6DoF, or robot eef action. E6 grayscale tracking and downward ctrl are observations; ctrl is not controller pose. E2 has no grayscale cameras.