SO-101
Course overview/Chapter 08 · Data Collection
中文
Chapter 08Data & Training~20 min

Data Collection

Companion page: the "Data Collection" tab in the console

7 sections 3 figures 9 code blocks Collection30 fpsTimestamps
Data collection page

1. Why Collect Data#

What the first seven chapters did, in essence, was teach the robot arm to "understand commands and reach a position." But teaching the arm a motion (grabbing a cup, opening a drawer) takes more than commands: a human has to perform it once and the machine has to record it once:

text
Human guides the arm through the motion  →  record "what it sees" and "what moved" together at 30 Hz  →  train the policy  →  the arm performs it on its own

In this pipeline, collection quality directly determines the training result. An unstable frame rate, mismatched timestamps, cameras and joints half a frame apart — and the trained policy ends up "looking left while grabbing right."

So data collection has to answer three questions:

Question Answer on this page
What to record Six joint angles + camera frames
How fast 30 Hz, fixed period
How to align Three timestamps per frame; every difference is written down

While collecting, the robot base and the third-person camera must stay fixed. The base defines the robot frame and workspace; the third-person camera defines what the model sees. Move either one and the pixels, table coordinates and safety boundaries for the same joint angles all change. A dataset collected across different positions teaches the model contradictory state → action pairs, and both training and inference degrade badly. Fix the base and camera before recording; if you moved them mid-dataset, recalibrate and record a new dataset instead of appending to the old one.

2. What to Collect#

Joint angles (6-D)#

text
shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper

Why record joint angles instead of the TCP pose?

Joint angles TCP pose
Data volume 6 numbers 5 numbers
Unique Yes — one set of joint angles uniquely determines one pose No — the same position may admit several joint-angle solutions
Convertible joint angles → FK → pose, computable at any time pose → IK → joint angles, possibly multiple solutions or none
Information loss None the actual elbow/wrist configuration is lost

So joint angles are the raw quantity and the pose is a derived quantity. Record the raw quantity; when a pose is needed, compute it on the spot with the FK from Chapter 4.

This is also why this page does not collect TCP poses: they can be computed from joint angles, and storing a second copy only causes trouble when the two columns disagree.

Camera frames (multiple selectable)#

A real robot data-collection setup usually uses two cameras:

  • Wrist camera (wrist): mounted on the wrist or beside the gripper. It moves with the arm and shows the gripper, the object and the contact point up close.
  • Third-person camera (top / external): fixed to the side of the table, a bracket, or above the work area. It shows the whole arm, the table and the task region. The console calls this role top, but it does not have to be mounted directly overhead.

One episode can bind multiple cameras. For production data, tick both views by default. The three choices are:

Selection Files written
Top only top_rgb/episode-000.mp4 → observation.images.top_rgb_ns
Wrist only wrist_rgb/episode-000.mp4 → observation.images.wrist_rgb_ns
Both two mp4 files → two *_ns columns

Trade-offs:

Setup Pros Cons
Two cameras (recommended)
Production collection
The third-person view gives task context; the wrist view gives gripper/contact detail; when one is occluded the other still sees USB bandwidth, CPU and disk all double; both timestamp alignment and duplicate frames need attention
Third-person only
Quick validation, distant tasks
Low resource use, shows the whole arm and table; simple to debug The gripper/contact point is small and easily occluded; precise manipulation is hard
Wrist only
Contact close-ups, supplementary validation
Close view of the gripper and object, no body occlusion No global position or task context; fast arm motion can blur the image

One camera is not the recommended default; it is a fallback when resources are tight. Two 720p@30 streams do compete for USB bandwidth and CPU, but with MJPG, two driver buffers, one capture thread per stream and the 3D view turned off, the measured result is a stable 30.1 Hz. Prefer two cameras for production data; fall back to one only when bandwidth/CPU is genuinely insufficient, or when you are just validating the pipeline.

In the console, the top view is ticked by default and the wrist view is not; for production collection, tick the wrist too. "Recording views" records exactly the streams you tick; the file layout is in §4.

Timestamps (three per frame)#

Timestamp alignment
Field Meaning
hardware_ns monotonic_ns of this sampling period; the primary timeline
observation.state_ns instant the joint angles were read (same period as the row above, identical value)
observation.images.top_rgb_ns instant the camera frame was actually captured

The difference between the last two and the primary timeline is the frame's latency. A monotonic clock is used instead of wall time because monotonic is never adjusted by time synchronization (NTP), so cross-frame differences are always reliable.

3. How 30 fps Is Guaranteed#

The recording thread runs on a fixed period:

text
Every 1/30 s (33.3 ms):
  ① take a monotonic timestamp                  → hardware_ns
  ② read the six joint angles                    → observation.state
  ③ take the latest camera frame + its capture time  → observation.images.<key>_ns
  ④ write to the in-memory buffer, and feed the frame to the encoder

Three details decide whether the frame rate holds at 30:

A. The camera must be explicitly set to MJPG + 30 fps. On USB 2.0 bandwidth, 1280×720 only reaches 30 fps with MJPG encoding. Left unspecified, many cameras negotiate 10 fps by default — the recorder shows up every period, but the camera only produces a frame every 100 ms, hence a flood of duplicate frames. Measured:

Before After
Actual camera frame rate 10.3 fps 30.2 fps
Duplicate frames out of 108 72 2

B. Turn off the camera's "dynamic frame rate". Many UVC cameras (especially these small USB ones) have exposure_dynamic_framerate enabled by default: when the image darkens, the camera lengthens the exposure to compensate, at the cost of dropping to the low 20s fps. The wrist camera shows this most clearly — 30 fps pointing at the ceiling, low 20s the moment it turns toward the table, bouncing back and forth.

The camera is opened with:

bash
v4l2-ctl -d /dev/videoX -c exposure_dynamic_framerate=0

With it off, the driver can only compensate brightness with gain within 1/30 s, and the frame rate stays at 30. You can check it yourself: v4l2-ctl -d /dev/video0 --list-ctrls | grep dynamic — the value should be 0.

C. The driver buffer must not be set to 1. With a single V4L2 buffer the driver has nowhere to stage the next frame, so any slightly slow read drops it. Measured with two streams:

CAP_PROP_BUFFERSIZE Delivery rate per stream
1 21.0 / 20.3 fps
2 30.3 / 30.3 fps
4 30.3 / 30.3 fps

Two buffers are enough to run at full rate; latency grows by only one frame (about 33 ms).

D. Each camera gets its own capture channel. Top and wrist each have their own frame-grabbing thread and their own frame slot, so neither blocks the other. Each stream carries a frame sequence number and a delivery rate, shown directly on the page:

text
top /dev/video2 1280×720 · 30.3 fps
wrist /dev/video0 1280×720 · 30.3 fps

During recording the "Live" card also lists the delivered frame count and latency per stream. Measured with both streams recording:

Single stream Both (after fixing)
Actual frame rate 30.0 Hz 30.1 Hz
Duplicate frames 2/119 2/181
Max latency 10.7 ms 27.9 ms

E. Sampling must wait for a new frame, not take whatever the tick finds. Camera frame intervals jitter. One wrist camera measured: 32.5, 31.5, **36.0**, 32.1, 32.0, **36.0** … — 30 fps on average, but every few ticks there is a 36 ms gap. Ticking at a fixed 33.3 ms lands on the gap and picks up a stale frame, which is then recorded as a duplicate.

When the code finds the frame timestamp unchanged on a tick, it waits up to 20 ms for a new frame and only reuses the old one if none arrives. Measured:

Wait window Duplicate frames in a 20 s recording (600 frames)
none 40 ~ 90
12 ms 2
20 ms 0

The window cannot grow indefinitely: waiting too long makes sampling fall behind the tick. 20 ms is the measured value that suppresses duplicates without missing ticks.

F. If the loop falls behind, reset the tick — do not catch up by inserting frames. If a period overruns (a camera stall, for example), the next tick re-aligns to the current instant instead of firing several frames back to back. Catch-up frames create fake data that "looks like 30 fps" while the timestamps bunch up.

G. Duplicate frames are counted separately; timestamps are not rewritten. When a camera lags by one frame, that sample reuses the previous image, but the timestamp still belongs to this period — downstream code can tell from drop_stats that a frame here was reused.

H. Turn off the 3D view while recording. The pose view on the right of the page renders continuously; measured, it can push the browser to 8 cores and squeeze two camera streams down to 21 fps. Click "3D on / 3D off" in the title bar to turn it off; this releases the WebGL context and the render loop. The capture side is unaffected and both streams stay at a stable 30 fps.

INTERACTIVE LAB

Frame-rate check

Measured scenario replay

The recording thread ticks every 33.3 ms; if the camera misses a beat, that tick reuses the previous frame. Pick a scenario and watch 1 second of ticks, camera frames, and duplicate frames.

Recording ticks (30 Hz)New camera framesDuplicate frames
Camera output30.2fps
Duplicate frames2/ 108
Duplicate share1.9%
Scenarios and numbers come from the measured table in §3 of Chapter 8. The >2% warning and >10% error thresholds also come from Chapter 9.

4. File Structure#

Each recording writes one episode, appended to the dataset directory:

text
recordings/<dataset name>/                  ← relative path under the workspace; created if missing
├── meta/
│   ├── info.json                           ← summary: fps, features, cameras, path templates
│   ├── stats.json                          ← per-feature statistics (mean/variance/min/max)
│   ├── tasks.jsonl                         ← task names (task_index → task)
│   └── episodes/chunk-000/episode-000.parquet   ← metadata for this episode
├── data/
│   └── chunk-000/episode-000.parquet       ← per-frame data (one row per frame)
└── videos/
    └── chunk-000/
        ├── top_rgb/episode-000.mp4         ← top view (one video frame per parquet row)
        └── wrist_rgb/episode-000.mp4       ← wrist view (written only if selected)

chunk-000 is a shard: a new chunk every 1000 episodes, so a single directory never holds too many files.

meta/info.json#

json
{
  "format": "pooled_episodes_v1",
  "codebase_version": "v3.0",
  "fps": 30.0,
  "task_name": "UMI_9_29",
  "cameras": {"top_rgb": {"height": 720, "width": 1280}},
  "data_path": "data/chunk-{c:03d}/episode-{f:03d}.parquet",
  "video_path": "videos/chunk-{c:03d}/{image_key}/episode-{f:03d}.{ext}",
  "features": {
    "observation.state":            {"dtype": "float32", "shape": [6]},
    "observation.state_ns":         {"dtype": "int64", "shape": [1]},
    "observation.images.top_rgb_ns":{"dtype": "int64", "shape": [1]},
    "action":                       {"dtype": "float32", "shape": [6]}
  }
}

data/chunk-000/episode-000.parquet#

Each row = one frame, 30 rows per second.

Column Type Description
episode_index int64 which episode
frame_index int64 frame number within the episode, starting at 0
timestamp double seconds since the start of the episode
task_index int64 task ID
wall_time double absolute time (Unix seconds, for cross-device alignment)
hardware_ns int64 primary timeline (monotonic)
observation.state float32[6] the six joint angles
observation.state_ns int64 instant the joint angles were read
observation.images.top_rgb_ns int64 capture instant of the top camera frame
observation.images.wrist_rgb_ns int64 capture instant of the wrist camera frame (column present only if recorded)
action float32[6] measured joint angles written by the current recorder (the same data as observation.state)

The current recorder writes the same values into action and observation.state. If training does not apply a next-state/shift, ACT learns state → state; confirm the training-side sequencing before using this data. This page describes the current recorder as-is and does not change recording behavior.

meta/episodes/chunk-000/episode-000.parquet#

One row per episode, describing the recording itself:

Column Description
episode_index / task_index IDs
start_frame_index / end_frame_index / length position and length within the global frame sequence
duration_sec duration
drop_stats duplicate frames, dropped frames, max camera latency
video_codec encoder and parameters

Videos#

videos/chunk-000/<image_key>/episode-000.mp4 — H.264, 1280×720, 30/1 fps, with a frame count strictly equal to the parquet row count.

Duplicate frames do not compromise "720p30". The video is always exactly 30 frames per second with 33.3 ms row spacing; when a camera delivers no new image, the same picture is simply written twice and the timeline does not drift. Measured on a 581-frame recording: median row spacing 33.3 ms, maximum 33.5 ms.

There is one subdirectory per selected view, one mp4 per subdirectory, and each frame count equals the parquet row count. On read, row N corresponds to frame N; no extra index is needed.

5. Using the Page#

Data collection page: button locations

The numbers in the screenshot correspond to the steps below.

Prerequisite: connect both arms on the Teleoperation page first. Data collection is teleoperation with recording: a person moves the leader, the follower follows, and this page records the result. Connecting only the follower produces no demonstration motion. On the Teleoperation page, fill in both the leader and follower serial ports and confirm the Leader/Follower Comparison has leader readings and the follower is holding torque. If you use absolute mapping, align the arms and start teleoperation before returning here. "Start recording" on this page only records; it does not start teleoperation for you.

Before starting, confirm once more that the robot base and third-person camera have not moved (see the warning at the end of §1). If they did, restore/recalibrate first; do not keep recording at the new position.

Preparation

# Action
④ In "Camera view", click "Scan devices" to list the candidates, choose one device for top (third-person) and one for wrist (you decide which is which), then click "Connect" on each
⑤ Check both previews to confirm they are not swapped: the one showing the gripper fingers is the wrist. Each label includes the actual frame rate; 30 fps is normal
① Enter a dataset name (e.g. UMI_9_29); the last input is remembered
② Under "Recording views", tick the streams to record; for production collection, tick both top and wrist (see §2 Camera frames)
⑧ Turning the 3D view off first is recommended (see the note below)

Recording

# Action
③ Click "Start recording"; the card below shows Recording · episode #N · frames
⑥ While recording, watch "Live": actual rate, duplicate frames, latency
③ Click "Stop recording" when done; shows Saved episode #N · N frames / x.xs · duplicate frames n · max latency x ms
③ If you do not want this take: click "Abort"; the data is discarded, nothing is written to disk, no episode number is consumed, and the next recording reuses the same number
⑦ Under "Existing datasets", confirm the episode count went up by 1 and the size grew

Episode numbers on the page and in the logs are 1-based, starting at #1; file names on disk still follow the LeRobot v3 0-based convention (episode-000.mp4 corresponds to episode #1 in the log). The number is governed by the last_episode_index high-water mark in meta/info.json and is strictly increasing: deleting the last few episodes on the QC page and recording again only continues forward, never reusing an old number.

Automatic pre-flight check

When you click "Start recording", the server first measures for 1 second: if any camera is below 27.5 fps it refuses to start and names the offending stream:

text
wrist camera is at 24.8 fps (requires ≥27.5). Check the cable/USB port,
or close other CPU-heavy programs and retry

This way you do not waste a take on data full of duplicate frames.

Checks

Metric Normal range What an abnormal value means
Actual rate 29–30 Hz Clearly below 30 → the camera is not running at 30 fps
Duplicate frames < 2% 2–10% triggers a QC warning; a high ratio means the camera frame rate is too low or the buffer is wrong
Dropped frames 0 Non-zero → the camera disconnected
Max camera latency < 50 ms Too large → USB bandwidth or CPU contention
Per-stream stalls 0 Non-zero means that stream dropped out (a cable, most often)
Measured camera fps 30.0 This is the camera's own output rate (not the sampling rate); below 29 means that camera cannot keep up
Max camera interval ≈33 ms 60 ms or more means the camera occasionally stalls

Turn off the 3D view before recording (⑧)

The 3D scene in the pose view renders continuously; measured, it can push Chrome to 8 cores and squeeze camera capture down to 21 fps. Click "3D on / 3D off" in the card's title bar to turn it off; this releases the WebGL context and the render loop, and the placeholder shows "3D view off (no CPU/GPU use)".

With it off, both camera streams stay at a stable 30 fps. The switch state is remembered and persists the next time the page is opened.

6. FAQ#

Symptom Cause Fix
Actual rate only 10 Hz The camera did not negotiate 30 fps Reconnect (the system opens cameras with MJPG + 30 fps)
Many duplicate frames Camera frame rate below the sampling rate Same as above; or use 480×270 / 640×360
Only one stream has many duplicates That stream lost CPU / loose cable See "Diagnosing a single stream's dropped frames" below
The two views are swapped Wrong device chosen for top/wrist Judge from the picture, change the device path, reconnect
Error immediately on start Camera not connected / arm not connected Connect first
Episode numbering starts at 1 New dataset Normal; numbering is strictly increasing within a dataset, the log shows #N, and the QC page shows the same N
Something is wrong mid-recording This take is unusable Click "Abort": the data is discarded and no number is consumed; the next recording still uses this number
Video and parquet row counts differ The encoder dropped frames This implementation writes every frame, so it cannot happen; if it does, the disk is full

Diagnosing a single stream's dropped frames#

Duplicate frames are counted per stream; after recording, look at drop_stats in meta/episodes/*.parquet to see which camera:

json
{"top_rgb_duplicate": 1, "wrist_rgb_duplicate": 61,
 "top_rgb_lag_max_ms": 11.8, "wrist_rgb_lag_max_ms": 49.0}

Two diagnostics:

What to look at What it means
Only one stream has a high duplicate count The problem is in that stream (the camera itself / its cable / its thread being starved)
That stream's lag_max_ms is clearly larger That stream stalls periodically; it is not merely a low frame rate

Measured comparison: recording 20 seconds with the arm stationary gave 600 frames per stream with only 1 duplicate each — so the capture pipeline itself is fine; drops that appear only once the arm moves point first to the wrist camera cable being pulled along with the arm or the host CPU being saturated at the time.

Troubleshooting order:

  1. Watch the live frame rates on the two page labels (top … 30.0 fps). With the arm stationary, whichever stream stays below 29 is almost certainly that camera or its cable.
  2. Turn off the 3D view and record again (see §3 F).
  3. Do not run other preview/recording tools at the same time.
  4. Change USB ports: put the two cameras on different controllers if possible (lsusb -t shows Bus numbers).
  5. Wrist camera: use a softer, shorter cable, add strain relief, and do not bundle it with the motor harness.
  6. If it still fails, drop that stream to 640×480 and retry — decoding and data volume are both halved.

Both streams dropping at once means the problem is not the cameras but a saturated host. Clear it in this order:

  1. Close other tabs (every tab with the "Data Collection" page open pulls two previews)
  2. Turn off the 3D view
  3. Close other programs running alongside (browser automation, video playback, model inference)

The server caches previews for 200 ms, so repeated requests for the same stream are not re-encoded; the page also drops previews to about 1 fps while recording. But opening several tabs still adds up, and only closing tabs fixes that.

While recording, the page drops camera previews to about 1 fps, giving the CPU back to the capture thread.

References#