Data Collection
Companion page: the "Data Collection" tab in the console
1. Why Collect Data#
What the first seven chapters did, in essence, was teach the robot arm to "understand commands and reach a position." But teaching the arm a motion (grabbing a cup, opening a drawer) takes more than commands: a human has to perform it once and the machine has to record it once:
Human guides the arm through the motion → record "what it sees" and "what moved" together at 30 Hz → train the policy → the arm performs it on its own
In this pipeline, collection quality directly determines the training result. An unstable frame rate, mismatched timestamps, cameras and joints half a frame apart — and the trained policy ends up "looking left while grabbing right."
So data collection has to answer three questions:
| Question | Answer on this page |
|---|---|
| What to record | Six joint angles + camera frames |
| How fast | 30 Hz, fixed period |
| How to align | Three timestamps per frame; every difference is written down |
While collecting, the robot base and the third-person camera must stay fixed. The base defines the robot frame and workspace; the third-person camera defines what the model sees. Move either one and the pixels, table coordinates and safety boundaries for the same joint angles all change. A dataset collected across different positions teaches the model contradictory state → action pairs, and both training and inference degrade badly. Fix the base and camera before recording; if you moved them mid-dataset, recalibrate and record a new dataset instead of appending to the old one.
2. What to Collect#
Joint angles (6-D)#
shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper
Why record joint angles instead of the TCP pose?
| Joint angles | TCP pose | |
|---|---|---|
| Data volume | 6 numbers | 5 numbers |
| Unique | Yes — one set of joint angles uniquely determines one pose | No — the same position may admit several joint-angle solutions |
| Convertible | joint angles → FK → pose, computable at any time | pose → IK → joint angles, possibly multiple solutions or none |
| Information loss | None | the actual elbow/wrist configuration is lost |
So joint angles are the raw quantity and the pose is a derived quantity. Record the raw quantity; when a pose is needed, compute it on the spot with the FK from Chapter 4.
This is also why this page does not collect TCP poses: they can be computed from joint angles, and storing a second copy only causes trouble when the two columns disagree.
Camera frames (multiple selectable)#
A real robot data-collection setup usually uses two cameras:
- Wrist camera (
wrist): mounted on the wrist or beside the gripper. It moves with the arm and shows the gripper, the object and the contact point up close. - Third-person camera (
top/ external): fixed to the side of the table, a bracket, or above the work area. It shows the whole arm, the table and the task region. The console calls this roletop, but it does not have to be mounted directly overhead.
One episode can bind multiple cameras. For production data, tick both views by default. The three choices are:
| Selection | Files written |
|---|---|
| Top only | top_rgb/episode-000.mp4 → observation.images.top_rgb_ns |
| Wrist only | wrist_rgb/episode-000.mp4 → observation.images.wrist_rgb_ns |
| Both | two mp4 files → two *_ns columns |
Trade-offs:
| Setup | Pros | Cons |
|---|---|---|
| Two cameras (recommended) Production collection |
The third-person view gives task context; the wrist view gives gripper/contact detail; when one is occluded the other still sees | USB bandwidth, CPU and disk all double; both timestamp alignment and duplicate frames need attention |
| Third-person only Quick validation, distant tasks |
Low resource use, shows the whole arm and table; simple to debug | The gripper/contact point is small and easily occluded; precise manipulation is hard |
| Wrist only Contact close-ups, supplementary validation |
Close view of the gripper and object, no body occlusion | No global position or task context; fast arm motion can blur the image |
One camera is not the recommended default; it is a fallback when resources are tight. Two 720p@30 streams do compete for USB bandwidth and CPU, but with MJPG, two driver buffers, one capture thread per stream and the 3D view turned off, the measured result is a stable 30.1 Hz. Prefer two cameras for production data; fall back to one only when bandwidth/CPU is genuinely insufficient, or when you are just validating the pipeline.
In the console, the top view is ticked by default and the wrist view is not; for production collection, tick the wrist too. "Recording views" records exactly the streams you tick; the file layout is in §4.
Timestamps (three per frame)#
| Field | Meaning |
|---|---|
hardware_ns |
monotonic_ns of this sampling period; the primary timeline |
observation.state_ns |
instant the joint angles were read (same period as the row above, identical value) |
observation.images.top_rgb_ns |
instant the camera frame was actually captured |
The difference between the last two and the primary timeline is the frame's latency. A monotonic clock is used instead of wall time because monotonic is never adjusted by time synchronization (NTP), so cross-frame differences are always reliable.
3. How 30 fps Is Guaranteed#
The recording thread runs on a fixed period:
Every 1/30 s (33.3 ms):
① take a monotonic timestamp → hardware_ns
② read the six joint angles → observation.state
③ take the latest camera frame + its capture time → observation.images.<key>_ns
④ write to the in-memory buffer, and feed the frame to the encoder
Three details decide whether the frame rate holds at 30:
A. The camera must be explicitly set to MJPG + 30 fps. On USB 2.0 bandwidth, 1280×720 only reaches 30 fps with MJPG encoding. Left unspecified, many cameras negotiate 10 fps by default — the recorder shows up every period, but the camera only produces a frame every 100 ms, hence a flood of duplicate frames. Measured:
| Before | After | |
|---|---|---|
| Actual camera frame rate | 10.3 fps | 30.2 fps |
| Duplicate frames out of 108 | 72 | 2 |
B. Turn off the camera's "dynamic frame rate".
Many UVC cameras (especially these small USB ones) have exposure_dynamic_framerate enabled by default: when the image darkens, the camera lengthens the exposure to compensate, at the cost of dropping to the low 20s fps. The wrist camera shows this most clearly — 30 fps pointing at the ceiling, low 20s the moment it turns toward the table, bouncing back and forth.
The camera is opened with:
v4l2-ctl -d /dev/videoX -c exposure_dynamic_framerate=0
With it off, the driver can only compensate brightness with gain within 1/30 s, and the frame rate stays at 30. You can check it yourself: v4l2-ctl -d /dev/video0 --list-ctrls | grep dynamic — the value should be 0.
C. The driver buffer must not be set to 1. With a single V4L2 buffer the driver has nowhere to stage the next frame, so any slightly slow read drops it. Measured with two streams:
CAP_PROP_BUFFERSIZE |
Delivery rate per stream |
|---|---|
| 1 | 21.0 / 20.3 fps |
| 2 | 30.3 / 30.3 fps |
| 4 | 30.3 / 30.3 fps |
Two buffers are enough to run at full rate; latency grows by only one frame (about 33 ms).
D. Each camera gets its own capture channel. Top and wrist each have their own frame-grabbing thread and their own frame slot, so neither blocks the other. Each stream carries a frame sequence number and a delivery rate, shown directly on the page:
top /dev/video2 1280×720 · 30.3 fps
wrist /dev/video0 1280×720 · 30.3 fps
During recording the "Live" card also lists the delivered frame count and latency per stream. Measured with both streams recording:
| Single stream | Both (after fixing) | |
|---|---|---|
| Actual frame rate | 30.0 Hz | 30.1 Hz |
| Duplicate frames | 2/119 | 2/181 |
| Max latency | 10.7 ms | 27.9 ms |
E. Sampling must wait for a new frame, not take whatever the tick finds.
Camera frame intervals jitter. One wrist camera measured: 32.5, 31.5, **36.0**, 32.1, 32.0, **36.0** … — 30 fps on average, but every few ticks there is a 36 ms gap. Ticking at a fixed 33.3 ms lands on the gap and picks up a stale frame, which is then recorded as a duplicate.
When the code finds the frame timestamp unchanged on a tick, it waits up to 20 ms for a new frame and only reuses the old one if none arrives. Measured:
| Wait window | Duplicate frames in a 20 s recording (600 frames) |
|---|---|
| none | 40 ~ 90 |
| 12 ms | 2 |
| 20 ms | 0 |
The window cannot grow indefinitely: waiting too long makes sampling fall behind the tick. 20 ms is the measured value that suppresses duplicates without missing ticks.
F. If the loop falls behind, reset the tick — do not catch up by inserting frames. If a period overruns (a camera stall, for example), the next tick re-aligns to the current instant instead of firing several frames back to back. Catch-up frames create fake data that "looks like 30 fps" while the timestamps bunch up.
G. Duplicate frames are counted separately; timestamps are not rewritten.
When a camera lags by one frame, that sample reuses the previous image, but the timestamp still belongs to this period — downstream code can tell from drop_stats that a frame here was reused.
H. Turn off the 3D view while recording. The pose view on the right of the page renders continuously; measured, it can push the browser to 8 cores and squeeze two camera streams down to 21 fps. Click "3D on / 3D off" in the title bar to turn it off; this releases the WebGL context and the render loop. The capture side is unaffected and both streams stay at a stable 30 fps.
4. File Structure#
Each recording writes one episode, appended to the dataset directory:
recordings/<dataset name>/ ← relative path under the workspace; created if missing
├── meta/
│ ├── info.json ← summary: fps, features, cameras, path templates
│ ├── stats.json ← per-feature statistics (mean/variance/min/max)
│ ├── tasks.jsonl ← task names (task_index → task)
│ └── episodes/chunk-000/episode-000.parquet ← metadata for this episode
├── data/
│ └── chunk-000/episode-000.parquet ← per-frame data (one row per frame)
└── videos/
└── chunk-000/
├── top_rgb/episode-000.mp4 ← top view (one video frame per parquet row)
└── wrist_rgb/episode-000.mp4 ← wrist view (written only if selected)
chunk-000 is a shard: a new chunk every 1000 episodes, so a single directory never holds too many files.
meta/info.json#
{
"format": "pooled_episodes_v1",
"codebase_version": "v3.0",
"fps": 30.0,
"task_name": "UMI_9_29",
"cameras": {"top_rgb": {"height": 720, "width": 1280}},
"data_path": "data/chunk-{c:03d}/episode-{f:03d}.parquet",
"video_path": "videos/chunk-{c:03d}/{image_key}/episode-{f:03d}.{ext}",
"features": {
"observation.state": {"dtype": "float32", "shape": [6]},
"observation.state_ns": {"dtype": "int64", "shape": [1]},
"observation.images.top_rgb_ns":{"dtype": "int64", "shape": [1]},
"action": {"dtype": "float32", "shape": [6]}
}
}
data/chunk-000/episode-000.parquet#
Each row = one frame, 30 rows per second.
| Column | Type | Description |
|---|---|---|
episode_index |
int64 | which episode |
frame_index |
int64 | frame number within the episode, starting at 0 |
timestamp |
double | seconds since the start of the episode |
task_index |
int64 | task ID |
wall_time |
double | absolute time (Unix seconds, for cross-device alignment) |
hardware_ns |
int64 | primary timeline (monotonic) |
observation.state |
float32[6] | the six joint angles |
observation.state_ns |
int64 | instant the joint angles were read |
observation.images.top_rgb_ns |
int64 | capture instant of the top camera frame |
observation.images.wrist_rgb_ns |
int64 | capture instant of the wrist camera frame (column present only if recorded) |
action |
float32[6] | measured joint angles written by the current recorder (the same data as observation.state) |
The current recorder writes the same values into
actionandobservation.state. If training does not apply a next-state/shift, ACT learnsstate → state; confirm the training-side sequencing before using this data. This page describes the current recorder as-is and does not change recording behavior.
meta/episodes/chunk-000/episode-000.parquet#
One row per episode, describing the recording itself:
| Column | Description |
|---|---|
episode_index / task_index |
IDs |
start_frame_index / end_frame_index / length |
position and length within the global frame sequence |
duration_sec |
duration |
drop_stats |
duplicate frames, dropped frames, max camera latency |
video_codec |
encoder and parameters |
Videos#
videos/chunk-000/<image_key>/episode-000.mp4 — H.264, 1280×720, 30/1 fps, with a frame count strictly equal to the parquet row count.
Duplicate frames do not compromise "720p30". The video is always exactly 30 frames per second with 33.3 ms row spacing; when a camera delivers no new image, the same picture is simply written twice and the timeline does not drift. Measured on a 581-frame recording: median row spacing 33.3 ms, maximum 33.5 ms.
There is one subdirectory per selected view, one mp4 per subdirectory, and each frame count equals the parquet row count. On read, row N corresponds to frame N; no extra index is needed.
5. Using the Page#
The numbers in the screenshot correspond to the steps below.
Prerequisite: connect both arms on the Teleoperation page first. Data collection is teleoperation with recording: a person moves the leader, the follower follows, and this page records the result. Connecting only the follower produces no demonstration motion. On the Teleoperation page, fill in both the leader and follower serial ports and confirm the Leader/Follower Comparison has leader readings and the follower is holding torque. If you use absolute mapping, align the arms and start teleoperation before returning here. "Start recording" on this page only records; it does not start teleoperation for you.
Before starting, confirm once more that the robot base and third-person camera have not moved (see the warning at the end of §1). If they did, restore/recalibrate first; do not keep recording at the new position.
Preparation
| # | Action |
|---|---|
| ④ | In "Camera view", click "Scan devices" to list the candidates, choose one device for top (third-person) and one for wrist (you decide which is which), then click "Connect" on each |
| ⑤ | Check both previews to confirm they are not swapped: the one showing the gripper fingers is the wrist. Each label includes the actual frame rate; 30 fps is normal |
| ① | Enter a dataset name (e.g. UMI_9_29); the last input is remembered |
| ② | Under "Recording views", tick the streams to record; for production collection, tick both top and wrist (see §2 Camera frames) |
| ⑧ | Turning the 3D view off first is recommended (see the note below) |
Recording
| # | Action |
|---|---|
| ③ | Click "Start recording"; the card below shows Recording · episode #N · frames |
| ⑥ | While recording, watch "Live": actual rate, duplicate frames, latency |
| ③ | Click "Stop recording" when done; shows Saved episode #N · N frames / x.xs · duplicate frames n · max latency x ms |
| ③ | If you do not want this take: click "Abort"; the data is discarded, nothing is written to disk, no episode number is consumed, and the next recording reuses the same number |
| ⑦ | Under "Existing datasets", confirm the episode count went up by 1 and the size grew |
Episode numbers on the page and in the logs are 1-based, starting at
#1; file names on disk still follow the LeRobot v3 0-based convention (episode-000.mp4corresponds to episode #1 in the log). The number is governed by thelast_episode_indexhigh-water mark inmeta/info.jsonand is strictly increasing: deleting the last few episodes on the QC page and recording again only continues forward, never reusing an old number.
Automatic pre-flight check
When you click "Start recording", the server first measures for 1 second: if any camera is below 27.5 fps it refuses to start and names the offending stream:
wrist camera is at 24.8 fps (requires ≥27.5). Check the cable/USB port,
or close other CPU-heavy programs and retry
This way you do not waste a take on data full of duplicate frames.
Checks
| Metric | Normal range | What an abnormal value means |
|---|---|---|
| Actual rate | 29–30 Hz | Clearly below 30 → the camera is not running at 30 fps |
| Duplicate frames | < 2% | 2–10% triggers a QC warning; a high ratio means the camera frame rate is too low or the buffer is wrong |
| Dropped frames | 0 | Non-zero → the camera disconnected |
| Max camera latency | < 50 ms | Too large → USB bandwidth or CPU contention |
| Per-stream stalls | 0 | Non-zero means that stream dropped out (a cable, most often) |
| Measured camera fps | 30.0 | This is the camera's own output rate (not the sampling rate); below 29 means that camera cannot keep up |
| Max camera interval | ≈33 ms | 60 ms or more means the camera occasionally stalls |
Turn off the 3D view before recording (⑧)
The 3D scene in the pose view renders continuously; measured, it can push Chrome to 8 cores and squeeze camera capture down to 21 fps. Click "3D on / 3D off" in the card's title bar to turn it off; this releases the WebGL context and the render loop, and the placeholder shows "3D view off (no CPU/GPU use)".
With it off, both camera streams stay at a stable 30 fps. The switch state is remembered and persists the next time the page is opened.
6. FAQ#
| Symptom | Cause | Fix |
|---|---|---|
| Actual rate only 10 Hz | The camera did not negotiate 30 fps | Reconnect (the system opens cameras with MJPG + 30 fps) |
| Many duplicate frames | Camera frame rate below the sampling rate | Same as above; or use 480×270 / 640×360 |
| Only one stream has many duplicates | That stream lost CPU / loose cable | See "Diagnosing a single stream's dropped frames" below |
| The two views are swapped | Wrong device chosen for top/wrist | Judge from the picture, change the device path, reconnect |
| Error immediately on start | Camera not connected / arm not connected | Connect first |
| Episode numbering starts at 1 | New dataset | Normal; numbering is strictly increasing within a dataset, the log shows #N, and the QC page shows the same N |
| Something is wrong mid-recording | This take is unusable | Click "Abort": the data is discarded and no number is consumed; the next recording still uses this number |
| Video and parquet row counts differ | The encoder dropped frames | This implementation writes every frame, so it cannot happen; if it does, the disk is full |
Diagnosing a single stream's dropped frames#
Duplicate frames are counted per stream; after recording, look at drop_stats in meta/episodes/*.parquet to see which camera:
{"top_rgb_duplicate": 1, "wrist_rgb_duplicate": 61,
"top_rgb_lag_max_ms": 11.8, "wrist_rgb_lag_max_ms": 49.0}
Two diagnostics:
| What to look at | What it means |
|---|---|
| Only one stream has a high duplicate count | The problem is in that stream (the camera itself / its cable / its thread being starved) |
That stream's lag_max_ms is clearly larger |
That stream stalls periodically; it is not merely a low frame rate |
Measured comparison: recording 20 seconds with the arm stationary gave 600 frames per stream with only 1 duplicate each — so the capture pipeline itself is fine; drops that appear only once the arm moves point first to the wrist camera cable being pulled along with the arm or the host CPU being saturated at the time.
Troubleshooting order:
- Watch the live frame rates on the two page labels (
top … 30.0 fps). With the arm stationary, whichever stream stays below 29 is almost certainly that camera or its cable. - Turn off the 3D view and record again (see §3 F).
- Do not run other preview/recording tools at the same time.
- Change USB ports: put the two cameras on different controllers if possible (
lsusb -tshows Bus numbers). - Wrist camera: use a softer, shorter cable, add strain relief, and do not bundle it with the motor harness.
- If it still fails, drop that stream to 640×480 and retry — decoding and data volume are both halved.
Both streams dropping at once means the problem is not the cameras but a saturated host. Clear it in this order:
- Close other tabs (every tab with the "Data Collection" page open pulls two previews)
- Turn off the 3D view
- Close other programs running alongside (browser automation, video playback, model inference)
The server caches previews for 200 ms, so repeated requests for the same stream are not re-encoded; the page also drops previews to about 1 fps while recording. But opening several tabs still adds up, and only closing tabs fixes that.
While recording, the page drops camera previews to about 1 fps, giving the CPU back to the capture thread.
References#
- Chapter 4: FK and IK — how joint angles become poses
- Chapter 7: Camera Calibration — how cameras are connected and aligned
- Code:
webapp/recorder.py(capture and writing to disk),/api/collect/*inwebapp/server.py