SO-101
Course overview/Chapter 09 · Data QC
中文
Chapter 09Data & Training~15 min

Data QC

Companion page: the "Data QC" tab in the console Checker: webapp/dataqc.py (version 2.0, read-only, called in-process)

7 sections 1 figures 2 code blocks QCJoint anglesVideo framesTimestamps
Data QC page

1. Why QC Matters#

Chapter 8 covered how the data gets recorded. But recording it ≠ being able to use it.

A given recording may contain a few frames whose timestamps go backwards, one camera stream that was never written for the whole episode, a video with 3 fewer frames than the parquet, or an arm that did not move at all. None of this is visible during recording (the page only reports "saved 600 frames"), but at training time it pulls the model off course:

Problem in the data What happens during training
Out-of-order timestamps / gaps The temporal order of training samples is scrambled; the model learns "reverse-play" or "frame-skipping" actions
One camera stream missing That modality is null in those batches; the model learns to ignore it, or fails to learn that modality at all
Video half a beat off from the joints Images and actions do not line up; the model learns to "look left, grab right"
Joint angles constant for a whole episode A "the arm did not move" demonstration is taken as a valid action; the model learns to "stand still"
Joint-angle spikes / limit violations The model treats sensor glitches as actions and jitters or hits the limits at runtime

And none of these problems raises an error — training still runs to completion, it just turns out slightly worse. By the time you notice "why won't the model learn", you have wasted several hours. So QC right after recording; don't wait for training.

In one sentence: QC turns data that "looks fine" into data that "can be shown to be fine."

2. What the Checker Checks#

This project produces only three kinds of data that need checking:

  1. RGB video frames: frame count, resolution, fps, duplicate frames, and all-black / textureless frames for each camera's mp4;
  2. Joint-angle columns: the six joint columns of observation.state / action;
  3. Related timestamps: hardware_ns, observation.state_ns, observation.images.*_ns, timestamp.

The checker does not look for fixed columns this project does not have, such as glove / IMU / SLAM. It first discovers which columns actually exist in the data, then decides how to check each one from its type and name: observation.state / action are checked as joint angles, *_ns as timestamps, and mp4 files under videos/ as video frames. If a dataset has no video, only joint angles and timestamps are checked, and no error is raised for a missing glove column.

2.1 Check Overview#

ID Check What it looks at Criterion / threshold
S1 Dataset structure Whether meta/info.json, meta/episodes, data/*.parquet agree Missing file / declared length mismatch → error; episode_index formula mismatch, non-contiguous frame_index → warn
T1 Timestamp ordering Whether adjacent frames of each timestamp column go backwards or repeat Backwards → error; duplicates >0.5% → error, otherwise warn (duplicates in camera columns are left to V2)
T2 Timestamp interval Whether the inter-frame interval suddenly grows Gap > max(2× nominal, nominal+15 ms): ≤9 frames lost → warn; ≥10 frames lost or a single gap ≥250 ms → error
T3 Clock alignment Offset and jitter of camera / state / timestamp relative to the primary clock Camera jitter IQR >15 ms or p95 deviation >25 ms → warn; an IQR >40 ms or a p95 >60 ms → error; stable offset >50 ms → warn
J1 Joint-angle integrity Column count, NaN, limits, and state-vs-action consistency in observation.state / action Dimension mismatch / NaN / Inf → error; over limit ≤10° → info, >10° → warn, >20° → error; action-vs-state median difference >5° → warn
J2 Joint motion quality Whole-episode stillness, single-frame spikes, single-frame jumps Completely constant → error; still ≥80% of the time → warn; spike → warn; single-frame jump >20° → warn, >45° → error
V1 RGB video frames Frame count, resolution, fps, duration, and decodability of each camera's mp4 Missing file / unreadable / frame count ≠ parquet rows → error; resolution or fps mismatch → warn
V2 Image quality Duplicate frames, all-black, textureless Duplicate frames ≥2% → warn, ≥10% → error; sampled frames all black → error; dim / textureless → warn

Every finding carries a severity:

Level Meaning
error A hard defect; this episode should not be used
warn Suspicious; a human needs to take a look
info Bookkeeping only (e.g. "there is a stable offset", "joints unchanged throughout"); not a defect

2.2 Joint-Angle Columns#

This part is the focus of the revision. observation.state and action are both 6-D fixed-length lists, interpreted in the recorder's order:

text
0 shoulder_pan   1 shoulder_lift   2 elbow_flex
3 wrist_flex     4 wrist_roll      5 gripper (0–100 %)

For each column, compute: min, max, mean, standard deviation, span, frames over limit, and whether it is unchanged throughout. Then check across the whole episode:

  • NaN / Inf: any frame containing one is an error;
  • Dimensions: an actual column count that disagrees with what meta/info.json declares is an error;
  • Limits: compared against the URDF limits. On-site calibrated travel is a few degrees wider than the URDF, so ≤10° is only info, 10–20° is warn, and only beyond 20° is error;
  • Whole-episode stillness: every joint completely unchanged for the entire episode → error (this demonstration is empty);
  • Single-frame spike: one frame suddenly deviates from both neighbors while the neighbors agree with each other → warn (encoder / bus glitch);
  • Single-frame jump: some joint changes >20° between adjacent frames → warn, >45° → error;
  • state vs action: the current recorder writes a single joint trajectory, so a median difference >5° means the two are not the same thing — warn; small differences are only info.

When the joint-angle values are very small (|q|≤3.5, span≤6.3), the checker cannot tell whether the unit is degrees or radians, so it skips the limit check and records one info finding, to avoid misjudging radian data as over-limit.

2.3 Timestamps#

The primary clock is taken from hardware_ns, falling back to timestamp / wall_time. For every timestamp column:

  • Backwards: the timestamp of an adjacent frame decreases → error;
  • Duplicates: two adjacent frames are identical. Duplicates in a primary clock such as hardware_ns / state_ns are recorded as T1; duplicate camera *_ns means "the camera produced no new image for that tick" and is left to V2's duplicate-frame counting;
  • Gaps: the inter-frame interval is clearly larger than the nominal period (33.3 ms at 30 fps) → convert it into a number of dropped samples;
  • Offset / jitter: convert each stream to "elapsed time relative to the primary clock" and look at the median offset and the IQR. A stable fixed offset can be compensated once before training; large jitter is the real misalignment.

2.4 RGB Video Frames#

For each camera, each episode:

  • Count the real frame count with ffprobe -count_frames and compare it with the parquet row count, episode by episode. The two must be equal — this is the precondition for "fetch the k-th picture by frame_index";
  • Check resolution, fps, and duration against meta/info.json;
  • Read observation.images.<cam>_ns for two adjacent rows: identical = the camera reused the previous frame (duplicate frame);
  • Sample 3 frames with OpenCV and check brightness and grayscale standard deviation; all-black / almost textureless means the lens is blocked or no image is being produced.

3. Impact of Different Problems on Training#

Severe (error, the episode should be dropped)#

Problem Consequence
T1 timestamps going backwards The timeline is no longer monotonic; any time-aligned training or replay is misaligned
T2 large gap (≥10 frames lost) Actions are skipped inside the gap; the model learns discontinuous motion
J1 joint column NaN / dimension mismatch That column is random/zero during training; the model cannot learn the trajectory
J1 severe joint limit violation (>20°) Calibration or units are wrong; the data is untrustworthy
J2 whole-episode stillness A "the arm did not move" demonstration is an invalid sample
V1 video frame count ≠ row count Row N is paired with something other than frame N; images and actions are misaligned throughout
V2 duplicate frames ≥10% The camera cannot sustain 30 fps; the temporal correspondence between images and actions degrades
S1 structural inconsistency The loader may read the wrong episode, or read another recording's episode

Moderate (warn, needs judgment)#

Problem Consequence
T1 a few duplicate timestamps Two samples at the same instant; mildly redundant — a high ratio means the effective sample rate is not what it claims
T2 small gap (1–9 frames lost) A few actions are skipped; usually acceptable
T3 camera jitter Images and joints are more than half a beat apart; grasping tasks are the most sensitive
J1 limit violation 10–20° Either the calibrated travel is wider than the URDF, or the arm really hit the limit
J1 action inconsistent with state The target and the measured value are not the same trajectory; confirm before training
J2 single-frame spike / jump A sensor glitch teaches the model badly and causes jitter at runtime
J2 still most of the time Too few useful actions in the demonstration; poor sample efficiency
V2 duplicate frames 2–10% Some dropped frames; usually acceptable
V1 resolution / fps mismatch Possibly re-encoded; just confirm that no frames were cut

Minor (info, generally harmless)#

Problem Consequence
Stable fixed offset (e.g. the camera is consistently 20 ms late) As long as it is stable it can be compensated once before training
A single joint unchanged throughout (e.g. the gripper never moved) This episode did not use the gripper; normal
Small joint limit violation (≤10°) Normal difference between on-site calibrated travel and the URDF

The criterion is not "are there any findings" but "can this episode still represent a normal demonstration".

INTERACTIVE LAB

QC verdict simulator

Chapter 9 rules

Tick the findings that really exist in this episode and see what verdict the QC tool returns. The rules follow Chapter 9: hard defects are error, judgment calls are warn, and a stable fixed offset is only info.

Duplicate frame ratio (V2)0.0%
>2% warn>10% error
Ready to train

No hard defects found.

No findings yet
This is a teaching simulation based on the documented rules, not dataqc.py itself. A real verdict also depends on replay, motion plausibility, and task requirements.

4. Using the Page#

The numbers in the screenshot correspond to the steps below.

Selecting and Viewing Data#

# Action
① Select a dataset (the dropdown shows episode count and total frames); click "Refresh List" to rescan
② Clicking a row loads it for replay; the checkbox adds it to QC/delete; "Select All" and "Clear Selection" operate in bulk
③ "Play" starts/continues; "Stop" stops on the current frame, it does not jump back to the beginning; "Previous Frame" / "Next Frame" step one frame at a time (playback pauses automatically)
④ Progress bar: drag to any instant and the picture and the arm jump there together
⑤ Video on the left (multiple streams side by side), the 3D arm on the right, on the same timeline

Replay uses the parquet's hardware_ns as the primary timeline:

  • Per-frame relative time = (hardware_ns - first frame) / 1e9
  • Drag the progress bar → the video's currentTime jumps to the same second, and the 3D view runs FK with that frame's joint angles
  • During playback the clock advances at 60 fps, and video drift beyond 0.12 s is corrected automatically

So "where the arm's hand is" in the 3D view and "where the hand is" in the video are the same instant, and can be compared directly.

Running QC#

# Action
⑥ "QC Selected Episodes" checks only the ticked ones; "QC Entire Dataset" checks everything
⑦ Result table: one row per episode, eight columns S1 / T1 / T2 / T3 / J1 / J2 / V1 / V2 (✗ error / △ warning / i info / · none), the last column is the verdict; the specific problems are listed below
⑧ History: every QC run is archived, in reverse chronological order; click an entry to bring that run's results back

When you click the play button its label does not change; the status next to it shows Playing…. The button is grayed out until you press "Stop" or drag the progress bar.

History is stored in ~/.starm/qc_history/<dataset>-<time>.json and survives a page refresh or a browser restart. The list is in reverse time order (not by dataset name). The "Verdict" column reports the effective verdict — for each episode, whichever is more severe: the checker's error/warn, or the duplicate-frame threshold being exceeded.

Click "Export PDF" to produce an A4 report for the current scope. The report covers only this project's three check categories:

  1. Dataset overview (cameras, joint columns, frame counts);
  2. QC verdict + per-episode summary table;
  3. Joint-angle data: for the six joints of each episode, min/max/mean/std/span/limit/frames over limit/unchanged throughout;
  4. Timestamps: for each stream, median interval, max interval, backwards, duplicates, dropped frames, offset relative to the primary clock;
  5. RGB video: for each camera, resolution, fps, video frames, parquet rows, difference, duration, duplicate frames, brightness;
  6. Problem details + conclusion and recommendations.

Click "Delete Selected Episodes" to remove the ticked episodes from the dataset entirely — this deletes the parquet, all the videos, and the corresponding meta records at the same time, and cannot be undone; a confirmation dialog appears first.

5. A Complete QC Workflow#

text
1. Data QC page → select the dataset
2. Click "QC Entire Dataset"       ← look at the whole picture first: which episodes have problems
3. For each warning/error episode:
   - Click that row to load it for replay
   - Play it through, watching the video and the 3D view for plausible motion
   - Read the problem details below, decide whether it is a real defect or a false positive
4. Tick the episodes to drop → "Delete Selected Episodes"
5. Run QC again, confirm that only passes remain
6. "Export PDF" for the record (you can also re-export from History at any time)

Look at the whole picture first, then replay episode by episode — much faster than going episode by episode from the start: typically only 1–2 of 7 episodes need human judgment.

6. FAQ#

Symptom Cause Fix
Old history entries still show C1–C6 That entry was saved by the 1.x generic checker Run QC again to get the new S1–V2; old entries can still be opened
An episode recorded with a single stream reports V1 Other episodes in this dataset have two cameras; this one has only one Normal; ignore it if you only care about this episode
J1 reports "over limit ≤10°" On-site calibrated travel is wider than the URDF, or the arm sits slightly below the limit after power-off Not a defect; only look at it beyond 10°
Joint-angle values are very small and J1 did not flag limits Degrees and radians cannot be told apart, so the checker skipped the limit check Confirm the unit used at recording time; this project records degrees
info-level "stable offset" The camera is consistently tens of milliseconds late and stable Not a defect; compensate once before training
QC finishes fast but everything is green Only the timeline / frame counts / joint values were checked Motion quality and blurry images are beyond it; human replay is still required
PDF export fails websocket-client missing, or no Chrome/Edge pip install websocket-client; install Chrome
Episode numbers are non-contiguous after deleting Only episodes in the middle were deleted Normal; episode keeps its original number, and new recordings continue after "the highest number ever used" without reusing old ones
Page episode numbers do not match the log Older versions displayed 0-based file numbers The page, the log, and the QC report now all use the 1-based episode_index; log #18 is episode 18 on the QC page
Results gone after refresh Older versions kept them only in memory They are now archived automatically; see "History"

The checker only reads the dataset and never modifies any file; only "Delete Selected Episodes" on the page actually deletes data.

Relationship to the old generic QC tool#

Early versions of the repository contained a generic timeline checker under data_qc_v1.1.3/, but the current release bundle no longer includes that directory. The console's QC and PDF export both use webapp/dataqc.py; do not go looking for the old path.

References#

  • Chapter 8: Data Collection — how the data gets recorded
  • Code: webapp/dataqc.py (check logic), webapp/report_pdf.py (PDF printing), webapp/dataq.py (list datasets / load replay / call the checker / delete)
  • Tests: webapp/tests/test_dataqc.py (synthetic datasets covering joint angles, video frames, timestamps)