Data QC
Companion page: the "Data QC" tab in the console Checker: webapp/dataqc.py (version 2.0, read-only, called in-process)
1. Why QC Matters#
Chapter 8 covered how the data gets recorded. But recording it ≠ being able to use it.
A given recording may contain a few frames whose timestamps go backwards, one camera stream that was never written for the whole episode, a video with 3 fewer frames than the parquet, or an arm that did not move at all. None of this is visible during recording (the page only reports "saved 600 frames"), but at training time it pulls the model off course:
| Problem in the data | What happens during training |
|---|---|
| Out-of-order timestamps / gaps | The temporal order of training samples is scrambled; the model learns "reverse-play" or "frame-skipping" actions |
| One camera stream missing | That modality is null in those batches; the model learns to ignore it, or fails to learn that modality at all |
| Video half a beat off from the joints | Images and actions do not line up; the model learns to "look left, grab right" |
| Joint angles constant for a whole episode | A "the arm did not move" demonstration is taken as a valid action; the model learns to "stand still" |
| Joint-angle spikes / limit violations | The model treats sensor glitches as actions and jitters or hits the limits at runtime |
And none of these problems raises an error — training still runs to completion, it just turns out slightly worse. By the time you notice "why won't the model learn", you have wasted several hours. So QC right after recording; don't wait for training.
In one sentence: QC turns data that "looks fine" into data that "can be shown to be fine."
2. What the Checker Checks#
This project produces only three kinds of data that need checking:
- RGB video frames: frame count, resolution, fps, duplicate frames, and all-black / textureless frames for each camera's mp4;
- Joint-angle columns: the six joint columns of
observation.state/action; - Related timestamps:
hardware_ns,observation.state_ns,observation.images.*_ns,timestamp.
The checker does not look for fixed columns this project does not have, such as glove / IMU / SLAM. It first discovers which columns actually exist in the data, then decides how to check each one from its type and name: observation.state / action are checked as joint angles, *_ns as timestamps, and mp4 files under videos/ as video frames. If a dataset has no video, only joint angles and timestamps are checked, and no error is raised for a missing glove column.
2.1 Check Overview#
| ID | Check | What it looks at | Criterion / threshold |
|---|---|---|---|
| S1 | Dataset structure | Whether meta/info.json, meta/episodes, data/*.parquet agree |
Missing file / declared length mismatch → error; episode_index formula mismatch, non-contiguous frame_index → warn |
| T1 | Timestamp ordering | Whether adjacent frames of each timestamp column go backwards or repeat | Backwards → error; duplicates >0.5% → error, otherwise warn (duplicates in camera columns are left to V2) |
| T2 | Timestamp interval | Whether the inter-frame interval suddenly grows | Gap > max(2× nominal, nominal+15 ms): ≤9 frames lost → warn; ≥10 frames lost or a single gap ≥250 ms → error |
| T3 | Clock alignment | Offset and jitter of camera / state / timestamp relative to the primary clock | Camera jitter IQR >15 ms or p95 deviation >25 ms → warn; an IQR >40 ms or a p95 >60 ms → error; stable offset >50 ms → warn |
| J1 | Joint-angle integrity | Column count, NaN, limits, and state-vs-action consistency in observation.state / action |
Dimension mismatch / NaN / Inf → error; over limit ≤10° → info, >10° → warn, >20° → error; action-vs-state median difference >5° → warn |
| J2 | Joint motion quality | Whole-episode stillness, single-frame spikes, single-frame jumps | Completely constant → error; still ≥80% of the time → warn; spike → warn; single-frame jump >20° → warn, >45° → error |
| V1 | RGB video frames | Frame count, resolution, fps, duration, and decodability of each camera's mp4 | Missing file / unreadable / frame count ≠ parquet rows → error; resolution or fps mismatch → warn |
| V2 | Image quality | Duplicate frames, all-black, textureless | Duplicate frames ≥2% → warn, ≥10% → error; sampled frames all black → error; dim / textureless → warn |
Every finding carries a severity:
| Level | Meaning |
|---|---|
error |
A hard defect; this episode should not be used |
warn |
Suspicious; a human needs to take a look |
info |
Bookkeeping only (e.g. "there is a stable offset", "joints unchanged throughout"); not a defect |
2.2 Joint-Angle Columns#
This part is the focus of the revision. observation.state and action are both 6-D fixed-length lists, interpreted in the recorder's order:
0 shoulder_pan 1 shoulder_lift 2 elbow_flex
3 wrist_flex 4 wrist_roll 5 gripper (0–100 %)
For each column, compute: min, max, mean, standard deviation, span, frames over limit, and whether it is unchanged throughout. Then check across the whole episode:
- NaN / Inf: any frame containing one is an error;
- Dimensions: an actual column count that disagrees with what
meta/info.jsondeclares is an error; - Limits: compared against the URDF limits. On-site calibrated travel is a few degrees wider than the URDF, so ≤10° is only info, 10–20° is warn, and only beyond 20° is error;
- Whole-episode stillness: every joint completely unchanged for the entire episode → error (this demonstration is empty);
- Single-frame spike: one frame suddenly deviates from both neighbors while the neighbors agree with each other → warn (encoder / bus glitch);
- Single-frame jump: some joint changes >20° between adjacent frames → warn, >45° → error;
- state vs action: the current recorder writes a single joint trajectory, so a median difference >5° means the two are not the same thing — warn; small differences are only info.
When the joint-angle values are very small (|q|≤3.5, span≤6.3), the checker cannot tell whether the unit is degrees or radians, so it skips the limit check and records one info finding, to avoid misjudging radian data as over-limit.
2.3 Timestamps#
The primary clock is taken from hardware_ns, falling back to timestamp / wall_time. For every timestamp column:
- Backwards: the timestamp of an adjacent frame decreases → error;
- Duplicates: two adjacent frames are identical. Duplicates in a primary clock such as
hardware_ns/state_nsare recorded as T1; duplicate camera*_nsmeans "the camera produced no new image for that tick" and is left to V2's duplicate-frame counting; - Gaps: the inter-frame interval is clearly larger than the nominal period (33.3 ms at 30 fps) → convert it into a number of dropped samples;
- Offset / jitter: convert each stream to "elapsed time relative to the primary clock" and look at the median offset and the IQR. A stable fixed offset can be compensated once before training; large jitter is the real misalignment.
2.4 RGB Video Frames#
For each camera, each episode:
- Count the real frame count with
ffprobe -count_framesand compare it with the parquet row count, episode by episode. The two must be equal — this is the precondition for "fetch the k-th picture byframe_index"; - Check resolution, fps, and duration against
meta/info.json; - Read
observation.images.<cam>_nsfor two adjacent rows: identical = the camera reused the previous frame (duplicate frame); - Sample 3 frames with OpenCV and check brightness and grayscale standard deviation; all-black / almost textureless means the lens is blocked or no image is being produced.
3. Impact of Different Problems on Training#
Severe (error, the episode should be dropped)#
| Problem | Consequence |
|---|---|
| T1 timestamps going backwards | The timeline is no longer monotonic; any time-aligned training or replay is misaligned |
| T2 large gap (≥10 frames lost) | Actions are skipped inside the gap; the model learns discontinuous motion |
| J1 joint column NaN / dimension mismatch | That column is random/zero during training; the model cannot learn the trajectory |
| J1 severe joint limit violation (>20°) | Calibration or units are wrong; the data is untrustworthy |
| J2 whole-episode stillness | A "the arm did not move" demonstration is an invalid sample |
| V1 video frame count ≠ row count | Row N is paired with something other than frame N; images and actions are misaligned throughout |
| V2 duplicate frames ≥10% | The camera cannot sustain 30 fps; the temporal correspondence between images and actions degrades |
| S1 structural inconsistency | The loader may read the wrong episode, or read another recording's episode |
Moderate (warn, needs judgment)#
| Problem | Consequence |
|---|---|
| T1 a few duplicate timestamps | Two samples at the same instant; mildly redundant — a high ratio means the effective sample rate is not what it claims |
| T2 small gap (1–9 frames lost) | A few actions are skipped; usually acceptable |
| T3 camera jitter | Images and joints are more than half a beat apart; grasping tasks are the most sensitive |
| J1 limit violation 10–20° | Either the calibrated travel is wider than the URDF, or the arm really hit the limit |
| J1 action inconsistent with state | The target and the measured value are not the same trajectory; confirm before training |
| J2 single-frame spike / jump | A sensor glitch teaches the model badly and causes jitter at runtime |
| J2 still most of the time | Too few useful actions in the demonstration; poor sample efficiency |
| V2 duplicate frames 2–10% | Some dropped frames; usually acceptable |
| V1 resolution / fps mismatch | Possibly re-encoded; just confirm that no frames were cut |
Minor (info, generally harmless)#
| Problem | Consequence |
|---|---|
| Stable fixed offset (e.g. the camera is consistently 20 ms late) | As long as it is stable it can be compensated once before training |
| A single joint unchanged throughout (e.g. the gripper never moved) | This episode did not use the gripper; normal |
| Small joint limit violation (≤10°) | Normal difference between on-site calibrated travel and the URDF |
The criterion is not "are there any findings" but "can this episode still represent a normal demonstration".
4. Using the Page#
The numbers in the screenshot correspond to the steps below.
Selecting and Viewing Data#
| # | Action |
|---|---|
| ① | Select a dataset (the dropdown shows episode count and total frames); click "Refresh List" to rescan |
| ② | Clicking a row loads it for replay; the checkbox adds it to QC/delete; "Select All" and "Clear Selection" operate in bulk |
| ③ | "Play" starts/continues; "Stop" stops on the current frame, it does not jump back to the beginning; "Previous Frame" / "Next Frame" step one frame at a time (playback pauses automatically) |
| ④ | Progress bar: drag to any instant and the picture and the arm jump there together |
| ⑤ | Video on the left (multiple streams side by side), the 3D arm on the right, on the same timeline |
Replay uses the parquet's hardware_ns as the primary timeline:
- Per-frame relative time =
(hardware_ns - first frame) / 1e9 - Drag the progress bar → the video's
currentTimejumps to the same second, and the 3D view runs FK with that frame's joint angles - During playback the clock advances at 60 fps, and video drift beyond 0.12 s is corrected automatically
So "where the arm's hand is" in the 3D view and "where the hand is" in the video are the same instant, and can be compared directly.
Running QC#
| # | Action |
|---|---|
| ⑥ | "QC Selected Episodes" checks only the ticked ones; "QC Entire Dataset" checks everything |
| ⑦ | Result table: one row per episode, eight columns S1 / T1 / T2 / T3 / J1 / J2 / V1 / V2 (✗ error / △ warning / i info / · none), the last column is the verdict; the specific problems are listed below |
| ⑧ | History: every QC run is archived, in reverse chronological order; click an entry to bring that run's results back |
When you click the play button its label does not change; the status next to it shows Playing…. The button is grayed out until you press "Stop" or drag the progress bar.
History is stored in ~/.starm/qc_history/<dataset>-<time>.json and survives a page refresh or a browser restart. The list is in reverse time order (not by dataset name). The "Verdict" column reports the effective verdict — for each episode, whichever is more severe: the checker's error/warn, or the duplicate-frame threshold being exceeded.
Click "Export PDF" to produce an A4 report for the current scope. The report covers only this project's three check categories:
- Dataset overview (cameras, joint columns, frame counts);
- QC verdict + per-episode summary table;
- Joint-angle data: for the six joints of each episode, min/max/mean/std/span/limit/frames over limit/unchanged throughout;
- Timestamps: for each stream, median interval, max interval, backwards, duplicates, dropped frames, offset relative to the primary clock;
- RGB video: for each camera, resolution, fps, video frames, parquet rows, difference, duration, duplicate frames, brightness;
- Problem details + conclusion and recommendations.
Click "Delete Selected Episodes" to remove the ticked episodes from the dataset entirely — this deletes the parquet, all the videos, and the corresponding meta records at the same time, and cannot be undone; a confirmation dialog appears first.
5. A Complete QC Workflow#
1. Data QC page → select the dataset
2. Click "QC Entire Dataset" ← look at the whole picture first: which episodes have problems
3. For each warning/error episode:
- Click that row to load it for replay
- Play it through, watching the video and the 3D view for plausible motion
- Read the problem details below, decide whether it is a real defect or a false positive
4. Tick the episodes to drop → "Delete Selected Episodes"
5. Run QC again, confirm that only passes remain
6. "Export PDF" for the record (you can also re-export from History at any time)
Look at the whole picture first, then replay episode by episode — much faster than going episode by episode from the start: typically only 1–2 of 7 episodes need human judgment.
6. FAQ#
| Symptom | Cause | Fix |
|---|---|---|
| Old history entries still show C1–C6 | That entry was saved by the 1.x generic checker | Run QC again to get the new S1–V2; old entries can still be opened |
| An episode recorded with a single stream reports V1 | Other episodes in this dataset have two cameras; this one has only one | Normal; ignore it if you only care about this episode |
| J1 reports "over limit ≤10°" | On-site calibrated travel is wider than the URDF, or the arm sits slightly below the limit after power-off | Not a defect; only look at it beyond 10° |
| Joint-angle values are very small and J1 did not flag limits | Degrees and radians cannot be told apart, so the checker skipped the limit check | Confirm the unit used at recording time; this project records degrees |
info-level "stable offset" |
The camera is consistently tens of milliseconds late and stable | Not a defect; compensate once before training |
| QC finishes fast but everything is green | Only the timeline / frame counts / joint values were checked | Motion quality and blurry images are beyond it; human replay is still required |
| PDF export fails | websocket-client missing, or no Chrome/Edge |
pip install websocket-client; install Chrome |
| Episode numbers are non-contiguous after deleting | Only episodes in the middle were deleted | Normal; episode keeps its original number, and new recordings continue after "the highest number ever used" without reusing old ones |
| Page episode numbers do not match the log | Older versions displayed 0-based file numbers | The page, the log, and the QC report now all use the 1-based episode_index; log #18 is episode 18 on the QC page |
| Results gone after refresh | Older versions kept them only in memory | They are now archived automatically; see "History" |
The checker only reads the dataset and never modifies any file; only "Delete Selected Episodes" on the page actually deletes data.
Relationship to the old generic QC tool#
Early versions of the repository contained a generic timeline checker under data_qc_v1.1.3/,
but the current release bundle no longer includes that directory. The console's QC and PDF
export both use webapp/dataqc.py; do not go looking for the old path.
References#
- Chapter 8: Data Collection — how the data gets recorded
- Code:
webapp/dataqc.py(check logic),webapp/report_pdf.py(PDF printing),webapp/dataq.py(list datasets / load replay / call the checker / delete) - Tests:
webapp/tests/test_dataqc.py(synthetic datasets covering joint angles, video frames, timestamps)