Model Training and Inference
Companion pages: the "Model Training", "Dataset Inference" and "Real-Robot Inference" tabs Tech stack: LeRobot 0.6.1 + ACT policy
1. Where This Chapter Fits in the Pipeline#
Ch. 8 collection → recordings/<dataset>/ raw data
Ch. 9 QC → drop bad episodes
↓
[I] build dataset → datasets/<name>/ LeRobot v3 format; the only format training reads
[II] train ACT → outputs/train/<run>/ checkpoint (.safetensors)
[III] dataset inference → robot stays still; check whether it learned the right thing
[IV] real-robot inference → the model drives the arm directly
In one sentence: training never reads recordings, only datasets. One "build" step in between is mandatory.
2. Model Inputs and Outputs#
Understand this section first; every parameter below follows from it. ACT is an imitation-learning policy: given the current image and joint angles, it predicts joint targets for a short window into the future.
Inputs (one set per inference)#
| Input | Shape | Source | Processing |
|---|---|---|---|
| Camera image | (3, H, W) per stream |
dataset video frame / live camera | resize to the resolution chosen at build time (640×360 by default), /255 into [0,1], then MEAN_STD normalization using dataset statistics |
| Joint-angle state | (6,) |
that frame's observation.state |
raw values (degrees), normalized with the same statistics |
| Language instruction | — | — | ACT does not use language; the "task text" field on the page is a placeholder and does not affect training |
The normalization mean/std come from training-set statistics (the
stats.jsoncomputed when the dataset was built). That is why "new data means rebuilding": feed new data with old statistics and the model sees a different distribution.
Output#
Predict one chunk: shape (chunk_size, 6)
each row = the six joint targets (degrees) at one instant in the future
With the default chunk_size = 60, that is the next 2 seconds of actions at 30 fps, returned in a single inference.
The model does not infer once per frame; it infers 2 seconds at a time and executes the first N frames of that chunk — that is what n_action_steps means.
Timing from one inference to execution#
t=0 read image + joint angles → model → chunk[0..59]
execute chunk[0], send to the follower arm
t=1/30 execute chunk[1]
...
t=59/30 execute chunk[59], queue empty
→ read the image again, infer again
The "Queue Remaining" field on the real-robot inference page is exactly how many frames are left in this queue.
3. Building the Dataset#
Why a build step is needed#
Recordings are stored as pooled_episodes_v1 (one parquet per episode + one mp4 per stream), whereas LeRobotDataset needs a v3 dataset with statistics, with feature descriptions and at a uniform resolution. Three gaps:
| Difference | How the build handles it |
|---|---|
| The two cameras may have different sizes | resize everything to the chosen resolution (640×360 by default) |
| No normalization statistics | recompute min/max/mean/std/q01…q99 and write them to stats.json |
| Episodes often start with static frames | tick "Trim leading static frames" |
After building, the dataset is actually loaded once with LeRobotDataset as a check; if it cannot be loaded, the build errors out instead of leaving a half-written dataset.
Page fields#
| Field | Description |
|---|---|
| Data source | a recording under recordings/, or any directory containing meta/info.json |
| Camera streams | which streams go into the dataset. However many streams you train on, you must open that many at inference time |
| Episodes | each episode's file index, frame count, duration and available cameras; episodes that are corrupt or missing a camera are unticked by default |
| Dataset name | written to datasets/<name>; letters, digits, underscore, dot and hyphen only |
| Resolution | options are 640×360 (aspect-preserving, recommended), 480×270 (smaller), 1280×720 (slow, not recommended). Height follows the source aspect ratio: 640 wide is 640×360 for 16:9, 640×480 only for 4:3; for small objects prefer 1280×720 |
| Trim leading static frames | cuts the first second or two after recording starts, where nothing moves |
4. Training ACT#
4.1 Page parameters (what you can change)#
| Parameter | Default | Meaning | How to tune |
|---|---|---|---|
| preset | standard |
sets the total step count: quick=3000, standard=derived from 30 epochs, full=100000 |
use quick for a first end-to-end run; standard for real training; consider full only with a lot of data (>200 episodes) |
| batch size | 8 | how many samples go in per step. Larger means steadier gradients and more GPU memory | drop to 4 or 2 if GPU memory overflows; 16/32 if memory is plentiful, though it is usually not noticeably better |
| chunk size | 60 | how many future frames are predicted at once; 60 = 2 seconds at 30 fps | slow, predictable motions (pick and place) → 30–60; fast, jerky motions → 20–30. Smaller is more "responsive" but infers more often |
| n_action_steps | = chunk | how many frames are actually executed after one inference before inferring again | set it to half the chunk (e.g. 30) so the model "re-observes" more often, at the cost of doubling inference compute |
| steps | auto | total training steps | fill it in to override; the automatic value is 30 × total frames / batch, clamped to 3000–100000 |
| learning rate | 1e-5 (already filled in the box) |
learning rate, editable directly in the main parameter area | ACT is not sensitive to lr, do not tweak it casually; the range is clamped to 1e-7 ~ 1e-2, but in practice only try 1e-5 ~ 3e-5 |
| save freq | 10000 | how often a checkpoint is saved; only the latest 3 are kept | lower it (e.g. 500) to compare training stages, but only 3 stay on disk |
| num workers | 4 | data-loading threads | raise to 8 when video decoding is the bottleneck; drop to 2 when the CPU is tight |
| device | cuda | training device | without CUDA it warns "tens of times slower" |
| seed | 1000 | random seed | only change it to reproduce an experiment |
4.2 Internal model hyperparameters (not exposed on the page; these are the lerobot defaults)#
| Hyperparameter | Default | Description |
|---|---|---|
dim_model |
512 | Transformer hidden width |
n_heads |
8 | number of multi-head attention heads |
dim_feedforward |
3200 | feed-forward width |
n_encoder_layers / n_decoder_layers |
4 / 1 | 4 encoder layers, 1 decoder layer |
latent_dim |
32 | VAE latent dimension |
kl_weight |
10 | weight of the VAE KL term (higher = smoother and more conservative) |
use_vae |
True | use a VAE to model action multimodality (the same image can admit several correct actions) |
vision_backbone |
resnet18 | image backbone, fine-tuned together with the policy during training |
optimizer_lr |
1e-5 | this is the lr on the page |
The page exposes only
chunk_size / n_action_steps / lr; everything else uses defaults. This set is fairly stable in the ACT paper and in common practice — beginners should not touch it.
4.3 Training log and loss curve#
While training: the chart at the top of the training page updates every 50 steps. The x-axis is the
step, the y-axis is log10(loss) (log scale). The metric cards next to it show
loss / l1 / kld / lr / memory / speed.
After training: the page stops refreshing every second, but the curve and numbers are not lost.
Click a run name in the Training History table, or its Loss Curve button, and the chart above
switches to that run; Back to Live Training returns to the active job. The data comes from
outputs/train/<run>/metrics.jsonl, so it survives a Web-console restart.
| Metric | Meaning | How to use it |
|---|---|---|
loss |
total loss = l1_loss + kl_weight × kld_loss |
watch the overall trend; early KL can dominate, so do not judge by it alone |
l1_loss |
mean absolute action-reconstruction error (after normalization) | the most direct measure of action accuracy; look at this first |
kld_loss |
KL divergence of the VAE | rising then flattening is normal; stuck at 0 long-term means the VAE is doing nothing |
lr / mem / speed |
learning rate, GPU memory, speed | auxiliary health checks, not quality metrics |
What l1_loss means and how it is computed
Before training, each action dimension is normalized with the dataset statistics:
â = (a − mean) / std
l1_loss is the mean absolute error in that normalized action space:
l1_loss = (1 / N_valid) × Σ |â_pred − â_gt| × mask
- the sum covers the batch, every frame in the chunk, and the six action dimensions;
maskremoves padded frames (action_is_pad);N_validis valid frames × 6;- it is therefore dimensionless, not degrees.
l1 = 0.04does not mean 0.04°.
To roughly convert to a joint's angular error, multiply by that joint's std:
typical |error| for that joint ≈ l1_j × std_j
std is in datasets/<name>/meta/stats.json under action.std. For example, a joint with
std = 20° and l1 ≈ 0.04 corresponds to about 0.8°; with std = 60° the same l1 is about 2.4°.
The logged l1 is an average over all six joints, so this is only a rough scale; for per-joint degree
errors, use the MAE / RMSE / max error from dataset inference in §5.
Reading the curve to judge training
- Watch the l1 trend first, not only loss. A normal curve drops quickly in the first few hundred
steps, then becomes a long slow tail. Total loss can be inflated by the KL term early;
l1reflects action error. - If l1 stays flat for thousands of steps, suspect insufficient data, misaligned cameras, poor action coverage or an unstable lr; adding more steps usually will not help.
- If loss falls but l1 does not, check the KL term; a
kld_lossstuck at 0 or unusually large is worth investigating. - If training l1 is low but held-out test inference is much worse, the model is overfitting or the test distribution differs; confirm with §5 and add data/diversity rather than more steps.
- Do not compare l1 across different datasets directly: their
stdvalues differ, so the same0.04means a different angular error. Compare runs on the same dataset and normalization stats.
Here is a real review screenshot after a completed 100,000-step run:
Example: SO101_v1_2cam, 50 episodes / 34,258 frames, 100k steps; final loss = 0.0380,
l1 = 0.0380, kld = 0.0000, 4h49m.
What the screenshot shows:
- The curve falls from 15.19 to 3.8e-2; most of the drop happens in the first few thousand steps, then it becomes a long tail. The y-axis is logarithmic, so a nearly straight line means exponential decay.
- l1 and loss nearly coincide (0.0380), because
kld = 0; the total loss is essentially the action reconstruction error for this run. - The small jitter at the tail is mini-batch noise, not a broken run; judge the plateau and whether l1 is still falling.
l1 = 0.038is a normalized value, not 0.038°. With an actionstdaround 20–60°, that is roughly a 1–2° scale; use dataset-inference MAE for the final degree-level number.- The run looks converged, but convergence is not generalization: still run a held-out test set (next section) to check that it did not merely memorize the training data.
In one sentence: the loss curve shows the trend, l1 shows action error; l1 is dimensionless — multiply by the joint's std for a rough degree estimate, or read the MAE from dataset inference.
4.4 Sample size sets the ceiling#
The page gives a verdict from the sample size:
Sample size is low (2 episodes / 760 frames); in practice start at 50 episodes,
so this dataset is only good for validating the pipeline
| Episodes | What it can do |
|---|---|
| < 10 | only proves the pipeline runs end to end |
| 20–50 | simple straight-line tasks may barely succeed |
| 50–200 | usable for ordinary grasping tasks |
| > 200 | robustness only starts to be meaningful here |
With too few samples, parameter tuning is basically pointless — this is the most common way to waste time.
4.5 Checkpoints, Pause, Resume, Abandon, Queue#
Checkpoint rotation: by default a checkpoint is saved every 10000 steps to
outputs/train/<run>/checkpoints/<step>/; only the latest 3 are kept on disk (about 2 GB, roughly 600–700 MB each),
older ones are deleted automatically; the training history table also has a "Clean Up Old Ckpts" button
for manual pruning. checkpoints/last always points to the newest one. Pausing forces an extra save,
so the step you paused at is not necessarily a multiple of 10000.
Inferring from a checkpoint while training continues: no need to wait for training to finish. After
each save completes, it becomes selectable in the "Model checkpoint" dropdown on the dataset-inference /
real-robot inference pages; click "Refresh Model List" to refresh immediately.
Half-written checkpoints never appear in the list (the training_state/training_step.json file marks
completion), so you cannot pick partially written weights.
Pause and save: click "Pause and Save" mid-training; the process saves a checkpoint after the current step finishes and then exits. The page shows "Paused"; at that point:
- "Resume Training": continues from the latest checkpoint, still toward the original total step count;
- "Abandon This Run": ends the process immediately and saves no further checkpoint; checkpoints already written are kept and can still be used for inference.
Note that a pause only takes effect after a complete checkpoint write, so there is a delay of seconds to tens of seconds between the click and the actual stop; it cannot pause while "Starting" (still loading data/weights).
Queue: after "Add to Queue" the run starts immediately if the trainer is idle; if a run is already going it waits, and starts automatically once the current run completes / fails / is abandoned. A few conventions:
- "Pause" is a brake the user applies deliberately: a paused run blocks the queue until "Resume Training" or "Abandon This Run";
- each queue entry records dataset, steps, batch, checkpoint interval and status, persisted in
outputs/train/queue.json; - the queue survives a Web-service restart but does not start on its own (so it cannot fight a leftover training process over the GPU).
5. Dataset Inference (robot stays still)#
Example in the figure: latest checkpoint
SO101_v1_2cam @ 100000, datasetSO101_v1_2cam, episode 009 (658 frames), chunk step 60; 11 chunks, mean |error| 0.570° (first frame of each chunk).
What it answers#
"How far are the actions the model predicts from the demonstrated actions?" Purely offline; the robot arm does not need to be connected.
Flow: take one frame's image + joint angles → model → one chunk (60 steps by default) → every step of that chunk is plotted; the ground truth is read separately from the dataset, one value for every frame of the episode. Re-observe and infer again every "chunk step" frames; each chunk start is marked with a thin dashed vertical line.
Parameters#
| Parameter | Description | How to tune |
|---|---|---|
| Model | outputs/train/*/checkpoints/*/pretrained_model |
pick different steps to compare training stages |
| Dataset | a built datasets/<name> |
use episodes the model never trained on to see generalization |
| episode | which episode | — |
| chunk step | how many frames between re-observations (1, chunk, 2×chunk…) | fill in the model's chunk (default 60); it must be ≤ the chunk the model was trained with; by default n_action_steps = chunk. This matches the real "execute a chunk, then re-observe" cadence; a smaller value only makes re-inference denser, which is not the real cadence |
| Start / end frame | restrict to a range | start with the first 100 frames; fast |
The chunk step cannot exceed the model's training chunk. If it is larger (say the model chunk is 60 but you enter 90), the model would re-observe before a chunk has been executed, which never happens on the robot; the page rejects it. A smaller value only makes re-inference denser: the model is queried far more often than on the robot, so it does not represent real inference.
If training used
n_action_stepssmaller than the chunk, you may usen_action_steps; with the defaults they are equal, so just use the chunk size.
Chunk step and the model's horizon#
First, distinguish two "chunks":
- Model chunk: fixed during training (60 frames for this checkpoint). It sets how far into the future actions the model predicts; the page cannot change it.
- Chunk step: a page parameter that sets how many frames to run before re-observing and inferring again; it must be ≤ the model chunk.
For the same model and dataset, different chunk steps behave like this:
| Chunk step | Re-inference rate (30 fps) | Prediction actually used | Calls for a 635-frame episode | Use |
|---|---|---|---|---|
| 1 | every frame, 30 Hz | 1 / 60 | 635 | inspect per-frame limits; not the real cadence |
| 15 | every 0.5 s, 2 Hz | 15 / 60 | 43 | see more frequent replanning |
| 30 | every 1 s, 1 Hz | 30 / 60 | 22 | compromise |
| 60 | every 2 s, 0.5 Hz | 60 / 60 | 11 | default; matches real n_action_steps=60 |
| >60 | — | — | — | rejected (exceeds the trained chunk) |
The number of calls is approximately ceil(frames / chunk_step): a smaller step queries the model more
often, produces denser orange segments and costs more compute, but uses only part of each prediction.
Only step = model chunk (60 by default) means "execute the full chunk, then observe again".
What chunk does to the horizon — separate spatial field of view from temporal horizon:
- Spatial field of view is set by the camera lens, resolution and mounting position; chunk does not zoom, crop or otherwise change what the camera sees.
- Temporal horizon is set by the trained model chunk: chunk=60 means predicting 2 seconds of future actions at 30 fps. A larger chunk looks further ahead and gives smoother motion, but stays open-loop longer, accumulates more end-effector error and reacts more slowly. A smaller chunk replans more often and reacts faster, but costs more inference compute and can be jittery.
- The page's chunk step only discards the rest of the prediction and looks again; it does not enlarge the model's own horizon. Example: model chunk=60 and step=15 — the model still predicts 60 steps, but the loop re-observes after 15 and never executes the remaining 45.
- On the plot, the white line is the full per-frame GT and each orange segment is a full 60-step prediction. A smaller step therefore creates more overlapping orange segments and a denser chart. The error table counts only the first frame of each chunk (the moment the real robot would re-infer); do not judge the model by the tail of the curve.
Checking overfitting with a held-out test set#
A low error on the training set only shows that the model fit the training data. To decide whether this run overfits, record a separate set that was never used for training and run dataset inference on it:
- Record 5–10 new episodes for the same task, with the same camera positions, lighting and table layout. Keep these episodes out of the training dataset and do not reuse training episodes.
- Build them into a separate test dataset (for example
data_1111_test_2camin this workspace) with the same cameras, resolution and fps as training. - In dataset inference, select the trained checkpoint + the test dataset, set the chunk step to the model's chunk, and run the whole episode. Record mean |error| and per-joint MAE/RMSE.
- Run the same checkpoint on one training episode and compare the two sets of numbers and curves.
| Training set | Held-out test set | Conclusion |
|---|---|---|
| Low | Close to training | Generalizes normally; no clear overfitting |
| Very low | Clearly higher | Overfitting or a different test distribution; first confirm task/camera/lighting match, then add data/diversity or reduce training steps |
| High | High | Underfitting or too little data; add data and check training |
| Low, but a few test joints are much worse | Those joint/object motions are underrepresented | Add demonstrations for that motion; do not just add steps |
Use the test set only for the final judgment. Do not tune repeatedly against it; selecting checkpoints by test-set score indirectly fits the test set.
Reading the results#
Trajectory plots: white = the dataset's demonstration GT for every frame (the full curve, independent of
the chunk step), orange = each chunk's whole predicted trajectory, thin dashed lines = chunk starts, one panel
per joint; the x-axis is the frame number (relative to the start of the episode, 0 = the episode's first
frame), matching the video frames on the same page one-to-one.
Error table: pred − GT, with GT mean / Pred mean / MAE / RMSE / max error per joint, computed only on
each chunk's first frame (the moment the robot actually re-infers). The orange line is the whole open-loop
chunk prediction, so its error grows towards the end: in the example the chunk-start mean is 1.135° while the
whole-chunk mean is about 3.1°; that horizon error is expected, so do not judge the model by the line's tail.
RGB video + frame cursor#
Selecting an episode loads that episode's RGB video (as many streams as the model uses, all sharing one clock). The video and the plots share the same "current frame":
- Click "Play": the plots' vertical line follows whatever frame the video reaches (the video is the clock; nothing is extrapolated from a timer)
- Click "Previous Frame" / "Next Frame" to step one frame back or forward; the video and the cursor move together (playback pauses automatically)
- Drag the vertical line in any plot: the video jumps to that frame (all cameras jump together)
- Dragging the progress bar does the same
When the orange line drifts somewhere, drag the cursor over and look at the picture; no need to go back to the QC page to match timestamps. The vertical line can be dragged across the whole episode; the ground truth (white) is always drawn for every frame of the episode, while the orange line is each chunk's full prediction, with thin dashed lines marking chunk starts. The chunk step only controls how often the model is re-queried; it never changes the ground-truth curve.
| Symptom | Meaning | Action |
|---|---|---|
| Orange line hugs the white line | the main trend was learned | ready to try on the real robot |
| Orange line lags throughout | chunk too long / n_action_steps too large | lower chunk or n_action_steps |
| One joint's orange line barely moves | that joint barely varies in the training set | add data where that joint actually moves |
| Mean error > 20° | essentially not learned | add data and retrain; do not tune parameters |
| Bad only on a few episodes | those episodes are poor quality | go back to Chapter 9 QC |
| Training curves fit, held-out test clearly worse | overfitting, or a different test distribution | confirm task/camera/lighting match, then add data/diversity; do not just add steps |
Dataset inference only proves the model can fit the selected data. Low training error plus high held-out test error is the signal of overfitting. A held-out test set estimates generalization; it still does not guarantee real-robot success, which also depends on dynamics, latency and the safety layer.
6. Real-Robot Inference#
Throughout real-robot inference, do not move the robot base or the third-person camera. The base defines the robot frame, workspace and safety boundaries; the third-person camera defines what the model sees. Moving either one changes the viewpoint, the pixels for a given joint pose, and the table coordinates. The model then sees states outside its training distribution, its actions degrade sharply, and the arm can hit something. Fix the base and camera before starting. If either is bumped mid-run, stop immediately, restore the position and re-check the view/calibration; if you cannot restore it exactly, re-collect and retrain — do not just continue.
How the closed loop runs#
queue empty → read both cameras + joint angles → model → chunk (60 frames)
→ enqueue, take only the first n_action_steps frames
every 1/30 s:
pop one frame from the queue → per-joint clamp (default ±3°) → record as target
→ SDK 30 Hz control loop: read measurement → safety layer → write servos
last step of the batch sent → (inference interval s) → back to "queue empty", infer again
Prerequisites#
| Condition | If not met |
|---|---|
| robot arm connected (real hardware) | button stays gray |
| every camera the model needs is open | error "camera top is not open yet" |
| camera frames are no more than 1.5 s old | error "camera frame has expired" |
| the arm is not in e-stop | error "robot arm is in emergency stop" |
| dataset inference is not running | error "dataset inference is running, stop it first" |
| the base and third-person camera have not moved | the viewpoint/frame changed; inference is not trustworthy; stop and restore |
Control rate, per-tick clamp and inference interval#
| Parameter | Default | Effect | How to tune |
|---|---|---|---|
| control rate | 30 Hz | how many targets are sent per second | match the training frame rate (30). Lower feels jerky; higher and the model cannot keep up |
| max joint change per tick | 3° | how many degrees a joint may move per tick | set 1–2° for the first run. Limits and velocity are not enforced by default, so this number alone sets the speed: 3° × 30 Hz = 90°/s is already fast |
| inference interval | 0 s | how long to wait after the last step of a batch before acquiring frames and inferring again | normally 0.3–0.5 s: let the arm settle and the camera deliver a fresh frame, avoiding a stale or motion-blurred frame from the previous move. 0 = re-infer as soon as the queue empties (lowest latency, least stable frame); too large creates a visible pause between chunks |
The clamp is applied per joint: target = previous target + clamp(model request − previous target, ±limit). So even if the model outputs something absurd (say, a wrist jump of 80°), the arm only moves 3°.
Inference interval: a stable window for frame acquisition#
The inference interval is not the model "resting"; it is a wait after the last action of a batch, so the arm and camera can settle before the next inference:
last action of the batch sent
→ wait for gap (the control loop keeps holding the last target)
→ read joint angles + take the latest camera frame
→ infer → next batch of actions
It fixes the frame-acquisition timing. If you infer immediately after the last step, camera.latest may still
be an old frame captured while the arm was moving, or exposure/transport may not have caught up; the model
would see the tail of the previous motion while the joint state has already moved on. After 0.3–0.5 s the arm
has settled and the camera has produced a fresh frame, so the model input and state are consistent.
0: re-infer as soon as the queue empties; lowest latency, but most likely to use a stale frame. Use it only with very low camera latency and slow motion.0.3–0.5 s: at 30 fps this waits roughly 9–15 frames, usually enough to settle and get a fresh frame; start with 0.5 s on the first real-robot run.>1 s: a visible pause between chunks; the task cadence slows down. Use it only when the task itself calls for "move, stop, observe".
During the wait the control loop still holds the last target at 30 Hz, so the arm does not go limp; "Stop" interrupts the wait immediately. If
camera ageremains large, investigate the camera/USB/CPU instead of increasing the interval further.
How joint speed affects inference#
Joint speed is set by control rate × max joint change per tick (approximately):
| Control rate | Per-tick clamp | Approximate maximum joint speed |
|---|---|---|
| 30 Hz | 1° | 30 °/s |
| 30 Hz | 2° | 60 °/s |
| 30 Hz | 3° | 90 °/s |
| 30 Hz | 5° | 150 °/s |
- Too fast: the policy was trained at human demonstration speed. If inference is much faster than the training distribution, the arm overshoots states it has seen, the image motion-blurs, the servo lags, and the next observation no longer matches the model's assumptions; motion can jitter or overshoot.
- Too slow: the model outputs actions organized as a chunk. If the arm never reaches the predicted position, the next inference still sees the previous segment's state and the model keeps "catching up"; the task becomes slow or stalls.
- This is a bound on the target change, not an exact velocity setting: actual joint speed also depends on load, servo response and tracking error.
- For the first run use 1–2° (30–60 °/s); after confirming stable motion, go to 3°; use 5° (150 °/s) only with plenty of free space and a simple task.
- If tracking error grows, the image blurs, or the arm overshoots: lower the per-tick clamp. Do not try to "slow down" with a larger inference interval — the interval only pauses between batches, it does not change the speed inside a batch.
What the safety layer does in real-robot inference#
Real-robot inference and leader–follower teleoperation are the same class of mode: a human is in the loop.
In teleoperation the reference is the operator's hand; in inference it is the model output — the policy has
already learned from the demonstrations how to move, so if the software then clips the target against limits
or keep-out zones, the follower just sticks at the boundary while the model keeps pushing outward every tick.
So targets still pass through SafetyLayer.filter(), but by default joint limits, velocity/acceleration,
table height and the base keep-out zone do not block them (ActionSource.POLICY, the same exemption as
ActionSource.TELEOP).
| Rule | Ordinary motion (move / replay / align) | Real-robot inference (default) |
|---|---|---|
| Joint limits | block | do not block |
| Velocity / acceleration | limit | no limit: only "max joint change per tick" remains |
| Table height / base keep-out zone | block | do not block |
| Tracking error | not applicable | on: the command may lead the measurement by at most 15°, so the target cannot drift away under a stall |
| E-stop / over-temperature / heartbeat | block | block, same as teleoperation |
Recommendation: for real-robot inference, also turn the safety zone / keep-out enforcement off (keep
policy_enforce_limits = false). The policy emits a continuous position stream. If the safety zone clips the target at the boundary, the follower stops halfway while the model keeps emitting the next chunk; the trace, observation and action no longer correspond, and the inference result is badly distorted. Use the same precautions: clear workspace and e-stop within reach. Turn enforcement back on if fixed obstacles or people can enter the workspace.
So "max joint change per tick" is the only speed knob in the inference chain: 3° × 30 Hz = 90 °/s. For the first run on real hardware, still use 1–2°, keep the mouse/finger on the e-stop button and have a switchable physical power switch as a backup.
To make inference stop on limits / keep-out zones as well: tick "Enforce limits / keep-out zones / speed during inference" on the real-robot inference card, or set
[safety] policy_enforce_limits = truein~/.starm/config.toml. Once ticked, the follower stops at the boundary and the page lights up "Limit / Outside workspace".
Preview: live view and "Model Input"#
A button at the top right of each camera card switches between two previews:
| Preview | What it shows | When it updates |
|---|---|---|
| Live camera view (default) | the raw frames the camera is capturing right now | continuously, about 3 fps |
| Model Input | the exact frame fed to the model in the most recent inference: BGR→RGB, resized to the resolution chosen at build time (e.g. 640×360) | once per inference; the image stays frozen between inferences |
This is the most direct tool for diagnosing "the model looks at A but grabs B": if the camera preview looks fine but in the model input the object is not centered, the colors are swapped, or it has been cropped away, the problem is in collection or dataset build, not in the model.
The input readout at the bottom right of the page is the index of the inference whose frame is shown;
gap is the current inference interval.
Joint-angle traces (x-axis = frame number)#
The bottom of the page has six trajectory plots; one point is recorded for every action frame sent: orange = the target sent, white = the joint angle observed at that moment. The x-axis is the frame number (how many action frames have been sent; 1 second = 30 frames at 30 Hz), not seconds — so "it twitched at frame 900" maps directly onto frame 900 of the recording.
Only the most recent 9000 frames are kept (about 5 minutes at 30 Hz). When old frames are dropped the
numbering keeps counting up, so the frames N–M label on the page never goes out of sync; the traces
remain after inference stops, for review.
What to watch while it runs#
| Metric | Normal | When abnormal |
|---|---|---|
Inference time infer_ms |
20–60 ms (GPU) | > 33 ms cannot keep up with 30 Hz; ticks get dropped |
| Queue remaining | cycles 0–60 | pinned at 0 → inference is too slow |
| Camera latency | < 100 ms | larger means the camera is holding things back |
When it misbehaves#
| Symptom | Action |
|---|---|
| Does not move / jitters slightly | per-tick limit too small (1°), or the model output is nearly constant (see the plots in §5) |
| Motion goes the wrong way | top/wrist were swapped at training time; check the device mapping in Chapter 8 |
| Lunges on start | hit "Stop" immediately, drop the per-tick limit to 1° and retry |
| Grasps air | not enough data, or the placement is outside the training distribution |
7. One Complete Path#
1. Collect 50+ episodes (same task, same placement, same lighting)
2. Run QC; drop episodes with a high duplicate-frame rate or bad timestamps
3. Build the dataset (top+wrist, 640×360)
4. Train: a quick 3000-step run first to prove the pipeline → then standard
5. Dataset inference: chunk step = chunk (60 by default); run the training set first, then a held-out test set, and compare MAE
6. Real-robot inference: 30 Hz, per-tick limit 1–2°, mouse/finger on the e-stop for the first run
7. If it is unstable, add data and retrain; do not keep tuning inference parameters
8. FAQ#
| Symptom | Cause | Fix |
|---|---|---|
| "Build Dataset" is grayed out | no source selected / no camera ticked / no episodes ticked | fill them in, in that order |
| Build fails with "no video features" | the source recording has no camera stream | go back to Chapter 8 and re-record |
| Training is very slow | no CUDA | use another machine, or validate with quick first |
l1_loss does not fall |
too little data / cameras misaligned / lr oscillating | check the sample-size warning first |
| Inference fails with "model not found" | wrong checkpoint directory | confirm checkpoints/<step>/pretrained_model/ contains config + weights |
| Inference fails with "model has no image input" | no camera was ticked at training time | retrain |
| Real-robot inference does not move | camera not open / the model needs two streams but only one is open | open each camera listed in the model info |
References#
- Chapter 8: Data Collection — where training data comes from
- Chapter 9: Data QC — what data should not be used for training
- Code:
webapp/trainer.py(build + training),webapp/inference.py(both inference modes)