SO-101
Course overview/Training log
中文
TRAINING LOGSO-101

Teaching a robot arm one thing: notes from a pick-and-place training run

2026-11-11 · Hardware: SO101 leader + follower arm, one camera on top and one on the wrist · Task: pick the plug up from the table and put it in the box · Status: training finished (100k steps)

The result first: real-robot inference from the 30k checkpoint#

The clip above is the full twenty-four seconds (converted to an animated image, so no audio). This is the checkpoint saved at step 30k, loaded straight onto the real arm — no teleoperation anywhere in it. The arm sees the plug on the table by itself, reaches for it, picks it up and drops it in the box.

Full video: 00-inference-30k.mp4 (1080p, 24 s)

1. First, the arm has to follow me#

A robot arm does not figure things out on its own. It can only learn from "what a person does when the situation looks like this". So the first step is to be that person for a while.

I connected the leader and the follower arm and tried them out — I moved the leader, the follower moved with it. That part is oddly satisfying, like holding an invisible string. Once it tracked my hand properly, I opened the data collection page, gave the dataset a name, and switched on both cameras: the top one shows the whole scene and tells you where the plug and the box are, the wrist one shows the gripper and tells you whether you are actually lined up.

Then it is the same loop over and over:

Place the plug somewhere on the table → guide the follower there with the leader → line up → close the gripper → lift → move over the box → release → pull back.

Stop the recording, move the plug, go again. I deliberately spread the positions out — left, right, closer, further, on an angle, even right against the edge of the box. What the arm learns is "what to do in this situation", so the more situations it has seen, the less it freezes when a new one shows up.

This is the dullest step and the most important one. Fifty runs of twenty-odd seconds each does not sound like much, but in practice it is half an hour of bending over, watching two screens, and nudging things around. Your hand has to stay steady too — anything shaky you record, the model will happily learn to be shaky as well.

Data collection page

One thing went wrong partway through: the two camera feeds dropped out one after the other. The log window said the camera had disconnected, then a moment later that it was reconnecting. Those few seconds are a black frame or no frame at all, which effectively ruins that run. So I kept the log open at the side the whole time, kept the laptop away from the USB cable, and stopped it from going to sleep on me.

2. Recording something doesn't make it usable#

Raw recordings are not automatically good data. The data QC page does something plain: it walks through the runs one by one and checks whether frames went missing, whether the picture is blurry, whether there are duplicate frames, whether the actions line up with the joint angles, and whether all the timestamps agree.

The verdict was slightly alarming at first — the page flat out suggested deleting them. But when I opened up the details, only one run was actually an error. The rest were in the "worth a look" category, things like a couple of duplicate frames in the wrist camera or a joint column that looked a little odd. It was a report full of arrows, not a diagnosis.

Data QC page

So I deleted exactly that one run, put the plug back in roughly-but-not-exactly the same spot, and recorded a replacement.

The point I only understood afterwards: "suggest deleting" is an overall hint, not a verdict. If I had followed it literally I would have thrown away most of the fifty runs and wasted the whole afternoon. What you must deal with is the error tier; warnings are a judgement call, and the informational notes can be ignored.

3. Turning raw recordings into a dataset#

A raw recording and a dataset you can train on are not the same thing. The raw files carry all the little defects from collection: every run starts with a stretch where the hand has not begun to move, and the two camera streams are not even stored in the same format.

Building the dataset is basically washing vegetables. You point it at the raw recording folder, it scans and finds every run and confirms both cameras are present, you select them all, tick "trim the idle opening", write one sentence describing the task so the model knows what it is looking at, give it a name, and press build.

Then you wait. It validates each run and re-encodes both video streams, one line of log at a time. Nothing difficult about it, but it is slow — a good moment to go and make tea.

Build dataset page

The finished dataset is noticeably smaller than the raw footage. What got dropped is not "less data", it is "less junk" — those idle openings are exactly what confuses a model, because they teach it that doing nothing is a reasonable action.

4. Letting it learn on its own#

Training uses ACT, and the console already had the presets ready. I barely touched the parameter page: batch size, action chunk length, learning rate, total steps — the page had worked out sensible defaults from the size of the dataset, so I glanced at them and hit start.

The only thing I hesitated over was how many steps to run. The page's suggested ceiling was a hundred thousand, and there was also a higher number for "run the full preset number of epochs". I took the ceiling. The reasoning is that the payoff curve for this kind of grasping task is steep: most of the change happens in the first ten thousand steps, it has largely converged by twenty or thirty thousand, and after that you are mostly polishing details — and polishing too long just overfits the particular table you trained on. If time is short, twenty or thirty thousand steps is enough to try it on the real arm once.

Training page

Once it was running I watched two things.

The first was the loss. It starts in the teens, dives almost vertically for the first few hundred steps, is down around 1 by roughly a thousand steps, and then turns into a long slow tail. That is the most satisfying stretch — the moment it goes from guessing to having a rough idea of which way to move. After a few thousand steps the loss is down in the fractions, and the action-prediction error is low too, which means it has a decent grip on what the next step should look like.

The second was speed. A few steps per second, the progress bar visibly creeping, and on that pace it would take most of a day. So I let it run and went off to do something else. It is the one genuinely restful part of the whole pipeline: collection needs a human, the dataset build needs patience, but training is the machine doing your shift for you.

The raw recordings live under recordings/, the cleaned dataset under datasets/, and the training output under outputs/train/ in a directory named after the dataset and the start time.

5. When training ends: where the loss landed#

Loss curve and final numbers after 100k steps, opened from the training history table

This is the run opened from the training history table after all hundred thousand steps had finished: loss fell from the teens all the way to 0.038, and the tail of the curve has flattened out. The numbers underneath are the final l1 / kld / learning rate / memory / speed, plus the batch, chunk and checkpoints this run used. Once training is over — even after restarting the console — clicking the run in the history table brings the curve and these numbers back, without retraining anything.