SO-101
Course overview/Chapter 07 · Camera Calibration
中文
Chapter 07Teleoperation & Vision~28 min

Camera Calibration

Companion page: the "Camera Calibration" tab in the console

8 sections 4 figures 11 code blocks VisionHomographyArUco

1. Two Different Things: Intrinsics and Extrinsics#

The term "camera calibration" often conflates two things. Let's separate them first:

Intrinsics Extrinsics (hand-eye)
Question answered How large an angle does one pixel in the image subtend? Which point on the table does a pixel correspond to?
Parameters Focal length, principal point, distortion coefficients Pose of the camera relative to the robot arm
How to calibrate Photograph a checkerboard and use Zhang's method Collect several (pixel, robot-arm coordinate) pairs
What forces recalibration Changing the lens or resolution, refocusing The camera has been moved

This page covers extrinsics (hand-eye calibration): mapping pixels reliably to the robot arm's table coordinates.

2. Eye-in-Hand or Eye-to-Hand#

Eye-in-hand Eye-to-hand
Where the camera is mounted On the wrist, moving with the arm Fixed to a bracket / the table
What is calibrated Camera → flange Camera → base frame
Advantages Can get close for detail, immune to occlusion Stable field of view, calibrated once and left alone
Disadvantages The viewpoint changes with every move, so calibration is harder The arm may block the view

This page uses eye-to-hand, looking down at the table — the most common setup for desktop arms like the SO-101.

3. Homography: Turning Pixels into Table Coordinates#

Assuming all objects lie on the same plane (the table), mapping pixel (u, v) to table (x, y) is a homography transform:

text
        [h11 h12 h13]
[x y 1]ᵀ ~ [h21 h22 h23] · [u v 1]ᵀ
        [h31 h32 h33]

Expanded, that is:

text
x = (h11·u + h12·v + h13) / (h31·u + h32·v + h33)
y = (h21·u + h22·v + h23) / (h31·u + h32·v + h33)

The matrix can be scaled as a whole, so it has 8 degrees of freedom. Each point pair yields 2 equations, hence:

4 point pairs are exactly enough to solve for the 8 unknowns.

Important: Residuals Are Always Zero with 4 Points#

8 equations for 8 unknowns — mathematically a unique solution, so the residuals are necessarily zero. In other words:

With only 4 points, you can never tell how accurate the calibration is.

To evaluate the error you must use more points (the nine-point method) or hold back a few points purely for validation. The solver on this page computes a residual for every point; with fewer than 5 points, those zeros carry no information.

4. How to Do Nine-Point Calibration#

This is the most classic approach, and the easiest to set up on a tabletop:

Steps#

  1. Place 9 points on the table, spread out in a grid — don't cram them into one corner (the wider the spread, the more stable the extrapolation)
  2. Have the TCP touch each of the 9 points in turn: - Real robot: teach them by hand, or use the "End-Effector Pose" page to move the TCP above each point and then lower it - Each time you touch a point, record the robot arm's (x, y) — this is the ground-truth table coordinate of that point
  3. Click the same point in the image to get its pixel coordinate (u, v)
  4. Fill both columns into the "Calibration Points" table: one row per point
  5. Click "Solve Homography"

How to Place the Calibration Board#

Nine-point calibration needs known, clearly visible points on the table. Three approaches, from simplest to most refined:

Approach Materials Best for
Print a nine-point sheet Print 9 crosses/dots on A4 paper and tape it flat to the table Least effort, recommended
ArUco board Print markers and let a detection algorithm find their centers Accurate points, automatable
Use table features directly Existing screw holes, corners of stickers Emergencies only, poor point accuracy

Key points:

  1. Spread them out: the 9 points should cover the whole work area; don't cluster them in a small patch in the middle. Extrapolation (image edges) has far larger error than interpolation.
  2. Press them flat onto the table: the paper must lie flat with no curled edges — a homography assumes all points lie on the same plane.
  3. Points must be clearly visible: solid dots or crosses 5–10 mm in diameter are best; anything smaller can't be clicked accurately in the image.
  4. Same height: the calibration points should lie in the plane of the actual grasp height (usually the table surface).
Nine-point calibration board: a 3×3 grid of ArUco markers spread across the top camera's work area

Example: a 3×3 nine-point board under the top camera. The nine markers are roughly evenly spaced across the work area, not clustered in the middle of the frame.

After laying out the 3×3 grid in the photo, check a few things:

  • Cover the work area: the nine points should span the region you will actually grasp, including the four corners and edges. A small cluster in the middle gives large extrapolation error near the edges.
  • One plane, flat on the table: the paper or board must not curl or sit on a spacer; the homography is only valid on this plane.
  • All nine points inside the frame: the top camera must see every marker completely; do not let the frame edge cut a marker off, and keep the arm/gripper out of the way.
  • Markers large enough, not overexposed: in the photo they are still legible in the 640×360 preview. If they look white or washed out, lower the brightness (about -20 to -40) before detecting.
  • Do not move the board after detection: IDs #1..#9 are paired with table coordinates one-to-one. If the board moves, detect again and re-teach the coordinates.
  • The count is flexible: five points can solve a homography, but 8–9 spread-out points give a more trustworthy residual; nine is the usual compromise.

Markers on the Gripper or on the Table#

These are two different calibration methods, both valid, depending on what you have available:

Markers on the table (this page) Markers on the gripper
Also called Planar calibration / nine-point method Hand-eye calibration (eye-to-hand)
Requires The ground-truth table coordinate of each marker (taught with the TCP, or measured) Camera intrinsics + multiple poses
Does the arm move Only to "teach coordinates" Many moves, one pose each
What is solved Homography H (pixels → table mm) Rigid transform (rotation + translation) of the camera relative to the base
Handles different heights No (valid only on the calibration plane) Yes (it is a full 3D transform)

So: when calibrating a top camera, the markers can indeed be fixed to the gripper — that is standard hand-eye calibration (cv2.calibrateHandEye, AX=XB). It does not require knowing the coordinates of any point on the table, but it does require:

  1. Intrinsics calibration first (otherwise the marker's 3D pose cannot be solved)
  2. Moving the arm through 5–10 positions with clearly distinct poses, recording the robot's TCP pose and the marker's camera pose at each position
  3. Solving for the camera → base transform

What this page does today is the left-hand column (table markers + homography), because it needs no intrinsics and a single calibration is enough to grasp flat objects. The right-hand approach (markers on the gripper, multiple poses) requires intrinsics calibration and 3D pose solving first — that is the next step.

In one sentence: board on the table → calibrate the planar "pixel to table" mapping; board on the gripper → calibrate the 3D "camera to arm" relationship.

Clicking: One Point at a Time#

The point picker on the page can record a point each time you click the image, automatically attaching the current TCP coordinates (there is a toggle at the top; remember to uncheck it when the arm is not at a point):

text
① Park the TCP directly above a marker → lower it until the TCP's vertical projection lands on the marker
② Click that marker in the image
   → a new row appears in the table: pixel (u,v) from the click, (x, y) taken from the current TCP
③ Move the TCP to the next point and repeat

So the order is position the arm first, then click the image. If you click the image first and then move the arm, that row's (x, y) won't match — re-click the point after moving, or edit the numbers in the table manually.

With the robot arm disconnected, clicking the image only records the pixel; (x, y) must be filled in manually.

Top-Camera Calibration Workflow#

The top camera is fixed, so one calibration is globally valid.

Preparation

Step Action
1 Connect the robot arm (real hardware) and confirm the torque is normal
2 Camera page → view top → device path (e.g. /dev/video2) → "Connect This View"
3 Set brightness to -20 ~ -40 (overexposure turns markers gray and they won't be detected)
4 Tape the markers to the table, spread out over the work area (at least 5, 8–9 recommended)

Capturing Points

Step Action
5 Select "Any Square Marker" as the marker type (for standard ArUco, choose ArUco + dictionary)
6 Click "Detect ArUco Markers" → the image outlines #1..#N and the table is pre-filled with pixels
7 Check the correspondence between the numbering and the physical markers on the table (the image shows #N labels)

Teaching Coordinates (repeat for each point)

Step Action
8 Use "End-Effector Pose" to move the TCP directly above the marker
9 Lower z to 32–35 (almost touching the table) — too high introduces perspective offset
10 Fine-tune x/y so the TCP points straight at the marker center
11 Click that row in the table (it turns blue) → "Fill Table Coordinates from Current TCP"

Can't hold it steady by hand? Click "Teach Mode" on the camera card: torque is released, so you can place the TCP on the marker by hand, then click "Fill Table Coordinates from Current TCP", record it, and move on to the next one. Click "Finish Teaching" to re-enable torque when done.

Solving and Saving

Step Action
12 Click "Solve Homography"
13 Check the RMS: < 3 mm passes; otherwise find the row with the largest residual and go back to re-teach it
14 Click "Save as Calibration for Current View"

Wrist-Camera Calibration Workflow#

The wrist camera is mounted on the hand, and one calibration is valid for one pose only, so there is an extra "record pose" step.

Step Action
1 Switch the view to wrist → device path (e.g. /dev/video0) → "Connect This View"
2 Move the arm into a pose that sees the table clearly (looking down at the marker area)
3 Click "Record Calibration Pose" — this step matters; everything below depends on it
4 Detect → the table is pre-filled with pixels (note: switching views clears the point table, so switch the view first, then detect)
5 Teach coordinates one by one (same as the top view: move the arm → select the row → "Fill from Current TCP"); "Teach Mode" lets you position it by hand
6 When done, return the arm to the pose from step 3 (the hint line shows "Pose matches ✅")
7 Solve → Save

How do you get back to that pose in step 6? The hint line shows the full set of numbers:

text
Recorded 11:48:10  x 45.3  y -219.9  z 154.2  pitch 69.9  roll -90.5

Enter them on the "End-Effector Pose" page → click "Move". Note that the input boxes on that page are target values, not live readings; there is a separate green "Current: ..." line at the top of the page.

The pose is written into the file when you save, so if you look at this wrist record days later you still know which pose it belongs to.

Why the Wrist Camera Can't Be Calibrated This Way#

The procedure above holds only for a fixed camera (the top view).

The wrist camera is mounted on the arm, so the image changes completely as soon as the arm moves; the same table produces totally different pixels in different poses. Therefore, for the wrist camera:

  • A single fixed homography is valid only in one pose and fails as soon as the pose changes
  • The proper approach is to calibrate camera → flange (eye-in-hand), then combine it with FK to work back to the base
  • The H saved for wrist on this page is only suitable for the case "arm held still, wrist camera watching a fixed area"

In one sentence: for grasping, calibrate the top camera for hand-eye; the wrist camera is mostly for looking at things up close.

How This Maps to the UI#

Operation Action on the page
Capture one point Click directly on the camera image (pixel and TCP coordinates are recorded together)
Edit table coordinates Edit x / y directly in the table
Fill in coordinates Select the row → "Fill Table Coordinates from Current TCP"
Start over "Clear"
Solve "Solve Homography"

Using an ArUco Board#

Hand-clicked pixels carry 1–3 pixels of error; ArUco detection reaches sub-pixel accuracy, so the points are more accurate and you don't have to click as carefully.

Printing Markers

python
import cv2
d = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50)
for i in range(6):
    img = cv2.aruco.generateImageMarker(d, i, 300)   # 300 px ≈ 40 mm printed size
    cv2.imwrite(f"marker_{i}.png", img)

Key points:

Item Recommendation
Dictionary 4X4_50 (the most common choice for self-printed markers); for a commercial board, use the dictionary printed on it
ID Consecutive numbering 0, 1, 2…; the table later maps rows by ID
Size Measure the actual side length after printing (e.g. 40 mm) and write it down
White border Leave at least 10 mm of white margin around each marker, or detection will be unstable

Placement

  1. Tape them to the table (or print them on one sheet and lay it flat); as with the nine-point method, they must be on the same plane, with no curled edges
  2. Spread them out: cover the work area; don't cluster them in the middle
  3. Don't orient them all the same way: a slight rotation makes them easier to recognize and helps check whether rotation is calibrated correctly
ArUco detection
Any square marker

What If the Board Is Not Standard ArUco

Many boards (self-printed or purchased) are not in OpenCV's twenty-odd dictionaries. Don't guess the dictionary in that case — just switch:

Marker type What it recognizes Best for
Any square marker Only "quadrilateral + black border + pattern inside" Any board, regardless of which family it belongs to
Standard ArUco Must match a pattern in the dictionary Boards you printed yourself with generateImageMarker

The criteria for "Any Square Marker" (these are also the rules that filter out false detections such as table corners):

  1. The contour is a convex quadrilateral with an aspect ratio of 0.7–1.4
  2. After perspective correction, the outer ring is dark (the marker's black border)
  3. The interior has enough contrast (it contains a black-and-white pattern, not a plain sheet of paper)

The cost: no IDs, and no sub-pixel accuracy; markers are numbered automatically "top to bottom, left to right". For nine-point calibration neither matters — you have to teach the coordinates yourself anyway.

Using It in the UI

  1. Select view top and connect the camera (or use the simulator)
  2. Choose the marker type: if the board's provenance is unknown, choose "Any Square Marker"; for standard markers you printed yourself, choose "Standard ArUco" and pick the dictionary
  3. Click "Detect ArUco Markers" - Each marker is outlined in the image and labeled #ID - The table is auto-filled with the marker ID and pixel coordinates
  4. Teach the table coordinate of each marker: - Move the TCP directly above marker 0 → lower it so the TCP points straight at the marker center - Select the #0 row in the table → click "Fill Table Coordinates from Current TCP" - Move to the next marker and repeat
  5. When everything is filled in, click "Solve Homography"

What detection does for you is "find the pixels"; the table coordinates still have to be taught with the TCP — unless you measure each marker's position with a ruler and type it in directly. If the markers are printed on a board with fixed spacing, measuring the spacing once is enough to derive all the coordinates (currently these have to be entered by hand).

What If Nothing Is Detected

Symptom Cause
Nothing detected at all Wrong dictionary (try "Any Square Marker"); markers too small; too blurry; image overexposed or too dark
Only some detected Markers occluded, out of the field of view, or at too oblique an angle
IDs don't match A different dictionary or different IDs were used when printing

The "Fill Simulated Markers" button on the page works only in simulation and uses the scene's built-in known points.

How to judge "too small": look at how many pixels the marker occupies in the image. Measured on a real tabletop image with a standard 4X4_50 marker:

Marker side length Result
24 px ✅ detected
40 px ✅
100 px ✅

In other words, a standard marker is enough at 24 pixels. If markers are clearly present in the image but none is detected, you can be fairly sure that the pattern is not in the selected dictionary — common causes:

  • The "ArUco board" you printed from vendor A or vendor B actually uses a different dictionary (5X5_50, APRILTAG_36h11, …)
  • The image was resized or re-encoded and the pattern was distorted
  • The white border was cropped off when printing

The quickest check: use the generateImageMarker snippet above to regenerate a DICT_4X4_50 marker, print it, and it is guaranteed to be detected.

INTERACTIVE LAB

Pixel → table

Homography · 4 points

Four corner points define a homography that maps table coordinates to pixels. Drag any corner and watch the “estimated grid” warp; the residual at those four points themselves is always 0.

True gridEstimated gridCalibration points
4-point residual0.00mm
9-point spot-check RMS0.00mm
Max deviation0.00mm

During calibration you click 4 known points on the table; the solved matrix always passes through those 4 point pairs—so 4 points alone can never tell you whether it is accurate.

You have to spot-check a 5th and 6th point to see whether the matrix is warped in between.

Table coordinates are illustrative, 0–400 × 0–300 mm; the real calibration numbers are in §4 of this chapter. This lab only demonstrates the mapping structure and does not represent the actual error of any particular camera.

5. Where the Error Comes From#

Measurement: What One Bad Point Does#

Solving a homography from 5 points:

Case RMS Per-point residual
All 5 points captured accurately 0.0 mm all 0
Point 5 clicked more than a pixel off 0.52 mm 0.14 ~ 0.94 mm
Point 5's table coordinate deliberately wrong by 15 mm 5.38 mm 2.4 ~ 9.7 mm

The third row is worth noting: point 5 is the wrong one, and it also has the largest residual (9.7 mm) — the residual table helps you find which point is off.

Main Error Sources#

Source Magnitude How to mitigate
Inaccurate point capture (hand-clicked pixels) 1–3 pixels → 1–3 mm Zoom the image; use the marker center
Uneven table / tilted camera a few mm Keep the camera as vertical as possible
Lens distortion most visible at image edges Do intrinsics calibration first to undistort
Objects not at the same height offset ≈ height difference × distance from optical axis / camera height; not necessarily 1:1 Recalibrate at the actual grasp height; only a low camera or a far-off-axis object approaches 10 mm per 10 mm

The last one is the easiest to overlook: a homography assumes all objects lie on that plane. A 30 mm-tall object without recalibration means grabs are systematically off — this is the "failure demo" for G2 in the syllabus.

How to Validate#

Hold back 3 points that do not participate in the solve:

  1. Use them to compute predicted coordinates and compare with the measured ones
  2. Only a residual < 3 mm passes (the acceptance criterion for G2 in the syllabus)
  3. Spread the three points across different parts of the image — testing only the center overestimates accuracy

6. Using the Page#

Camera calibration page
  1. Connect the camera (see "Connecting a Camera" below) and wait for the image before picking points
  2. Select a view: top for overhead / wrist for the wrist
  3. Click a point on the image → a row appears in the table showing the pixel coordinates
  4. Select that row → click "Fill Table Coordinates from Current TCP" (or type x / y manually)
  5. Repeat steps 3–4 for at least 5 points (9 recommended, spread out)
  6. Click "Solve Homography" → check the RMS and the per-point residuals
  7. If a residual is large, check the corresponding point; one value that is much larger than the rest usually means that point is off

Connecting a Camera#

At the top of the page there are "View", "Device Path", and "Scan Devices", with a row of buttons below them: Connect / Disconnect / Refresh / Upload.

Each of the two cameras takes one role:

Role Where it is mounted What it is used for
top Fixed directly above the table Hand-eye calibration, watching the material
wrist On the wrist, moving with the arm Close-up view of the gripper and the grasp target

Select the role first, then fill in the device path (e.g. /dev/video0) and click "Connect This View".

Resolution: the camera opens at 1280×720 by default. Many webcams fall back to 640×480 when opened with OpenCV defaults, so the width and height are set explicitly on connect — the status line shows the actual values (e.g. top /dev/video2 1280×720). The higher the resolution, the more easily small distant markers are detected.

Live view: once connected, the image refreshes automatically (about 6–7 fps); there is no need to click "Refresh Image" again. Switching to another tab pauses it, and it resumes automatically when you come back.

Brightness / gamma: overexposure smears the markers' black borders into gray, which makes detection hard. There are two sliders on the card:

Control Range Effect
Brightness -64 ~ 64 Overall lightness, the most direct control
Gamma 100 ~ 500 Midtone curve; the higher, the brighter

Measured (same camera): at brightness 0, 4.7% of pixels were overexposed; at -40 this dropped to 0% and the markers' black borders became crisp again.

Manual exposure does not work on some UVC cameras (after switching to manual the whole image goes black), so brightness/gamma are used to darken the image here rather than changing the exposure time. When both views are connected, the status line shows both:

text
top /dev/video2 640×480 | wrist /dev/video0 640×480

Switch the "View" dropdown and the image switches between the two.

Both Views Can Be Connected at Once#

The two cameras are independent and can run at the same time:

text
top   /dev/video2 640×480     ← table
wrist /dev/video0 640×480     ← wrist

The status line shows both, and switching the "View" dropdown switches the image. Each view grabs independently without affecting the other (one background thread each), so you can watch the wrist view for alignment while picking points on the table view.

Calibrate and Save Separately#

Calibration results are stored per role, one each for top and wrist:

  1. Select view top → pick points → solve → click "Save as Calibration for Current View"
  2. Switch the view to wrist → pick points again (the points in the wrist image are completely different from the table's) → solve → save again

After saving, the "Result" card lists both entries:

View Saved at RMS Points
top 11-09 10:21 0.52 mm 5 points
wrist 11-09 10:21 1.80 mm 9 points

The files are written to:

text
$STARM_HOME/camera_calibration.json     # default: ~/.starm/camera_calibration.json

One entry per role, containing the homography H, the RMS, the maximum residual, the point count, the device path used at the time, the save time, and the raw point pairs from capture (for traceability).

What is saved is the calibration result, not the video stream. If you change the camera, move the fixed bracket, or change the resolution, you must recalibrate.

Telling the Two Cameras Apart#

One Camera = Two /dev/videoN Nodes

On Linux, each USB camera usually exposes two video nodes: only the lower-numbered one can capture; the other is a metadata node, and opening it fails. So when you see /dev/video0 ~ video3, don't assume there are four cameras.

Group them by physical device with this command:

bash
v4l2-ctl --list-devices
text
USB HD Camera: USB HD Camera (usb-0000:09:00.3-2):     ← camera A
    /dev/video0
    /dev/video1
    /dev/media0

USB HD Camera: USB HD Camera (usb-0000:09:00.4-1):     ← camera B
    /dev/video2
    /dev/video3
    /dev/media1

Clicking "Scan Devices" on the page does the same thing and marks nodes that cannot capture as "(metadata node)".

Which Camera Is Top and Which Is Wrist

The two cameras are the same model, so you can only tell them apart by looking at the image:

  1. Connect each path once (the role doesn't matter) and look at the image: - You can see the whole table, the table edges, and the arm from above ← this is top - Two gripper fingers at the bottom of the image, viewpoint right at the table surface ← this is wrist
  2. Once identified, connect both and assign each its role

You can also unplug and replug to confirm: unplug the wrist camera and see which node disappears.

Measured on this machine: /dev/video2 is the top view (looking down at the table), /dev/video0 is the wrist (the gripper is visible). The numbering may change after switching USB ports or computers; trust the image.

Device Path or Device Number?

Form Notes
/dev/video2 Explicit, but the number may change after replugging / changing ports
2 Equivalent to opening by number, and it drifts the same way

Only device paths are supported now: numbers drift when devices are plugged and unplugged, whereas a path at least tells you at a glance which view it is.

The Camera Won't Open#

Error Cause Fix
Cannot open /dev/video0 Device missing / busy / no permission ls /dev/video* to list devices; close other preview programs; add the user to the video group
Stuck on "Opening" Device held by another process, read() blocks Disconnect and reconnect, or restart the service to release it
Image frozen Camera unplugged / unstable USB power Replug and reconnect
Only one of the two views works Both views are given the same path Give each view its own path

A quick check from the command line saves time:

bash
v4l2-ctl --list-devices
python -c "import cv2; c=cv2.VideoCapture('/dev/video0'); print(c.isOpened())"

In the simulator, "Fill Simulated Markers" inserts 4 known markers in one click so you can walk through the whole flow; but remember — with 4 points there are no residuals to inspect.

7. FAQ#

Symptom Cause Fix
Solve button does nothing / reports too few points Fewer than 4 points Add points
Residuals are all zero Only 4 points Add more until you have at least 5, then look again
One point's residual is much larger That point is off Re-capture it
No image Camera not connected / busy Click "Connect This View"; or use "Upload Image" to import a screenshot
Grasps are consistently off in one fixed direction Object height differs from the calibration plane Recalibrate at the grasp height, or compensate for the height
Edges are off more than the center Lens distortion Do intrinsics calibration first to undistort

References#

  • Syllabus module G: G1 camera setup and intrinsics, G2 hand-eye calibration
  • Chapter 4: FK and IK — where robot-arm coordinates come from
  • Code: solve_homography() in webapp/server.py (normalized DLT + SVD)
  • Simulated camera: render_top_view() in starm/backends/sim_scene.py