Week 11 — Machine Learning for Robotics Day 2 of 7
Why This Matters
A warehouse robot moving at 1.5 m/s has roughly 200 ms to react to a person stepping into its path. That budget covers sensor readout, perception inference, motion planning, and motor command. If your object detector misses a pedestrian at 10 m in low light, the robot cannot stop in time. At 1.5 m/s with 0.5 s of total braking latency, the stopping distance is 0.75 m plus the perception delay distance. A single missed detection is not a model metric problem. It is a safety incident.
Supervised perception is the part of the ML pipeline that turns raw pixels or point clouds into labeled boxes, masks, and 6-DoF poses that a planner can consume. Without it, your robot is blind to the world beyond its own encoders. The failure modes are specific and measurable: a detector that scores 0.45 mAP on COCO can still miss 8% of small objects at range, and a pose estimator with 3 cm translation error will fail a 5 mm peg-in-hole insertion every time. This lesson builds the engineering judgment to know which numbers matter for your robot and why.
Core Concepts
Object Detection: Anchors, IoU, and the NMS Trap
Modern detectors fall into two families. Two-stage detectors (Faster R-CNN, Mask R-CNN) propose regions then classify them. One-stage detectors (YOLO family, SSD, RetinaNet) predict boxes and classes in a single pass. For a 30 Hz control loop, one-stage wins on latency. A YOLOv8-small at 640×640 runs about 12 ms on an NVIDIA Jetson Orin NX at INT8, which fits the budget. A Faster R-CNN with ResNet-50 backbone runs roughly 45 ms — too slow for a 30 Hz loop but fine for a 5 Hz pick-and-place cell.
The core metric is Intersection over Union (IoU). A prediction counts as a true positive only if IoU with the ground-truth box exceeds a threshold, typically 0.5. Mean Average Precision (mAP) averages precision across recall levels and IoU thresholds. COCO mAP@[.5:.95] averages over ten IoU thresholds from 0.5 to 0.95 in steps of 0.05. This is stricter than PASCAL VOC mAP@0.5 and better reflects localization quality.
Non-Maximum Suppression (NMS) merges overlapping predictions. Here is the trap: standard NMS uses a fixed IoU threshold, often 0.45. Two people standing shoulder to shoulder produce boxes with IoU around 0.4. Standard NMS may suppress one as a duplicate. Soft-NMS or class-aware NMS with a higher threshold (0.6–0.7) fixes this. In a crowded warehouse aisle, this single parameter change can recover 5–10% recall on person detection.
Segmentation: Semantic, Instance, and Panoptic
Semantic segmentation labels every pixel with a class (floor, shelf, person). Instance segmentation separates individual objects of the same class. Panoptic segmentation unifies both: “stuff” classes (floor, walls) get semantic labels, “thing” classes (boxes, people) get instance IDs.
For a robot arm picking translucent plastic bins, semantic segmentation of the bin surface is not enough. You need instance masks to compute each bin’s centroid and grasp point. Mask R-CNN adds a small mask head (a 14×14 or 28×28 per-ROI fully convolutional branch) to Faster R-CNN. The mask head adds about 20% latency but gives pixel-accurate boundaries. For a 2 cm tolerance on grasp point, mask quality matters more than box quality.
A practical number: for a 1280×720 depth-aligned RGB stream, a real-time instance segmentation model like YOLOv8-seg at 640×640 runs about 18 ms on an Orin NX. The mask resolution is 160×160 before upsampling, so a 1 cm object spans about 1.3 pixels in the mask logits. Sub-centimeter grasp tolerance requires either a higher input resolution or a refinement step using the depth point cloud.
Pose Estimation: 6-DoF and the PnP Chain
Pose estimation gives the rigid transform from object frame to camera frame: a rotation $R \in SO(3)$ and translation $t \in \mathbb{R}^3$. Two approaches dominate. Direct regression (PoseCNN, YOLO-6D) predicts pose from a single image. Keypoint-based methods (PVNet, BB8) predict 2D projections of known 3D keypoints, then solve Perspective-n-Point (PnP) with RANSAC.
Keypoint+PNP is more accurate and more interpretable. For a known object with 8 corner keypoints, PnP with RANSAC needs at least 4 non-coplanar points. With 8 points and 1-pixel keypoint noise on a 1280×720 camera with 600-pixel focal length, translation error is roughly $\frac{z \cdot \sigma_{px}}{f} = \frac{0.5 \cdot 1}{600} \approx 0.8$ mm at 0.5 m range. At 2 m range, the same noise gives about 3.3 mm. This is why pose accuracy degrades with distance — a fact many teams discover only during commissioning.
Mathematical Foundation
The detection loss for a one-stage detector combines classification and box regression. A common form (YOLO-style) is:
$$ \mathcal{L} = \lambda_{cls} \sum_{i} \text{BCE}(p_i, \hat{p}i) + \lambda{box} \sum_{i \in \text{pos}} \left[ (1 - \text{IoU}(b_i, \hat{b}_i)) + \text{GIoU penalty} \right] $$
where $p_i$ is the predicted class probability, $\hat{p}i$ is the ground-truth label, $b_i$ is the predicted box, $\hat{b}i$ is the ground-truth box, and $\lambda{cls}, \lambda{box}$ are loss weights (typically 1.0 and 5.0). Generalized IoU (GIoU) adds a penalty when boxes do not overlap, which stabilizes training on small objects.
For pose, the PnP reprojection error minimized by RANSAC is:
$$ \min_{R,t} \sum_{j=1}^{N} \left| \pi(R X_j + t) - x_j \right|^2 $$
where $X_j \in \mathbb{R}^3$ is a 3D object keypoint, $x_j \in \mathbb{R}^2$ is its detected 2D projection, $\pi(\cdot)$ is the camera projection $\pi(X) = (f_x X_z^{-1} X_x + c_x,\ f_y X_z^{-1} X_y + c_y)$, $R$ is rotation, and $t$ is translation.
Worked example. A camera has $f_x = f_y = 600$ px, $c_x = 640$, $c_y = 360$. An object keypoint sits at $X = (0.05, 0, 0.8)$ m in object frame, and the true pose is $R = I$, $t = (0, 0, 0)$. The projected pixel is:
$$ u = 600 \cdot \frac{0.05}{0.8} + 640 = 37.5 + 640 = 677.5 $$
$$ v = 600 \cdot \frac{0}{0.8} + 360 = 360 $$
Now inject 2-pixel noise in $u$: detected $u = 679.5$. Solving for $X_x$ at fixed depth: $\Delta X_x = \frac{2 \cdot 0.8}{600} = 2.7$ mm. If depth is also estimated with 1% error (8 mm at 0.8 m), the lateral error scales with depth, giving roughly 3.4 mm total. For a 5 mm insertion tolerance, this leaves only 1.6 mm margin — tight but workable with a compliant wrist.
Practical Example
Scenario: A 6-DoF arm (payload 5 kg, repeatability ±0.02 mm) performs bin picking from a shelf 1.2 m away. The camera is a 1280×720 RGB-D sensor with 600 px focal length mounted on the wrist. The task: detect a 15 cm × 10 cm × 8 cm cardboard box, estimate its 6-DoF pose, and grasp it with a 2-finger gripper. Grasp tolerance is ±8 mm lateral and ±5° rotation.
Step 1 — Detection. Train a YOLOv8-s detector on 3,000 labeled images of the box in varied lighting. Target mAP@0.5 of 0.92 or better. At 1.2 m, the box spans roughly $\frac{0.15 \cdot 600}{1.2} = 75$ pixels wide. That is small-object territory. Use input resolution 960×960, not 640×640, to keep the box above 100 pixels. Expect about 22 ms inference on Orin NX.
Step 2 — Segmentation. Run YOLOv8-seg to get an instance mask. Compute the mask centroid and the minimum-area bounding rectangle. The rectangle’s short axis gives the gripper approach direction. With 160×160 mask logits upsampled to 1280×720, each mask pixel is 8×4.5 original pixels — about 1 cm at 1.2 m. This is too coarse for the 8 mm tolerance, so refine the grasp point using the depth point cloud within the mask.
Step 3 — Pose. Define 8 box corners as 3D keypoints. Detect their 2D projections with a keypoint head, then run RANSAC PnP with a 3-pixel inlier threshold. With 8 points and 1.5-pixel noise at 1.2 m, expected translation error is $\frac{1.2 \cdot 1.5}{600} \approx 3$ mm. Rotation error from 8 points is typically under 2°. Both fit inside tolerance.
Step 4 — Validation. Before commanding the grasp, check: (a) reprojection RMSE under 3 pixels, (b) at least 6 of 8 keypoints as inliers, (c) pose depth within 5 cm of the depth sensor’s median reading. If any check fails, re-image or request human intervention. This gate prevents 90% of bad grasps in practice.
| Stage | Latency (ms) | Accuracy target | Failure gate |
|---|---|---|---|
| Detection | 22 | mAP@0.5 ≥ 0.92 | confidence < 0.6 → retry |
| Segmentation | 18 | mask IoU ≥ 0.85 | area < 500 px → retry |
| Pose (PnP) | 4 | RMSE < 3 px | inliers < 6 → abort |
| Total | 44 | — | — |
Common Pitfalls
1. Training on lab data, deploying in the warehouse. A detector trained on 3,000 clean lab images drops 15–25 mAP points in a warehouse with fluorescent flicker and reflective floors. Why: the model latches onto background statistics, not object features. Detect: run a held-out set from the deployment site before shipping. Fix: collect 500–1,000 site images, apply domain randomization (brightness ±40%, blur, noise), and fine-tune the last layers.
2. Ignoring class imbalance in segmentation. Floor pixels outnumber person pixels 50:1 in a typical navigation dataset. The model predicts “floor” everywhere and still scores 98% pixel accuracy. Why: pixel accuracy is dominated by the majority class. Detect: report per-class IoU, not pixel accuracy. Fix: use class-weighted cross-entropy or focal loss ($\gamma = 2$), and report mean IoU across classes.
3. Trusting pose without a reprojection check. A PnP solver with 3 keypoints returns a valid-looking pose that is actually mirrored or flipped. Why: 3 points are insufficient for a unique solution; the solver picks one arbitrarily. Detect: log reprojection RMSE per frame; anything above 5 px is suspicious. Fix: require at least 6 inlier keypoints and reject solutions with high RMSE.
4. Fixed NMS threshold in crowded scenes. As noted above, IoU 0.45 NMS merges adjacent people. Why: the threshold was tuned on sparse COCO images. Detect: measure recall on a crowded-scene validation set. Fix: use class-aware NMS at 0.65 for person class, or Soft-NMS with a Gaussian decay.
5. Forgetting the depth-camera-to-RGB extrinsic calibration. Pose from RGB is in camera frame. If the depth camera is offset by 2 cm and the extrinsic is wrong, the grasp point is off by 2 cm. Why: teams calibrate intrinsics but skip the stereo extrinsic. Detect: project a known checkerboard through both streams and measure offset. Fix: run a proper hand-eye or stereo calibration and store the transform in the robot’s TF tree.
Key Takeaways
- Match detector latency to your control loop. A 30 Hz loop needs inference under 25 ms. One-stage detectors (YOLOv8-s) hit this on Jetson Orin NX; two-stage detectors do not.
- Report per-class IoU and mAP@[.5:.95], never pixel accuracy. Class imbalance hides failures that pixel accuracy cannot reveal.
- Pose accuracy scales with range. Translation error is roughly $z \cdot \sigma_{px} / f$. At 2 m with 1-pixel noise and 600 px focal length, expect 3.3 mm — plan tolerances accordingly.
- Gate every perception output before the planner sees it. Reprojection RMSE, inlier count, and depth consistency checks catch most bad estimates for a few milliseconds of compute.
- Calibrate extrinsics, not just intrinsics. A 2 cm extrinsic error becomes a 2 cm grasp error. Store transforms in TF and validate with a checkerboard.
Tomorrow
We move from perception to decision-making: Reinforcement Learning Basics, where we formalize the Markov Decision Process and show why a robot arm learning to reach a target needs 10,000+ episodes without a good reward shape.