Week 13 — Deep Learning for Computer Vision in Robotics Day 2 of 7
Why This Matters
A warehouse AMR moving at 1.5 m/s needs to detect a pallet jack 8 m ahead. At that speed it covers 2.5 cm per millisecond. If your detector runs at 5 FPS, the object moves 30 cm between frames — enough to miss a person stepping out from behind a shelf. Detection latency and accuracy are not academic concerns; they decide whether the robot stops or collides.
The failure mode is concrete. A mobile manipulator with a 2-finger gripper rated for 5 kg tries to pick a box. If the detector returns a bounding box that is 15% too wide, the grasp planner computes a center offset of 3 cm on a 20 cm box. The gripper closes on air on one finger and the box tips. No amount of good inverse kinematics saves you. The perception stack fed the planner garbage.
Traditional detectors like HOG+SVM or Haar cascades break under the lighting and clutter of real warehouses. They need hand-tuned features. YOLO and DETR replaced feature engineering with learned representations and changed what robots can see. YOLO gives you 30–140 FPS on embedded hardware. DETR gives you set-based prediction with no NMS and better handling of overlapping objects. Knowing when to use each is the difference between a robot that works and one that stalls.
Core Concepts
YOLO: Single-Shot Grid Regression
YOLO (You Only Look Once, Redmon et al. 2016) reframes detection as regression. The image is divided into an $S \times S$ grid. Each cell predicts $B$ bounding boxes and $C$ class probabilities. For YOLOv1, $S=7$, $B=2$, $C=20$ on PASCAL VOC. The output tensor is $7 \times 7 \times (B \cdot 5 + C) = 7 \times 7 \times 30$.
Each box prediction has 5 values: $x, y, w, h, \text{confidence}$. The coordinates are offsets relative to the grid cell, normalized to $[0,1]$. Confidence is $\Pr(\text{object}) \cdot \text{IoU}_{\text{pred}}^{\text{truth}}$.
Modern YOLO (v5 through v11) uses anchor boxes and a feature pyramid. YOLOv8-s runs at 80 FPS on an RTX 3060 at 640×640 input and hits 44.9 mAP on COCO. YOLOv8-n hits 37.3 mAP at 160 FPS. On a Jetson Orin NX 16GB, YOLOv8-s with TensorRT INT8 runs around 45 FPS at 640×640 — enough for a 1.5 m/s AMR at 3 cm per frame.
The loss has three parts: box regression (CIoU in v5+), objectness (binary cross-entropy), and classification (BCE). The grid design means each cell is responsible for the object whose center falls inside it. That creates the “one object, one prediction” rule that eliminates NMS at inference only in the newer end-to-end variants.
DETR: Set Prediction with Transformers
DETR (Carion et al. 2020) treats detection as set prediction. A ResNet-50 backbone extracts features. A transformer encoder processes the flattened feature map. A transformer decoder takes $N$ learned object queries (default $N=100$) and outputs $N$ predictions in parallel. Each prediction is either a class + box or the special “no object” class.
The bipartite matching loss uses the Hungarian algorithm to pair each prediction with a ground-truth object. No NMS is needed because the loss encourages one-to-one assignment. This is the key architectural difference: YOLO emits thousands of candidate boxes and prunes them; DETR emits exactly 100 and each is a distinct object slot.
DETR’s weakness is convergence. It needs 500 epochs on COCO versus YOLO’s ~300, and it is slow at inference (about 28 FPS on a V100 with ResNet-50). Deformable DETR (Zhu et al. 2021) fixes this with sparse attention sampling 4 points per query per level instead of dense attention over all pixels. That drops training to 50 epochs and lifts small-object AP from 22.5 to 27.4 on COCO. RT-DETR (Zhao et al. 2023) reaches 53.1 AP at 108 FPS on a T4, competitive with YOLOv8-x.
| Model | mAP (COCO val) | FPS (T4, FP16) | NMS needed | Training epochs |
|---|---|---|---|---|
| YOLOv8-s | 44.9 | 130 | Yes | ~300 |
| YOLOv8-x | 53.9 | 28 | Yes | ~300 |
| DETR-R50 | 42.0 | 28 | No | 500 |
| Deformable DETR | 46.2 | 19 | No | 50 |
| RT-DETR-R50 | 53.1 | 108 | No | ~72 |
For robotics, the practical split: YOLO when you need max FPS on a Jetson and objects are well-separated. DETR-family when you have crowded scenes (people in a crowd, overlapping boxes on a shelf) and can afford 50–100 ms latency.
Mathematical Foundation
The YOLO box regression target for a cell at grid position $(c_x, c_y)$ with anchor $(p_w, p_h)$:
$$b_x = \sigma(t_x) + c_x, \quad b_y = \sigma(t_y) + c_y$$ $$b_w = p_w e^{t_w}, \quad b_h = p_h e^{t_h}$$
where $\sigma$ is the sigmoid, $t_x, t_y, t_w, t_h$ are raw network outputs, and $b_x, b_y, b_w, b_h$ are the predicted box center and size in feature-map units. The exponential keeps $b_w, b_h > 0$.
The DETR bipartite matching cost between prediction $i$ and ground truth $\sigma(i)$:
$$\hat{\sigma} = \arg\min_{\sigma \in \mathfrak{S}N} \sum{i=1}^{N} \mathcal{L}{\text{match}}(y_i, \hat{y}{\sigma(i)})$$
where $\mathfrak{S}_N$ is the set of all permutations of $N$ elements, $y_i$ is ground truth, $\hat{y}$ is prediction, and:
$$\mathcal{L}{\text{match}} = -\mathbb{1}{c_i \neq \varnothing} \hat{p}{\sigma(i)}(c_i) + \mathbb{1}{c_i \neq \varnothing} \mathcal{L}{\text{box}}(b_i, \hat{b}{\sigma(i)})$$
The $\mathbb{1}$ indicator is 1 when the ground-truth class $c_i$ is not “no object”. The box loss is a weighted sum of L1 and generalized IoU.
Worked example. Take YOLOv8-s at 640×640 input with stride 8 at the P3 level. Feature map is 80×80. A person’s center falls at pixel (312, 248). Grid cell is $(c_x, c_y) = (39, 31)$. Suppose the network outputs $t_x = 0.2$, $t_y = -0.1$, $t_w = 0.05$, $t_h = 0.3$ with anchor $p_w = 32, p_h = 64$ px. Then:
- $\sigma(0.2) = 0.550$, so $b_x = 0.550 + 39 = 39.550$ cells → 316.4 px
- $\sigma(-0.1) = 0.475$, so $b_y = 0.475 + 31 = 31.475$ cells → 251.8 px
- $b_w = 32 \cdot e^{0.05} = 33.64$ px
- $b_h = 64 \cdot e^{0.3} = 86.4$ px
The predicted box is centered at (316.4, 251.8) px with size 33.6×86.4 px — a plausible standing person at 8 m range.
Practical Example
Scenario: A shelf-scanning AMR with a front-facing 1920×1080 RGB camera on a Jetson Orin NX 16GB. It must detect misplaced boxes on shelf faces at 2–4 m range to trigger a re-shelving task. Boxes are 30×20×15 cm. Two boxes may sit side by side with a 2 cm gap. The robot travels at 1.0 m/s. Detection tolerance: box center within ±3 cm in image-space projection, since the arm’s reach envelope is 60 cm and gripper needs 2 cm clearance.
Step 1 — Compute required FPS. At 1.0 m/s, a 30 cm box passes a fixed camera point in 300 ms. To get 3 detections per box, you need 10 FPS minimum. Add margin for the re-shelve trigger logic: target 30 FPS. On Orin NX, YOLOv8-s INT8 at 640×640 gives ~45 FPS. Pass.
Step 2 — Check the 2 cm gap problem. Two boxes 2 cm apart at 2 m range with a 60° HFOV camera: 2 cm subtends ~1.7 pixels at 1920 width. YOLOv8-s at 640 input loses 3× resolution → 0.6 px. The two boxes will merge into one detection. This is the NMS failure case. Two options: run YOLO at 1280 input (drops to ~12 FPS, still acceptable), or switch to RT-DETR at 800 input which handles adjacent instances better via query separation.
Step 3 — Decide. At 1280 input, YOLOv8-s runs ~15 FPS on Orin NX INT8. That’s above the 10 FPS floor. Choose YOLOv8-s at 1280. Verify with a test set of 200 shelf images that adjacent-box recall exceeds 90%.
Step 4 — Latency budget. Camera capture 8 ms + preprocess 4 ms + inference 66 ms + NMS 3 ms + postprocess 2 ms = 83 ms. At 1.0 m/s that is 8.3 cm of travel per frame. Box position uncertainty from motion is under 1 cm after camera-to-base transform with 10 ms timestamp alignment. Within the ±3 cm tolerance.
Common Pitfalls
1. NMS threshold too aggressive on crowded scenes. Default NMS IoU is 0.45 in Ultralytics. Two adjacent boxes with 2 cm gap at 2 m range produce overlapping proposals with IoU around 0.5. The second box gets suppressed. Fix: raise NMS IoU to 0.6–0.7 for shelf/warehouse scenes, or use class-agnostic NMS with soft-NMS. Detect by plotting missed detections on a crowded test set.
2. Anchor mismatch after retraining. YOLOv5/v8 use k-means-derived anchors. If you fine-tune on a robot dataset where all boxes are 30×20 cm at 2–4 m, the default COCO anchors (which span person, car, bicycle sizes) are wrong. The model wastes capacity on irrelevant scales. Fix: recompute anchors on your dataset with yolo autanchor, or use anchor-free YOLOv8 which predicts distances directly.
3. DETR slow convergence mistaken for broken training. DETR needs 500 epochs and a low initial LR on the backbone (1e-5 vs 1e-4 on the transformer). Engineers see mAP flat at 5 after 50 epochs and kill the run. Fix: use Deformable DETR or RT-DETR if training budget is tight, or follow the DETR recipe exactly (AdamW, lr 1e-4 transformer, 1e-5 backbone, 500 epochs).
4. Class imbalance hiding rare objects. A warehouse dataset has 10,000 pallet boxes and 40 fire extinguishers. YOLO’s BCE loss averages over all anchors, so the extinguisher class gradient is drowned. Fix: use focal loss (YOLOv3+ supports it) or oversample rare classes. Detect by checking per-class AP — the aggregate mAP hides it.
5. Ignoring calibration between detection and 3D projection. A 2D box at pixel (316, 252) needs the camera intrinsics and the depth or ground-plane assumption to give a 3D position. If intrinsics are stale by 5% (camera bumped on a shelf), a 3 cm box-center error becomes 15 cm at 3 m range. Fix: recalibrate with a checkerboard after any mechanical shock, and monitor reprojection error weekly.
Key Takeaways
- YOLO gives you 30–140 FPS on embedded hardware with NMS; DETR-family gives you NMS-free set prediction with better crowded-scene handling at 20–100 FPS. Match the model to your latency budget and object density.
- Compute your FPS floor from robot speed and object size before picking a model. At 1.5 m/s and 30 cm objects, you need at least 10 FPS for 3 detections per object.
- Resolution is the lever for small or adjacent objects. Doubling input size from 640 to 1280 costs ~4× compute but resolves 2 cm gaps at 2 m range.
- Recompute anchors and check per-class AP after fine-tuning. Default COCO settings fail silently on warehouse-scale objects and rare classes.
- Calibrate camera intrinsics and monitor reprojection error. A 5% intrinsic error turns a 3 cm detection error into a 15 cm grasp failure at 3 m.
Tomorrow
Semantic and Instance Segmentation — pixel-level masks, Mask R-CNN vs. YOLOv8-seg, and how to use segmentation masks for grasp point selection on irregular objects.