Week 13 — Deep Learning for Computer Vision in Robotics Day 7 of 7
The Week in One Paragraph
This week was about teaching a robot to understand what it sees. We started with the backbones — ResNet-50, ResNet-101, EfficientNet-B0 through B7 — and learned why a 50-layer residual network with skip connections trains where a 20-layer plain CNN fails, and why EfficientNet’s compound scaling (depth × width × resolution at fixed ratio) gives you more accuracy per FLOP than scaling one dimension. From there we moved to detection: YOLOv8’s anchor-free decoupled head and DETR’s set-prediction with Hungarian matching, two very different answers to “where is the object and what is it.” Day 3 covered segmentation — U-Net’s encoder-decoder with skip connections for semantic masks, Mask R-CNN’s RoIAlign for instance masks — the pixel-level output a mobile robot needs to distinguish a drivable floor from a rug from a cable. Day 4 tackled 6D pose: recovering rotation R ∈ SO(3) and translation t ∈ ℝ³ from a single RGB or RGB-D frame, using PnP on 2D-3D correspondences or direct regression, and why symmetric objects (a bowl, a cylinder) break naive loss functions. Day 5 explained attention: ViT’s patch tokens, DETR’s object queries, and why self-attention over 196 patches costs O(N²) and how windowed attention and token pruning bring that down. Day 6 was hands-on — we trained a small CNN in PyTorch to regress a quaternion and translation vector, and watched it fail on symmetric objects exactly as predicted. The through-line: every architecture choice is a trade between latency budget, accuracy, and how much labeled data you can actually get.
What Each Day Built
| Day | Topic | The one idea you must retain |
|---|---|---|
| 1 | CNN Architectures for Robotics | Residual connections let gradients flow through 50+ layers; a ResNet-50 at 25 ms/frame on a Jetson Orin is often the right accuracy/latency point, and EfficientNet-B0 gives ~same ImageNet top-1 (77.1% vs 76.1%) at roughly a third of the FLOPs when your compute is tight. |
| 2 | Object Detection: YOLO and DETR | YOLO predicts boxes in one forward pass with no proposal stage (fast, anchor-free since v8); DETR removes NMS entirely by predicting a fixed set of 100 queries and matching them to ground truth with the Hungarian algorithm — simpler pipeline, slower to converge (needs ~500 epochs vs ~300 for YOLO). |
| 3 | Semantic and Instance Segmentation | Semantic segmentation labels every pixel by class (floor, wall, person) and is what a navigation stack consumes; instance segmentation separates individual objects with per-instance masks via RoIAlign, which preserves spatial alignment that naive RoIPool quantization destroys. |
| 4 | 6D Object Pose Estimation | Pose = rotation R (3 DoF) + translation t (3 DoF); the practical pipeline is detect keypoints or a bounding box, then solve PnP with RANSAC, and the hard part is symmetry — a loss on R alone is ambiguous for a rotationally symmetric object, so you need a symmetry-aware loss or point-matching loss. |
| 5 | Vision Transformers for Robotics | Self-attention computes pairwise token interactions, so a 14×14 patch grid is 196 tokens and full attention is 196² ≈ 38k pairs per layer — fine at that size, expensive at higher resolution; DETR’s object queries are learned embeddings that attend to the image and each specialize to a region. |
| 6 | Python Practice: Pose Estimation with PyTorch | A CNN regressing a 4-vector quaternion + 3-vector translation needs quaternion normalization on the output and a loss that respects the double cover (q and −q are the same rotation); plain MSE on raw quaternions trains badly. |
Formula Cheat Sheet
| Formula | Symbols | Where it’s used |
|---|---|---|
| y = F(x, {Wᵢ}) + x | x input, F residual branch, Wᵢ layer weights | ResNet skip connection (Day 1); identity shortcut when dimensions match, 1×1 conv projection when they don’t |
| L = L_cls + λ₁·L_box + λ₂·L_GIoU | L_cls classification loss, L_box L1 box loss, L_GIoU generalized IoU loss, λ weights | YOLO/DETR detection training (Day 2); DETR uses λ₁=5, λ₂=2 in the original paper |
| IoU = |B_p ∩ B_g| / |B_p ∪ B_g| | B_p predicted box, B_g ground-truth box | Detection evaluation and NMS thresholding (Day 2); NMS typically at IoU 0.45–0.5 |
| L_mask = −(1/N)Σ[y·log ŷ + (1−y)·log(1−ŷ)] | y binary pixel label, ŷ predicted mask probability, N pixels | Per-pixel binary cross-entropy for segmentation masks (Day 3) |
| p_c = (R·P_w + t) → π(p_c) = K·p_c | R rotation, t translation, P_w world point, K camera intrinsics, π projection | 6D pose projection model (Day 4); the equation PnP inverts to recover R, t |
| R = R_z(γ)R_y(β)R_x(α) | α, β, γ Euler angles; R_x, R_y, R_z axis rotations | Pose parameterization (Day 4); note gimbal lock at β = ±90°, which is why quaternions are preferred |
| q = (w, x, y, z), |q| = 1 | q unit quaternion representing rotation | Pose regression output (Day 6); normalize after the network’s linear layer |
| Attention(Q,K,V) = softmax(QKᵀ/√d_k)·V | Q,K,V query/key/value matrices, d_k key dimension | ViT and DETR self-attention (Day 5); √d_k scaling keeps logits from saturating the softmax |
Method Selection Guide
- You need a backbone for a real-time robot and have a Jetson-class GPU (Orin, ~60 TOPS INT8): ResNet-18 or ResNet-50 for reliability and pretrained weights; EfficientNet-B0 if you are compute- or power-constrained and can afford the depthwise-conv latency quirks on some hardware.
- You need detection at 30+ FPS on embedded hardware: YOLOv8n or YOLOv8s. The one-stage, anchor-free design plus NMS is a known quantity, and the ecosystem (export to ONNX/TensorRT) is mature.
- You need detection and can tolerate 10–20 FPS but want no NMS tuning and better handling of crowded scenes: DETR or a DETR variant. Its set prediction avoids duplicate boxes by construction, but budget for long training and a transformer-friendly accelerator.
- You need to know what is floor vs. wall vs. obstacle for navigation: semantic segmentation (U-Net or DeepLabv3+ with a ResNet backbone). Feed the class mask straight into the costmap.
- You need to grasp or avoid a specific object: instance segmentation (Mask R-CNN) or, better, go straight to 6D pose — segmentation gives you the mask, pose gives you the grasp frame.
- You have RGB-D and the object’s 3D model: keypoint detection + PnP with RANSAC, or a direct pose regressor if the object is textureless and keypoints are unreliable.
- The object is symmetric: use a symmetry-aware loss or a point-matching (correspondence) approach; a plain rotation regression will converge to an average of equivalent poses.
- You have lots of data and compute but little task-specific tuning budget: a ViT backbone with pretraining. With small datasets (a few thousand images), a CNN with ImageNet weights will beat a from-scratch ViT.
Common Pitfalls (Learned the Hard Way)
| Pitfall | Why it happens | Fix |
|---|---|---|
| Training a deep plain CNN and watching accuracy degrade past ~20 layers | Vanishing gradients — the signal through many non-residual layers shrinks | Add residual connections (ResNet) or use a pretrained backbone; never train deep plain CNNs from scratch on robot-sized datasets |
| Detection model performs great on the validation set, fails in the lab | Domain gap: lighting, camera intrinsics, and backgrounds differ from the training set | Collect 500–1000 images from the actual robot camera and fine-tune; check intrinsics match between training and deployment |
| DETR trains for days and still produces poor boxes | The set-prediction loss needs many epochs to assign queries to objects; the learning rate schedule matters | Use the reference 500-epoch schedule or a DETR variant with denoising/deformable attention; don’t expect YOLO-speed convergence |
| Semantic mask looks clean but the robot drives through a glass door | Segmentation classes don’t include transparent or reflective surfaces, and the model confidently labels them “floor” | Add a depth-based or geometric check; treat low-confidence or depth-discontinuous regions as obstacles regardless of the mask |
| Pose regressor outputs a quaternion with norm 1.3 | The final linear layer is unconstrained and MSE doesn’t enforce unit norm | Normalize the quaternion on the output and use a geodesic or cosine loss on rotation, not raw MSE |
| Pose estimate flips 180° between frames on a symmetric object | Multiple rotations map to the same visible appearance; the loss is ambiguous | Use symmetry-aware losses or match predicted 3D points to the model; filter with temporal smoothing (a simple Kalman filter on pose helps) |
| ViT fine-tuned on 2,000 images underperforms a ResNet | Transformers lack the inductive bias (locality, translation equivariance) CNNs have, so they need more data | Use a CNN for small datasets, or use a pretrained ViT with strong augmentation and a low learning rate |
Self-Assessment
1. Why does a ResNet-50 train successfully when a 50-layer plain CNN does not? The residual connection adds the input to the block output, so the gradient has a direct path back through the identity shortcut. In a plain deep network the gradient must pass through every weight matrix, and repeated multiplication shrinks it. ResNet makes the block learn a residual F(x) = H(x) − x, which is easier to drive to zero when a layer is redundant.
2. What does DETR do that YOLO does not, and what does it cost? DETR predicts a fixed set of object queries and matches them to ground truth with the Hungarian algorithm, so it needs no non-maximum suppression and handles overlapping objects by construction. The cost is slow convergence (hundreds of epochs) and heavier compute per frame from the transformer decoder.
3. Why is semantic segmentation not enough for grasping? Semantic segmentation labels every pixel with a class but does not separate two instances of the same class, and it gives no orientation. To grasp you need to know which object is which and where its grasp frame is — that requires instance segmentation and 6D pose.
4. Write the projection equation that PnP inverts. p_c = R·P_w + t, then the pixel coordinate is u = f_x·(X_c/Z_c) + c_x, v = f_y·(Y_c/Z_c) + c_y, where (X_c, Y_c, Z_c) = R·P_w + t. PnP takes known 3D points P_w and their observed pixels (u, v) and solves for R and t.
5. Why does a plain MSE loss on quaternions fail, and what do you use instead? A quaternion q and its negation −q represent the same rotation, so MSE sees them as maximally different and the loss is inconsistent. Also, MSE does not enforce unit norm. Use a loss based on the angle between rotations — for example, the geodesic distance via 2·arccos(|⟨q₁, q₂⟩|) — or normalize the output and use a cosine-based loss.
Preview: Week 14
Next week we move from perception to action: Reinforcement Learning for Robot Control. We will cover Markov decision processes, Q-learning and policy gradients, and how PPO and SAC are used to train manipulation and locomotion policies in simulation before transferring to hardware. Expect the same mix of theory and PyTorch practice, this time with a robot arm learning to reach a target.