Week 13 — Deep Learning for Computer Vision in Robotics Day 5 of 7
Why This Matters
A pick-and-place cell running a ResNet-50 detector on a 6-DOF arm can miss a 30 mm bolt at 1.2 m range when the bolt sits against a similar-colored conveyor. The CNN’s receptive field is local by construction. A 7×7 convolution at stride 2 sees 7 pixels; stacking 50 layers grows the theoretical receptive field, but the effective receptive field is much smaller and Gaussian-shaped. Objects that need long-range context to disambiguate — a bolt whose identity depends on the fixture 400 mm away — fall outside that effective window.
Vision Transformers (ViTs) fix this by making every patch attend to every other patch from layer one. On a 224×224 input with 16×16 patches, you get 196 tokens, and each self-attention layer computes a 196×196 attention matrix. That is 38,416 pairwise interactions per head, per layer, with no locality prior. The cost is quadratic in token count, which is exactly the problem robotics engineers hit: a 1280×720 industrial camera at 16×16 patches produces 3,600 tokens, and attention cost scales as 3,600² ≈ 13M per head.
The payoff is real. DETR (Carion et al., 2020) removed non-maximum suppression entirely by predicting a fixed set of object queries and matching them to ground truth with the Hungarian algorithm. For a bin-picking robot that previously spent 4 ms per frame in NMS with 300 overlapping boxes, that is a deterministic latency win. ViTs also transfer better across lighting shifts because attention weights adapt to content rather than fixed kernels. A ViT-B/16 pretrained on ImageNet and fine-tuned on 2,000 labeled frames from your cell often beats a from-scratch CNN trained on 20,000 frames.
This lesson covers the ViT patch embedding, multi-head self-attention, positional encodings, the DETR set-prediction loss, and how to deploy these on a robot with a 33 ms control loop.
Core Concepts
Patch Embedding and the Token Sequence
A ViT splits an image into non-overlapping patches. For input $H \times W = 224 \times 224$ and patch size $P = 16$, you get $N = (H/P) \times (W/P) = 14 \times 14 = 196$ patches. Each patch is flattened to $P^2 \cdot C = 16 \cdot 16 \cdot 3 = 768$ values and linearly projected to a model dimension $D = 768$. This is implemented as a single convolution with kernel 16, stride 16, and 768 output channels — identical math, faster on GPU.
A learnable [CLS] token is prepended, giving 197 tokens. The classification head reads the [CLS] output. For detection, DETR instead appends object queries.
| Model | Patch | Tokens (224²) | Layers | Heads | Params |
|---|---|---|---|---|---|
| ViT-B/16 | 16 | 196 | 12 | 12 | 86M |
| ViT-L/16 | 16 | 196 | 24 | 16 | 307M |
| DETR-R50 | — | 15×20=300 | 6 enc/6 dec | 8 | 41M |
| DeiT-S | 16 | 196 | 12 | 6 | 22M |
For a 1280×720 frame, ViT-B/16 yields 80×45 = 3,600 tokens. Attention memory for one head at float32 is $3600^2 \times 4$ bytes ≈ 52 MB. Twelve heads across 12 layers will not fit on a 8 GB Jetson Orin Nano at batch 1 without gradient checkpointing — but inference with torch.no_grad() and flash attention fits in about 2.1 GB.
Multi-Head Self-Attention
Self-attention computes, for each token, a weighted sum over all tokens. The weights come from dot products between queries and keys. With $D = 768$ and $h = 12$ heads, each head operates in $d_k = 64$ dimensions. The scaled dot product divides by $\sqrt{64} = 8$ to keep logits from saturating the softmax.
Concretely: token 0 (the [CLS]) produces a query vector. It dots against 197 key vectors, giving 197 scores. After softmax, suppose the top score is 0.31 on the patch containing the bolt and 0.02 on background patches. The output is a weighted mix — the [CLS] token now carries information from the bolt patch regardless of where it sits in the image. That is the long-range mechanism CNNs lack.
DETR Set Prediction
DETR outputs $N = 100$ object queries. Each query attends to the encoder’s image features through cross-attention and produces a class and a normalized box $(c_x, c_y, w, h) \in [0,1]^2$. Because there is no anchor and no NMS, the model must learn to assign exactly one query per object and push the rest to a “no object” class. The training loss uses the Hungarian algorithm to find the minimum-cost bipartite matching between the 100 predictions and the $M$ ground-truth objects (with $M \le 100$). Cost combines class probability, L1 box distance, and GIoU.
Mathematical Foundation
Patch embedding as convolution:
$$ X = \text{Conv2D}(I, W_p, b_p, \text{stride}=P) $$
$I \in \mathbb{R}^{H \times W \times C}$ is the image, $W_p \in \mathbb{R}^{D \times C \times P \times P}$ the projection weights, $X \in \mathbb{R}^{N \times D}$ the token matrix, $N = HW/P^2$.
Scaled dot-product attention:
$$ \text{Attention}(Q, K, V) = \text{softmax}!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V $$
$Q, K, V \in \mathbb{R}^{N \times d_k}$ are query, key, value projections; $d_k$ is per-head dimension; the softmax normalizes over the $N$ keys for each query row.
Multi-head output:
$$ \text{MHA}(X) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W_O $$
$h$ is head count, $W_O \in \mathbb{R}^{h d_k \times D}$ mixes heads back to model dimension.
Positional encoding (sine-cosine, 2D):
$$ PE_{(pos, 2i)} = \sin!\left(\frac{pos}{10000^{2i/D}}\right), \quad PE_{(pos, 2i+1)} = \cos!\left(\frac{pos}{10000^{2i/D}}\right) $$
$pos$ is the flattened patch index, $i$ the dimension index.
Worked example. Take $N = 196$, $d_k = 64$, one head. The score matrix $QK^\top$ has shape $196 \times 196$. For a single query row, the raw scores might be $[3.2, -1.1, 0.4, \dots]$ across 196 keys. Divide by $\sqrt{64} = 8$: $[0.40, -0.14, 0.05, \dots]$. The softmax over these gives roughly $[0.031, 0.018, 0.021, \dots]$ — a fairly flat distribution. If one score were 16.0 instead of 3.2, dividing by 8 gives 2.0, and the softmax weight jumps to about 0.31. That is why the $\sqrt{d_k}$ scaling matters: without it, large dot products saturate the softmax and gradients vanish.
DETR Hungarian matching cost for prediction $i$ and ground truth $j$:
$$ \mathcal{L}{\text{match}}(y_i, \hat{y}{\sigma(i)}) = -\mathbb{1}{c_i \neq \varnothing}, \hat{p}{\sigma(i)}(c_i) + \mathbb{1}{c_i \neq \varnothing}, \mathcal{L}{\text{box}}(b_i, \hat{b}_{\sigma(i)}) $$
$\sigma$ is the permutation found by Hungarian, $c_i$ the true class, $\hat{p}$ the predicted probability, $b$ the box. The box term is $\lambda_{\text{L1}}|b - \hat{b}|1 + \lambda{\text{giou}}\mathcal{L}{\text{giou}}$, with $\lambda{\text{L1}} = 5$, $\lambda_{\text{giou}} = 2$ in the original paper.
Practical Example
You run a 6-axis arm with a 2-finger gripper doing bin picking from a 400×300 mm bin. The overhead camera is a 1280×720 Basler acA1300-60gm at 60 fps. The arm’s control loop runs at 30 Hz (33 ms budget). You need to detect 12 part types plus a “no object” class, and the parts include 8 mm diameter washers that overlap.
Step 1 — Choose the backbone. ViT-B/16 on 1280×720 gives 3,600 tokens. On the Jetson Orin Nano (67 TOPS INT8), a quantized ViT-B/16 encoder runs at about 22 ms. That leaves 11 ms for the decoder and post-processing, which is tight but feasible. DETR-R50 with a ResNet-50 backbone at 1280×720 (after resizing to 800×600) runs at about 18 ms. The ResNet option is faster but loses the washer disambiguation because overlapping washers need global context.
Step 2 — Set the query count. You have at most 40 visible parts in the bin. Use $N = 100$ queries as in the paper. Fewer queries (say 50) speed up the decoder by 40% but risk missing parts in dense clusters.
Step 3 — Handle the small-object problem. At 16×16 patches, an 8 mm washer at 1.2 m range with a 60° FOV projects to roughly 14 pixels — smaller than one patch. It vanishes. Fix: use a 4×4 patch ViT (ViT-B/4) or feed a 2× upsampled crop around the bin. ViT-B/4 on a 640×480 crop gives 120×90 = 10,800 tokens, which is too many. The practical answer is a hybrid: a small CNN stem (4 conv layers, stride 4) reduces 1280×720 to 320×180, then 8×8 patches give 40×22 = 880 tokens. This is the pattern used in several industrial DETR variants.
Step 4 — Calibrate the matching. With 12 classes and heavy class imbalance (washers are 60% of parts), the Hungarian loss will underweight rare parts. Add a class-balanced sampler and monitor per-class AP. Set the “no object” class weight to 0.1 as in the paper.
Step 5 — Latency budget table.
| Stage | Time (ms) | Notes |
|---|---|---|
| Image capture + resize | 2.0 | USB3, 1280→800 |
| CNN stem (4 conv) | 5.5 | INT8 |
| ViT encoder (880 tokens) | 9.0 | Flash attention |
| DETR decoder (100 queries) | 6.0 | 6 layers |
| Box decode + threshold | 1.5 | CPU |
| Total | 24.0 | Fits in 33 ms |
The remaining 9 ms covers motion planning and the gripper command. Without NMS, the latency is deterministic — no frame-to-frame jitter from variable box overlap. That determinism matters for a 30 Hz loop.
Common Pitfalls
1. Forgetting positional encodings on the 2D grid. A ViT without 2D positional encoding treats the image as a bag of patches. Two identical parts at different locations produce identical [CLS] outputs. Why: self-attention is permutation-invariant. Detect it by shuffling patch order in a test — output should change; if it does not, your encoding is missing or zeroed. Fix: use learned 2D encodings or the sine-cosine 2D variant, and verify with an ablation.
2. Attention map memory blowup at high resolution. A 1280×720 frame at 16×16 patches costs 52 MB per head in float32. Twelve heads × 12 layers = 7.5 GB if materialized. Why: naive implementations store the full $N \times N$ matrix. Detect it with nvidia-smi during a forward pass — memory spikes then OOMs. Fix: use torch.nn.functional.scaled_dot_product_attention (flash attention path), which never materializes the full matrix, or reduce tokens with a CNN stem.
3. DETR slow convergence. DETR needs 500 epochs on COCO versus 100 for Faster R-CNN. Why: the Hungarian matching is unstable early in training because random queries produce random boxes, and the matching jumps between assignments. Detect it by plotting matching cost variance per epoch — if it stays high past epoch 50, you have the problem. Fix: use Deformable DETR or DAB-DETR, which converge in ~50 epochs by using sparse attention and anchor boxes as query priors.
4. Treating DETR box outputs as pixel coordinates. DETR outputs normalized $(c_x, c_y, w, h) \in [0,1]$. Engineers who multiply by image width get boxes that are systematically off when the image was resized with letterboxing. Why: the normalization is relative to the padded input, not the original. Detect it by drawing boxes on the raw frame — they will be shifted. Fix: invert the exact resize transform (scale and pad offsets) before converting to robot-frame coordinates.
5. Ignoring the “no object” class in deployment. DETR always outputs 100 boxes. If you threshold at 0.5 on class probability, you may still get 15 boxes on an empty bin because the model is uncertain. Why: the no-object class is learned but not calibrated. Detect it by running on 200 empty-bin frames and counting false positives. Fix: calibrate a temperature on a held-out set, or add a confidence threshold tuned to a target false-positive rate (e.g., 1 per 1,000 frames).
Key Takeaways
- ViT replaces local convolution with global self-attention; at 224×224 with 16×16 patches, each token attends to all 196 others from layer one. This is what lets it disambiguate objects that need long-range context.
- Attention cost is quadratic in token count. A 1280×720 frame at 16×16 patches gives 3,600 tokens and 52 MB per head. Use a CNN stem or flash attention to stay within a 33 ms loop.
- DETR removes NMS by predicting a fixed set of queries and matching with the Hungarian algorithm. Latency becomes deterministic, which matters for real-time control.
- Small objects smaller than one patch vanish. At 16×16 patches, an 8 mm washer at 1.2 m projects to ~14 pixels. Use smaller patches or a CNN stem for fine parts.
- Positional encodings are mandatory. Without them, the model is permutation-invariant and cannot localize. Verify with a patch-shuffle test.
Tomorrow
Tomorrow we put pose estimation into practice: building a 6D pose estimator in PyTorch, from dataset loading through the Ferrari-Canny metric and a live inference loop on a Jetson.