Week 13 — Deep Learning for Computer Vision in Robotics Day 1 of 7

Why This Matters

A 6-axis arm on a pick-and-place line has to find a 4 cm gearbox housing inside a tote. The camera is a 1280×720 Basler ace running at 45 fps. The compute box is a Jetson Orin NX 16 GB with a 100 TOPS budget shared with motion control. If your CNN backbone takes 40 ms per frame, you have already lost the cycle. If it takes 8 ms but confuses a gearbox with a bearing cap under a 30° specular glare, the gripper closes on air and the line stops. Both failures trace back to the same decision: which backbone, at which resolution, at which depth.

Backbone choice is where robotics vision projects quietly succeed or fail. A ResNet-50 trained on ImageNet gives you a strong feature extractor for free. It also costs roughly 4.1 GFLOPs at 224×224 and about 25.6M parameters. EfficientNet-B0 costs 0.39 GFLOPs and 5.3M parameters for the same input, but its depthwise convolutions are memory-bandwidth bound on embedded GPUs. On an Orin NX, EfficientNet-B0 often runs slower than its FLOP count suggests because the tensor cores sit idle on 3×3 depthwise kernels. That gap between theoretical and measured cost is the core engineering problem this lesson addresses.

Robotics adds constraints generic vision benchmarks ignore. Latency must be bounded, not just average. A backbone with 12 ms mean and 30 ms p99 jitter breaks a 50 Hz control loop. Input resolution is fixed by the sensor and the smallest object you must resolve. Domain shift is severe: warehouse lighting, motion blur from a moving base, and lens dirt all push features off the ImageNet manifold. Getting the backbone right means matching architecture family, receptive field, and compute profile to the physical task.

Core Concepts

ResNet: Residual Learning and the Identity Shortcut

ResNet (He et al., 2015) solved the degradation problem: deeper plain networks got worse training error, not just worse test error. The fix is the residual block. Instead of learning a mapping $H(x)$, the block learns $F(x) = H(x) - x$ and adds the input back through an identity shortcut. Gradients flow through the shortcut without passing every weight matrix, so a 50-layer net trains as readily as a 20-layer one.

For robotics, the practical consequences are concrete:

ModelParamsGFLOPs (224²)Top-1 (ImageNet)Typical Orin NX FP16 latency*
ResNet-1811.7M1.869.8%~4 ms
ResNet-3421.8M3.773.3%~7 ms
ResNet-5025.6M4.176.1%~9 ms
ResNet-10144.5M7.877.4%~16 ms

*Batch 1, TensorRT, indicative only — measure on your own board.

ResNet-18 is the workhorse for real-time detection backbones on edge robots. ResNet-50 is the default when you fine-tune a pretrained detector and can afford ~9 ms. The bottleneck blocks in ResNet-50 use 1×1 → 3×3 → 1×1 convolutions, which map well to GPU tensor cores. That is why ResNet-50 often beats EfficientNet-B0 on latency despite 10× the FLOPs.

One robotics-specific detail: the first conv is 7×7 stride 2, then a 3×3 max pool. That gives a 32× downsampling before the first residual stage. A 4 cm object at 1 m with a 6 mm lens on a 1/1.8” sensor spans about 60 pixels. After 32× downsampling it is under 2 pixels — effectively gone. For small-object tasks you must either crop, increase resolution, or replace the stem with a stride-1 3×3 conv (as in some detection backbones).

EfficientNet: Compound Scaling and Mobile Inverted Bottlenecks

EfficientNet (Tan & Le, 2019) scales depth, width, and input resolution together with a single compound coefficient $\phi$. The scaling rule is:

$$\text{depth} = \alpha^\phi,\quad \text{width} = \beta^\phi,\quad \text{resolution} = \gamma^\phi,\quad \text{s.t. } \alpha\beta^2\gamma^2 \approx 2$$

with $\alpha=1.2$, $\beta=1.1$, $\gamma=1.15$ found by grid search on the baseline B0. B0 has 5.3M params and 0.39 GFLOPs; B7 has 66M params and 37 GFLOPs at 600² input. The family spans a 95× compute range, which is exactly what you want when one product ships on a $99 module and another on an Orin AGX.

The building block is the MBConv (mobile inverted bottleneck): a 1×1 expansion conv (ratio 6), a 3×3 or 5×5 depthwise conv, a squeeze-and-excitation block, and a 1×1 projection. Depthwise convs cut FLOPs by roughly 8–9× versus a standard 3×3 at the same channel count. But they have low arithmetic intensity. On a Jetson, the depthwise layer is limited by memory bandwidth, not math. Measured throughput can be 2–3× worse than the FLOP ratio predicts.

EfficientNet also adds stochastic depth and a SiLU (swish) activation. SiLU is more expensive than ReLU on some embedded accelerators. Check that your TensorRT or TFLite backend fuses it; otherwise you pay a separate kernel launch per activation.

Task-Specific Backbones

Generic backbones are pretrained on 1000 ImageNet classes. Robotics tasks have different statistics: textureless industrial parts, low light, fisheye distortion. Three task-specific options matter:

The rule of thumb: pick the smallest backbone whose stride-8 feature map still resolves your smallest object, then verify p99 latency on the target board.

Mathematical Foundation

A standard convolution layer computes, for output channel $k$ at spatial location $(i,j)$:

$$y_{k}(i,j) = \sum_{c=1}^{C_{in}} \sum_{u=-r}^{r} \sum_{v=-r}^{r} w_{k,c}(u,v), x_{c}(i+u, j+v) + b_k$$

where $C_{in}$ is input channels, $r$ is the kernel radius (1 for 3×3), $w$ are weights, $x$ is the input feature map, and $b_k$ is a bias. Its cost is:

$$\text{FLOPs} = 2 \cdot H \cdot W \cdot C_{out} \cdot C_{in} \cdot (2r+1)^2$$

A depthwise conv sets $C_{out}=C_{in}=C$ and uses one filter per channel, so the cost drops to:

$$\text{FLOPs}_{dw} = 2 \cdot H \cdot W \cdot C \cdot (2r+1)^2$$

The ratio is $C_{in} \cdot C_{out} / (C_{in}+C_{out})$ for a pointwise-paired block — for $C=256$ that is roughly 128× fewer multiplies, but the memory traffic for reading and writing the full tensor is unchanged. That is the bandwidth wall.

Worked example. Take a 3×3 conv, $H=W=56$, $C_{in}=64$, $C_{out}=128$:

$$\text{FLOPs} = 2 \cdot 56 \cdot 56 \cdot 128 \cdot 64 \cdot 9 = 462{,}422{,}400 \approx 0.46\ \text{GFLOP}$$

Now a depthwise 3×3 at the same spatial size with $C=128$:

$$\text{FLOPs}_{dw} = 2 \cdot 56 \cdot 56 \cdot 128 \cdot 9 = 7{,}225{,}344 \approx 7.2\ \text{MFLOP}$$

The depthwise layer does 64× fewer FLOPs. But it still reads and writes $56 \cdot 56 \cdot 128 \cdot 2$ bytes (FP16) = 802 KB per pass. At an Orin NX LPDDR5 bandwidth of ~102 GB/s, that is about 7.9 µs of pure memory time — versus ~0.6 µs of math at 100 TOPS. The layer is memory bound by ~13×. Stack 16 of them and you have burned 126 µs on a layer that “should” take 10 µs. This is why FLOP counts mislead on edge hardware.

For the residual block, the forward pass is:

$$y = \mathcal{F}(x, {W_i}) + x$$

and backpropagation gives:

$$\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \mathcal{L}}{\partial y}\left(1 + \frac{\partial \mathcal{F}}{\partial x}\right)$$

The $+1$ term is the identity gradient. It guarantees the gradient never vanishes entirely, even if $\partial \mathcal{F}/\partial x \to 0$. That single term is why 50+ layer nets train on datasets as small as a few thousand labeled frames — common in industrial robotics.

Practical Example

Scenario. A UR10e arm with a 2-finger Robotiq 2F-85 gripper picks 30 mm aluminum bearing caps from a bin. The camera is a 5 MP Basler acA2440-75um mounted 800 mm above the bin, 2448×2048 pixels, 75 fps. The compute is a Jetson Orin NX 16 GB. The control loop runs at 30 Hz, so the full perception pipeline (backbone + head + post-processing) must finish in under 33 ms at p99. Gripper tolerance is ±1.5 mm in the image plane, which at 800 mm and a 16 mm lens works out to about ±3.7 pixels.

Step 1 — object size on sensor. Bearing cap is 30 mm. At 800 mm with a 16 mm lens on a 1/1.2” sensor, the magnification is $800/16 = 50$, so the cap projects to $30/50 = 0.6$ mm on the sensor. With a pixel pitch of 5.86 µm, that is about 102 pixels. Good — plenty of resolution.

Step 2 — backbone receptive field. To distinguish a bearing cap from a similar 28 mm washer, the network needs context. A ResNet-50 at stride 32 has a theoretical receptive field of ~483 px. That covers roughly 5 caps — enough context. ResNet-18’s effective receptive field is smaller (~200 px at the final stage) but the stride-8 features still resolve the 102 px object. ResNet-18 is the better latency choice.

Step 3 — latency budget. ResNet-18 as a detection backbone at 640×640 input: ~4 ms backbone, ~3 ms FPN neck, ~4 ms head, ~2 ms NMS. Total ~13 ms at p50, ~22 ms at p99 on TensorRT FP16. That leaves 11 ms of headroom for the arm planner. EfficientNet-B0 at the same resolution: ~0.4 GFLOPs backbone, but measured ~11 ms on the same board due to depthwise bandwidth limits — worse p99 jitter, no benefit.

Step 4 — validation. Run 500 bin images through both. ResNet-18 backbone with a YOLO-style head hits 97.2% mAP@0.5 on the cap class. EfficientNet-B0 hits 97.8% but with 1.7× the p99 latency. Pick ResNet-18. The 0.6% accuracy difference does not justify missing the 33 ms deadline on 1 in 200 frames.

Step 5 — domain check. The line has 850 lux overhead LED plus specular glare from the caps. Fine-tune the last two stages of the ResNet-18 on 300 labeled frames from the actual cell. Freeze the first two stages. This recovers ~4% mAP lost to domain shift at a cost of 20 minutes of training.

Common Pitfalls

1. Choosing backbones by FLOP count. EfficientNet-B0 has 10× fewer FLOPs than ResNet-50 but often runs slower on Jetson-class hardware because depthwise convs are memory bound. Why: the FLOP metric ignores arithmetic intensity and kernel launch overhead. Detect: profile with trtexec --avgRuns=100 on the actual board, not a desktop GPU. Fix: benchmark the top-3 candidates end-to-end and pick by p99 latency, not GFLOPs.

2. Ignoring the stem downsampling. ResNet’s 7×7 stride-2 conv plus max pool gives 32× downsampling before stage 1. Small objects vanish. Why: the classifier head was designed for 224² ImageNet images, not 5 MP industrial frames. Detect: visualize stride-8 feature maps and check whether your smallest object produces a non-zero activation. Fix: use a stride-2 or stride-4 stem, or feed higher-resolution crops.

3. Fine-tuning the whole backbone on 500 images. You overfit the first stages and destroy pretrained features. Why: early conv layers encode generic edge and texture filters that transfer well; 500 samples cannot re-learn them. Detect: training loss drops while validation mAP stalls or falls after epoch 3. Fix: freeze stages 1–2, fine-tune stages 3–4 with a 10× lower learning rate than the head.

4. Forgetting batch-norm statistics under domain shift. A ResNet trained on ImageNet has BN running means tuned to natural images. Your warehouse line has different mean intensity and contrast. Why: BN normalizes using stale statistics at inference. Detect: compare feature-map histograms on line images vs. ImageNet samples. Fix: either fine-tune BN layers (unfreeze them for the last 2 epochs) or recalibrate running stats by passing 200 unlabeled line images through in train mode.

5. Treating p50 latency as the deadline. A backbone with 12 ms mean and 30 ms p99 breaks a 30 Hz loop. Why: embedded GPUs share memory bandwidth with the CPU and other processes; contention spikes. Detect: log per-frame inference time and plot the distribution, not just the mean. Fix: size the model so p99 fits the budget, and pin the inference thread to dedicated cores.

Key Takeaways

  1. Match backbone to hardware, not to papers. ResNet-18/50 win on Jetson-class GPUs because their 1×1 and 3×3 convs map to tensor cores. EfficientNet wins on mobile NPUs with dedicated depthwise units. Measure p99, not p50.
  2. Small objects need stride-8 features and a shallow stem. ResNet’s 32× stem downsampling kills 4 cm parts. Use crops, higher input resolution, or a modified stem.
  3. Residual shortcuts are why transfer learning works at 500-image scale. The identity gradient keeps early features stable while you fine-tune the head and late stages.
  4. Budget latency end-to-end. Backbone + neck + head + NMS must fit the control period at p99. A 4 ms backbone is useless if the FPN neck adds 20 ms.
  5. Domain-pretrained beats ImageNet-pretrained. When you have unlabeled line images, self-supervised pretraining on them recovers more accuracy than any generic checkpoint.

Tomorrow

We move from feature extraction to localization: how YOLO’s anchor-free heads and DETR’s set-prediction transformer decoder turn a backbone into bounding boxes, and when each wins on a moving robot.