Week 13 — Deep Learning for Computer Vision in Robotics Day 3 of 7

Why This Matters

A warehouse AMR moving at 1.2 m/s has roughly 400 ms of reaction time before it covers the 0.5 m gap needed to stop safely. Object detection gives you axis-aligned boxes around pallets and people. Boxes lie. A tilted pallet at 30° produces a bounding box that overlaps 40% of the drivable aisle. A human leg poking out from behind a shelf produces a box centered on empty floor. The robot brakes for phantom obstacles or, worse, drives through the real one because the box centroid sat 0.8 m to the left of the actual leg.

Semantic segmentation labels every pixel with a class: floor, wall, human, pallet, shelf. Instance segmentation goes further and separates individual objects of the same class — person #1 versus person #2, even when they touch. For navigation, this is the difference between a 2D costmap that reflects geometry and one that reflects free space. A 640×480 mask at 30 FPS on an Orin NX (100 TOPS INT8) is now realistic with models like Mask2Former or YOLOv8-seg. The output feeds directly into nav2_costmap_2d as an observation layer, or into a 3D voxel grid via frustum projection.

Without pixel-level masks, your planner treats every bounding box as a solid rectangle. In a 6 m wide aisle with two pallets 2.1 m apart, box-based obstacles often close the gap entirely. Segmentation keeps the 2.1 m corridor open because it knows the pixels between pallets are floor. That single fact recovers roughly 30% of blocked paths in dense warehouse layouts, based on published AMR deployment reports.

Core Concepts

Semantic segmentation: one label per pixel

A semantic segmentation network outputs a tensor of shape $H \times W \times C$, where $C$ is the number of classes. For each pixel $(i,j)$ you take the argmax over the $C$ logits. Common robotics class sets are small — 8 to 20 classes — because every extra class costs labeled data and inference time.

The dominant architecture family is the encoder-decoder with skip connections. DeepLabv3+ uses an atrous spatial pyramid pooling (ASPP) head with dilation rates of 6, 12, and 18 on top of a ResNet-50 backbone. On a 512×512 Cityscapes crop, ResNet-50 ASPP runs at about 22 ms on a Jetson AGX Orin at FP16. The dilated convolutions keep the receptive field large without downsampling, which matters because a 3×3 conv with dilation 18 sees a 37×37 effective window — enough context to distinguish “floor” from “wall baseboard” at range.

The loss is per-pixel cross-entropy, usually with class weights to fight imbalance. Floor pixels can be 60% of a warehouse frame; humans might be 0.3%. Without weighting, the network predicts “floor” everywhere and reaches 60% pixel accuracy while being useless. Median frequency balancing (weights inversely proportional to class frequency) and Dice loss are standard fixes.

A concrete worked number: a 1280×720 frame has 921,600 pixels. If humans occupy 0.3%, that is 2,765 pixels — roughly one person at 8 m range. A well-trained model at 78% mIoU on the human class will mislabel about 600 of those pixels. Those 600 pixels are exactly the ones near the silhouette edge, which is where the costmap inflation boundary lives. This is why boundary accuracy (measured with the Boundary IoU metric) matters more than global mIoU for safety.

Instance segmentation: separating touching objects

Semantic masks merge two adjacent people into one blob. Instance segmentation assigns a unique ID per object. Two families dominate:

Proposal-based (Mask R-CNN). A region proposal network (RPN) generates ~1000 candidate boxes, RoIAlign samples 7×7 features per box, and a mask head predicts a 28×28 binary mask per instance, upsampled to the box. Mask R-CNN with ResNet-50-FPN hits 37.1 mask AP on COCO at about 80 ms per 800×800 image on an Orin NX. It is accurate but slow for a 30 Hz control loop.

Query-based (Mask2Former). A transformer decoder with $N=100$ learnable queries attends to multi-scale features and outputs 100 mask predictions plus class scores in one shot. Mask2Former with a Swin-T backbone reaches 47.2 mask AP on COCO at roughly 45 ms on an Orin NX at FP16. The Hungarian matching loss assigns each query to a ground-truth instance, so no NMS is needed.

Real-time single-stage (YOLOv8-seg). YOLOv8m-seg reaches 40.6 mask AP on COCO at about 12 ms on an Orin NX. It uses a prototype mask branch: the network predicts 32 prototype masks at 160×160, and each detection predicts 32 coefficients. The final mask is a linear combination:

$$M_k = \sigma\left(\sum_{c=1}^{32} a_{k,c} \cdot P_c\right)$$

where $P_c$ is prototype $c$, $a_{k,c}$ is the coefficient for detection $k$, and $\sigma$ is sigmoid. This keeps the mask head cheap — 32×160×160 floats is 819 KB per frame.

For a robot at 1.2 m/s, YOLOv8m-seg at 12 ms leaves 388 ms for the rest of the stack. Mask2Former at 45 ms leaves 355 ms. Both are viable; Mask R-CNN at 80 ms is borderline.

Mathematical Foundation

The core training objective for instance segmentation combines classification, box regression, and mask loss. For Mask R-CNN, the multi-task loss per RoI is:

$$L = L_{cls} + \lambda_{box} L_{box} + \lambda_{mask} L_{mask}$$

Here $L_{cls}$ is cross-entropy over $K+1$ classes (including background), $L_{box}$ is a smooth-L1 loss on the 4 box deltas, and $L_{mask}$ is a per-pixel binary cross-entropy over the 28×28 mask, averaged only over the ground-truth class. The standard weights are $\lambda_{box}=1$ and $\lambda_{mask}=1$.

The mask loss term expands to:

$$L_{mask} = -\frac{1}{28^2}\sum_{i=1}^{28}\sum_{j=1}^{28}\left[ y_{ij}\log \hat{y}{ij} + (1-y{ij})\log(1-\hat{y}_{ij}) \right]$$

where $y_{ij}\in{0,1}$ is the ground-truth mask pixel and $\hat{y}_{ij}\in(0,1)$ is the predicted probability.

For the query-based family, the matching cost between prediction $i$ and ground truth $j$ is:

$$C_{ij} = -\hat{p}i(c_j) + L{box}(b_i, b_j) + L_{mask}(m_i, m_j)$$

The Hungarian algorithm (a special case of the assignment problem solved in $O(N^3)$, with $N=100$ here) finds the permutation $\hat{\sigma}$ minimizing total cost. With $N=100$ queries, that is 1,000,000 cost evaluations per image — trivial on GPU.

Worked numeric example. Take a 640×480 frame with one person at 6 m. The person occupies a box of 60×140 pixels and a true mask of 5,800 pixels. Suppose the predicted mask has 5,400 true positives, 400 false positives, and 400 false negatives. Then:

An IoU of 0.871 passes the standard COCO threshold of 0.5 and even 0.75. But note the 400 false-negative pixels: if those are on the leading edge of the person (the side facing the robot), the costmap will under-estimate the person’s extent by roughly 3 pixels. At 6 m with a 60° horizontal FOV on a 640-pixel sensor, one pixel spans about 10 mm laterally. Three pixels is 30 mm — below the 50 mm costmap inflation radius, so acceptable. At 12 m, one pixel spans 20 mm and the same error becomes 60 mm, which exceeds inflation. This is why range-dependent mask accuracy matters.

Practical Example

Scenario: A 250 kg payload AMR in a cold-storage warehouse, aisle width 2.4 m, max speed 1.5 m/s, safety stop distance 0.4 m. Sensors: one front-facing 1920×1080 RGB camera with 70° HFOV, mounted at 0.9 m height, tilted 15° down. Compute: Jetson Orin NX 16GB.

Task: Build a segmentation-based costmap layer that distinguishes drivable floor from pallets, shelving, and humans.

Step 1 — Model selection. We need ≥20 FPS end-to-end. YOLOv8m-seg at 1920×1080 needs downscaling to 640×640, giving about 14 ms inference. Mask2Former-SwinT at 640×640 takes 45 ms. We pick YOLOv8m-seg for the 30 Hz budget, with Mask2Former as an offline validation oracle.

Step 2 — Class set. Eight classes: floor, shelf, pallet, human, forklift, wall, ceiling, other. Human gets a 5× class weight in the loss because it appears in under 1% of training pixels.

Step 3 — Frustum projection. For each mask pixel $(u,v)$ classified as obstacle, back-project to 3D using the camera intrinsics:

$$Z = \frac{f \cdot h}{v - c_y}, \quad X = \frac{(u - c_x)Z}{f}, \quad Y = \frac{(v - c_y)Z}{f} + h$$

with $f = 1100$ px, $c_x = 960$, $c_y = 540$, $h = 0.9$ m, and ground-plane assumption. A pixel at $v = 700$ gives $Z = 1100 \cdot 0.9 / 160 = 6.19$ m. A pixel at $v = 900$ gives $Z = 1100 \cdot 0.9 / 360 = 2.75$ m.

Step 4 — Costmap insertion. Obstacle points go into a 5 cm resolution local costmap as lethal cells. Floor points are marked free. Cells within 0.35 m of a lethal cell get inflations of 253 (inscribed radius). The Nav2 ObstacleLayer is replaced by a custom SegmentationLayer that subscribes to /segmentation/mask and /segmentation/instances.

Step 5 — Validation. Run 200 logged frames from the aisle. Ground truth from a FARO scanner. Measure: false-negative rate on humans (must be < 0.5% of person pixels), false-positive rate on floor (must be < 2%). The false-positive rate matters because every spurious obstacle cell forces a replan; above 2% the planner thrashes and the robot slows to 0.6 m/s average.

Step 6 — Failure mode. A pallet wrapped in translucent shrink-wrap confuses the model — it classifies 30% of the wrap as floor because the training set had few such examples. Fix: add 500 augmented frames with synthetic wrap texture, retrain for 8 epochs, and add a temporal consistency check that rejects mask flips faster than 200 ms.

Common Pitfalls

1. Treating semantic masks as instance masks. Two people standing shoulder-to-shoulder produce one semantic blob. The costmap marks the union as lethal — fine — but the planner cannot track either person, so it cannot predict motion. Detect: count distinct connected components per frame; if humans always appear as one component, you have semantic-only. Fix: run instance segmentation on the human class, or add a lightweight tracker (ByteTrack) on top of semantic masks.

2. Ignoring class imbalance in the loss. Floor dominates at 60% of pixels. The model converges to “all floor” and reports 60% pixel accuracy. Detect: per-class IoU on the validation set. If human IoU is under 0.3 while floor IoU is 0.95, you have imbalance. Fix: median frequency balancing or focal loss with $\gamma=2$.

3. Mask edge erosion from bilinear upsampling. A 28×28 mask upsampled to 200×200 loses 3–5 pixels of accuracy at the boundary. For a person at 10 m, that is 50–80 mm of under-estimated extent. Detect: measure Boundary IoU (should be within 5% of mask IoU). Fix: use a mask head with 4× transposed convolutions and add a boundary-aware loss term.

4. Domain shift from lighting and lens dirt. A model trained on daylight warehouse images drops 15–20 mIoU in cold-storage lighting (200 lux vs 800 lux). Detect: monitor the entropy of the softmax output; a jump above 2.0 nats per pixel indicates distribution shift. Fix: log frames, retrain quarterly, and add photometric augmentation (brightness ±40%, contrast ±30%, Gaussian noise).

5. Frustum projection with wrong camera pitch. A 2° pitch error at 10 m shifts the ground-plane estimate by 350 mm — enough to place a real obstacle outside the costmap. Detect: compare projected floor points to a known flat region; residuals should be under 20 mm. Fix: calibrate extrinsics with a checkerboard monthly and store per-robot.

6. Frame-rate mismatch between mask and odometry. If the mask arrives at 20 Hz and odometry at 100 Hz, the costmap insertion uses stale poses. A 50 ms lag at 1.5 m/s is 75 mm of smear. Detect: timestamp diff between /segmentation and /odom should stay under 30 ms. Fix: use message_filters with ApproximateTime and drop frames older than 100 ms.

Key Takeaways

  1. Semantic segmentation labels pixels; instance segmentation separates objects. For navigation costmaps, semantic is often enough for floor and walls, but humans and pallets need instance IDs for tracking.
  2. YOLOv8m-seg runs at 12 ms on an Orin NX and is the practical choice for 30 Hz control loops. Mask2Former is more accurate but 3–4× slower.
  3. Class imbalance will silently destroy your model. Always report per-class IoU, not pixel accuracy, and weight rare classes (humans, forklifts) by 5× or more.
  4. Boundary accuracy matters more than global mIoU for safety. A 400-pixel mask error on a human’s leading edge translates to 30–60 mm of costmap error depending on range.
  5. Frustum projection is only as good as your extrinsics. A 2° pitch error at 10 m is a 350 mm ground-plane shift — larger than your inflation radius.

Tomorrow

We move from 2D masks to full 6D object pose — recovering position and orientation of a known object from a single RGB-D frame, using methods like PVNet, FoundationPose, and iterative PnP refinement, with the accuracy targets your grasping pipeline actually needs.