Week 13 — Deep Learning for Computer Vision in Robotics Day 4 of 7
Why This Matters
A bin-picking robot at an automotive supplier must grab a cast aluminum housing from a jumbled tote. The part weighs 1.8 kg, the gripper is a two-finger parallel jaw with a 40 mm stroke, and the collision budget between finger and part is 3 mm. A 2D bounding box from YOLOv8 tells you where the part is in the image. It does not tell you which way the part is rotated about the camera axis. If the part is lying on its side and you approach from above with a vertical grasp, the fingers close on air and the cycle fails. At 900 cycles per hour, even a 4% grasp failure rate costs the cell 36 failed picks per hour and trips the safety PLC on jam detection.
This is the gap 6D object pose estimation closes. You need the full rigid transform between the object’s CAD frame and the camera frame: three translation values (x, y, z) and three rotation values (roll, pitch, yaw, or a quaternion). That is 6 degrees of freedom. With that transform, the robot can plan an approach vector, pre-rotate the wrist, and verify the grasp is collision-free before moving.
The problem is hard for specific engineering reasons. Metallic parts have specular highlights that move as the part rotates. Occlusion is severe in a tote — a typical bin-picking scene shows 30–50% of each part’s surface. Symmetric parts (washers, flanges, gears) have multiple valid poses that map to the same image, so the estimator must report ambiguity rather than a single confident answer. And the tolerance is unforgiving: a 5° rotation error on a 120 mm-long part displaces the far end by 10 mm, which exceeds the 3 mm collision budget.
Core Concepts
The Pose Representation and Why Rotation Is the Hard Part
A 6D pose is written as a rigid transform $T \in SE(3)$:
$$T = \begin{bmatrix} R & t \ 0 & 1 \end{bmatrix}$$
where $R$ is a $3\times3$ rotation matrix with $R^T R = I$ and $\det(R)=1$, and $t \in \mathbb{R}^3$ is the translation. The translation is easy — it is a regression target with a clean metric (millimeters). The rotation lives on the manifold $SO(3)$, which is not a vector space. You cannot average two rotation matrices and expect a valid rotation. You cannot subtract them and get a meaningful error. This is why pose estimation papers spend so much effort on rotation parameterizations.
Three common parameterizations and their failure modes:
| Parameterization | Dimension | Failure mode |
|---|---|---|
| Euler angles (roll-pitch-yaw) | 3 | Gimbal lock at pitch = ±90°; discontinuous |
| Quaternion | 4 | Double cover: $q$ and $-q$ are the same rotation |
| Rotation matrix (via SVD projection) | 9 → 3 | Needs orthogonalization; gradient is messy |
| 6D continuous (Zhou et al., CVPR 2019) | 6 | None of the above; preferred for regression |
The 6D continuous representation from “On the Continuity of Rotation Representations in Neural Networks” takes the first two columns of $R$ as a 6-vector, then Gram-Schmidt orthogonalizes them at inference. This avoids the discontinuity that makes quaternions and Euler angles hard to regress with an L2 loss.
Two Families: Direct Regression vs. Keypoint/Correspondence
There are two dominant architectural approaches, and the choice matters for your deployment.
Direct regression (PoseCNN, SSD-6D, YOLO-6D) predicts the pose parameters straight from the image. PoseCNN (Xiang et al., 2017) regresses a quaternion plus translation and adds a pixel-wise vector field to recover the center. These methods are fast — 20–40 ms on an RTX 3060 — but they struggle with occlusion because the network must memorize pose from partial appearance. Typical accuracy on the YCB-Video dataset is around 3–5 cm ADD error.
Keypoint/correspondence methods (PVNet, BB8, and the PnP family) predict 2D locations of known 3D object points, then solve Perspective-n-Point. PVNet (Peng et al., CVPR 2019) predicts a unit vector field per pixel pointing toward each keypoint and votes, which is robust to occlusion and truncation. Accuracy on LineMOD reaches 0.8–0.9 ADD-S, but inference is slower (80–120 ms) because of the voting step.
A third family has become dominant since 2020: render-and-compare and RGB-D fusion methods. CosyPose and MegaPose render the CAD model at candidate poses and compare against the observed image. FoundationPose (Wen et al., 2024) combines a learned feature matcher with a refinement network and reports 1–3 mm translation error on novel objects without per-object training. These are the methods you reach for when you have a CAD model and a depth camera.
What Depth Buys You
An RGB-only pipeline must infer scale and depth from appearance, which is ambiguous for textureless parts. An RGB-D pipeline gets metric depth directly. A RealSense D435i at 1 m range has roughly 2–3 mm depth noise on a matte surface and 10–15 mm on a shiny one. That noise floor sets your achievable translation accuracy. If your grasp tolerance is 3 mm, a shiny part at 1 m is already at the edge.
The practical consequence: for RGB-D pose estimation, the depth channel usually drives translation and the RGB channel drives rotation. Splitting the problem this way — a learned 2D detector for the region of interest, PnP or ICP on the depth for translation, and a learned rotation head — is a robust engineering pattern.
Mathematical Foundation
The core metric for evaluating pose error is ADD (Average Distance of model points), introduced with PoseCNN:
$$\text{ADD} = \frac{1}{m}\sum_{i=1}^{m} \left| (R,x_i + t) - (\hat{R},x_i + \hat{t}) \right|_2$$
Every symbol: $x_i$ is the $i$-th of $m$ sampled 3D model points in the object frame. $R, t$ are the ground-truth rotation and translation. $\hat{R}, \hat{t}$ are the estimated ones. The term $R x_i + t$ maps a model point into the camera frame under ground truth; the second term does the same under the estimate. ADD is the mean Euclidean distance between corresponding points after both transforms. A pose is counted correct if ADD is below 10% of the object’s diameter.
Worked example. Take a 100 mm cube. Sample $m = 8$ corner points. Suppose the estimate has a pure rotation error of 5° about the z-axis and zero translation error. The ground-truth and estimated corners differ by at most $2 \cdot r \cdot \sin(2.5°)$ where $r$ is the corner distance from the axis. The corner at $(50, 50, 50)$ mm sits at radius $r = 70.7$ mm from the z-axis, so the displacement is $2 \cdot 70.7 \cdot \sin(2.5°) = 6.17$ mm. The ADD over all 8 corners averages lower because corners near the axis move less, but the worst corner is what breaks your grasp. This is why practitioners report both mean ADD and max corner error.
For symmetric objects, plain ADD is wrong: a 180° rotation of a washer maps the model onto itself, so ADD reports a huge error for a physically correct pose. The fix is ADD-S, which uses the nearest-point distance:
$$\text{ADD-S} = \frac{1}{m}\sum_{i=1}^{m} \min_{j} \left| (R,x_i + t) - (\hat{R},x_j + \hat{t}) \right|_2$$
The inner min over $j$ lets each predicted point match its closest ground-truth point, so symmetric poses score near zero.
The second equation you need is the PnP reprojection. Given 3D model points $X_i$ and their observed 2D pixels $u_i$, with camera intrinsics $K$:
$$u_i = \pi(K (R X_i + t)), \quad \pi([x,y,z]^T) = [x/z,; y/z]^T$$
PnP solves for $R, t$ minimizing $\sum_i | u_i - \pi(K(R X_i + t)) |^2$. With 4 or more non-coplanar points and a calibrated $K$, this is solvable in closed form (EPnP) or by iterative least squares (Levenberg-Marquardt). RANSAC wraps it to reject outlier keypoints — critical when a keypoint lands on an occluding part.
Practical Example
Scenario. A UR10e arm with a Robotiq 2F-85 gripper picks M8 steel bolts from a tote. Bolt length 60 mm, head diameter 13 mm, mass 25 g. Camera: Intel RealSense D435i mounted on the wrist at 450 mm standoff. Grasp tolerance: the gripper must close within ±4 mm of the bolt shaft center and the approach axis must be within ±8° of the shaft axis. Cycle time budget: 1.2 s per pick, so pose estimation must complete in under 250 ms.
Step 1 — Detection. Run a YOLOv8n detector fine-tuned on 2,000 labeled frames. It outputs a bounding box per visible bolt at 15 ms. Expect 3–8 bolts visible per frame with 40% average occlusion.
Step 2 — Pose. Crop each box, feed a 256×256 RGB-D patch to a keypoint network that predicts the 2D projections of the bolt head center and two shaft endpoints, plus a per-pixel uncertainty. Run EPnP with RANSAC (100 iterations, 2 px reprojection threshold) at 8 ms per bolt. Total per bolt: about 25 ms. For 6 bolts: 150 ms. Within budget.
Step 3 — Ambiguity check. A bolt is rotationally symmetric about its shaft. The roll about the shaft axis is unobservable. The pipeline must report this: set the roll to a canonical value and flag the pose as “axis-only.” The grasp planner only needs the shaft axis, so this is fine — but if you fed the full pose to a planner expecting a unique answer, it would jitter.
Step 4 — Verification. Before commanding the grasp, back-project the estimated pose into the depth image and compute the mean absolute depth residual over the visible bolt pixels. If residual > 4 mm, reject the pose and re-image. This catches the common case where the keypoint network locked onto a neighboring bolt.
The numbers that matter: at 450 mm standoff with a 2 mm depth noise floor, translation error is about 1.5 mm RMS, well inside the 4 mm tolerance. Rotation error from PnP with 3 keypoints is about 3°, inside the 8° tolerance. The binding constraint is not accuracy — it is the 40% occlusion rate, which drops the visible keypoint count from 3 to sometimes 2, at which point PnP is underdetermined and you must fall back to a model-based fit or skip the pick.
Common Pitfalls
1. Reporting a single pose for a symmetric object. What goes wrong: the estimator returns a confident pose, the planner grasps, and the part is 180° off. Why: the network was trained with an L2 loss on quaternions, and for symmetric parts the loss surface has multiple minima; the network picks one arbitrarily. Detect it: check ADD-S versus ADD during validation — a large gap means symmetry. Fix: use a symmetry-aware loss, or classify the object’s symmetry group and collapse the rotation to its canonical representative before regression.
2. Trusting RGB-only translation. What goes wrong: depth error of 20–40 mm on a shiny part, grasp misses by a full finger width. Why: monocular scale is inferred from object size priors, which fail under partial occlusion. Detect it: compare RGB-only translation against a depth-based estimate on 50 frames. Fix: use the depth channel for translation and RGB only for rotation, or fuse both with a learned confidence weighting.
3. Uncalibrated or drifted camera intrinsics. What goes wrong: poses are systematically biased by 5–10 mm across the whole workspace. Why: PnP assumes $K$ is exact; a 1% focal length error produces a proportional depth error. Detect it: pose a calibration target at three known distances and check reprojection error; anything above 0.5 px means recalibrate. Fix: recalibrate with a ChArUco board, and re-check after any mechanical shock to the wrist mount.
4. Training on synthetic data with the wrong domain gap. What goes wrong: 95% accuracy in simulation, 40% on the real cell. Why: renderers produce clean edges and uniform lighting; real totes have motion blur, specular streaks, and cluttered backgrounds. Detect it: hold out 200 real frames and measure the drop. Fix: domain randomization on lighting, texture, and camera noise, plus 10–20% real frames in the training set. A 15% real-data mix typically recovers most of the gap.
5. Ignoring temporal consistency. What goes wrong: poses jitter frame to frame by 3–5°, the robot hesitates, cycle time blows up. Why: each frame is estimated independently and the network has no memory. Detect it: log the pose delta between consecutive frames at a static scene; anything above 1° is noise. Fix: apply a constant-velocity Kalman filter on $SE(3)$, or average over a 3-frame window when the part is stationary.
Key Takeaways
- 6D pose means $SE(3)$: three translation and three rotation values. Translation regresses cleanly; rotation lives on $SO(3)$ and needs a continuous parameterization (the 6D Zhou representation) to train well.
- Evaluate with ADD, and ADD-S for symmetric objects. A 5° rotation error on a 100 mm part already displaces corners by about 6 mm — check max corner error, not just the mean.
- RGB-D beats RGB-only for translation. Depth noise of 2–3 mm on matte parts sets your accuracy floor; shiny parts at 1 m can hit 15 mm and break tight tolerances.
- Pick the architecture to match occlusion. Direct regression is fast (20–40 ms) but fragile under heavy occlusion; keypoint-plus-PnP (PVNet, EPnP with RANSAC) is slower but robust; render-and-compare (FoundationPose) is the choice when you have CAD and need millimeter accuracy on novel objects.
- Budget the full pipeline. Detection, pose, ambiguity check, and depth verification together must fit your cycle time. For a 1.2 s pick cycle, target under 250 ms of vision compute.
Tomorrow
Vision Transformers for Robotics — how self-attention replaces convolution for long-range spatial reasoning, and when a ViT backbone actually beats a CNN on manipulation tasks.