Week 11 — Machine Learning for Robotics Day 1 of 7
Why This Matters
A robot arm that performed perfectly in your lab will fail on day one at the customer site. The lighting is different. The camera lens has a smudge. A new part variant arrived with a matte finish instead of glossy. Your model outputs garbage, the gripper closes on air, and the line stops. This is not a model architecture problem. It is a pipeline problem.
The failure almost always traces back to one of three places. The training data did not represent the deployment distribution. The preprocessing at inference time did not match preprocessing at training time. Or the training workflow leaked test information into the model, so your 99.2% validation accuracy was fiction.
Consider a concrete number. A pick-and-place cell running at 1,200 cycles per hour loses roughly €4,800 per hour of downtime at a typical automotive tier-2 supplier. A model that misfires on 2% of parts instead of 0.1% costs you 24 failed cycles per hour. That is the difference between a shipped product and a returned one.
The ML pipeline is the part of the system that turns raw sensor bytes into a model you can trust. Getting it right is 80% of the engineering work. The model itself is often 20 lines of PyTorch. This lesson builds that pipeline end to end: collection, preprocessing, splitting, training, and validation, with the robotics-specific traps called out.
Core Concepts
Data Collection: Cover the Deployment Distribution, Not the Lab
A supervised perception model learns $p(y \mid x)$. It only works at deployment if the deployment inputs $x$ come from the same distribution as training. In robotics, that distribution shifts constantly. Cameras move. Parts wear. Ambient light changes with the shift clock.
Start by defining your input space explicitly. For a 6-DOF arm doing bin picking with an Intel RealSense D435 at 848×480, your inputs are RGB frames, depth frames, and the robot’s joint encoders. Enumerate the factors that vary:
| Factor | Training range | Deployment risk |
|---|---|---|
| Ambient illuminance | 300–800 lux | Skylight glare at 2,000 lux |
| Part orientation | 0–360° yaw | Same, but occluded by neighbors |
| Camera-to-part distance | 0.4–0.7 m | 0.35 m when bin is full |
| Lens condition | Clean | Dust after 200 hours |
| Part finish | Glossy | Matte batch from new supplier |
You cannot collect every combination. You can, however, use domain randomization on the factors you control (lighting, texture, camera pose) and stratified sampling on the factors you observe. A practical target is 500–2,000 labeled frames per class for a CNN classifier, and 10,000+ annotated instances for a segmentation model like Mask R-CNN. Label with a tool that records timestamps, so you can join labels back to raw sensor logs.
Log everything raw. Store the unmodified sensor stream plus a timestamp and the robot state. Preprocessing is a function you can re-run; raw data you cannot recover.
Preprocessing: The Train-Serve Skew Trap
The single most common production bug is train-serve skew: the preprocessing applied during training differs from what runs on the robot. A model trained on images normalized with mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225] (the ImageNet statistics) will silently degrade if the inference node normalizes with [0.5, 0.5, 0.5]. Accuracy drops 10–30 points and nobody notices until the line stops.
Fix this by making preprocessing a single versioned function. The training script and the robot’s inference node must import the same code. Pin its hash in the model artifact metadata.
For depth data, handle invalid pixels. The D435 returns 0 for pixels with no return. If you feed 0 into a network as a valid depth, you teach it that “no return” means “zero meters away.” Mask invalid pixels and either inpaint them or pass a validity channel.
For time-series data (joint torques, IMU), resample to a fixed rate. A 100 Hz IMU logged with jitter must be resampled to exactly 100 Hz before windowing, or your 50-sample windows represent 0.47–0.53 s of real time. Use linear interpolation for smooth signals and zero-order hold for discrete states.
Mathematical Foundation
Normalization is the workhorse of preprocessing. For each channel $c$:
$$ \hat{x}_c = \frac{x_c - \mu_c}{\sigma_c} $$
where $x_c$ is the raw value, $\mu_c$ is the mean computed on the training split only, $\sigma_c$ is the standard deviation on the training split only, and $\hat{x}_c$ is the normalized value fed to the network.
Computing $\mu_c$ and $\sigma_c$ on the full dataset (including test) is data leakage. It inflates validation accuracy and hides real failures.
Worked example. Suppose you collect 1,000 depth frames. On the training split (700 frames), the valid depth pixels have $\mu = 0.62$ m and $\sigma = 0.11$ m. A new pixel reads $x = 0.90$ m:
$$ \hat{x} = \frac{0.90 - 0.62}{0.11} = 2.55 $$
That is 2.55 standard deviations from the mean, a plausible but somewhat distant value. Now suppose you had computed statistics on the full set and got $\mu = 0.58$, $\sigma = 0.09$. The same pixel normalizes to $3.56$. Different input to the network, different prediction, and your validation numbers no longer describe deployment behavior.
The splitting rule: train/validation/test = 70/15/15 by episode, not by frame. Frames from the same grasp attempt are highly correlated. Splitting by frame leaks near-duplicates into validation and inflates accuracy by 5–15 points.
Practical Example
You are building a grasp-success classifier for a UR5e with a Robotiq 2F-85 gripper, picking 40 mm aluminum brackets from a bin. The classifier takes a cropped RGB-D patch and predicts success or failure before the gripper closes. Target: false-positive rate below 1% (a failed grasp costs 8 s of recovery).
Step 1 — Collection. Run 300 grasp attempts over 3 days. Vary lighting (morning, afternoon, overhead off), bracket orientation, and bin fill level. Log raw RGB, depth, joint states, and the ground-truth outcome (did the part end up in the drop zone?). Result: 300 episodes, 41,200 candidate grasp poses, 4,180 successful.
Step 2 — Labeling. Success labels come from the robot, not a human. If the part is detected in the drop zone within 3 s of release, label 1; else 0. This is free, accurate, and scales.
Step 3 — Splitting. Split by episode: 210 train, 45 validation, 45 test. Never split by frame. A single episode contributes ~140 frames that are 95% identical.
Step 4 — Preprocessing. Crop to 224×224 around the grasp point. Normalize RGB with training-split statistics. Normalize depth with training-split statistics, mask invalid pixels, and append a binary validity mask as a 4th channel. Resize with bilinear interpolation, not nearest, because nearest aliasing hurts small-object classification by 3–6 points.
Step 5 — Training. ResNet-18, batch size 64, Adam at $1\times10^{-4}$, class-weighted cross-entropy to counter the 10:1 imbalance. Early stopping on validation F1 with patience 8.
Step 6 — Validation. Test-set false-positive rate: 0.8%. Validation F1: 0.91. Ship it.
Step 7 — Monitor. Log every inference input’s mean and std. If the deployment mean drifts more than 2σ from training, raise an alert. This catches lens dust and lighting changes before they cost a shift.
Common Pitfalls
1. Train-serve skew. What goes wrong: the robot’s inference code preprocesses differently from training. Why: two separate codebases drift apart. Detect: log the preprocessed tensor hash at inference and compare to a golden sample. Fix: one shared preprocessing module, versioned and imported by both.
2. Leakage through splits. What goes wrong: validation accuracy is 98%, deployment is 70%. Why: frames from the same episode land in both train and validation. Detect: check that no episode ID appears in two splits. Fix: split by episode or by time block, never by random frame.
3. Class imbalance ignored. What goes wrong: the model predicts “fail” for everything and gets 90% accuracy. Why: 10:1 imbalance dominates the loss. Detect: inspect per-class recall, not just accuracy. Fix: class-weighted loss, focal loss, or resampling.
4. Label noise from humans. What goes wrong: the model learns the labeler’s biases. Why: human annotators disagree on ambiguous grasps. Detect: have two annotators label 200 samples and compute Cohen’s κ; below 0.7 means the task definition is unclear. Fix: derive labels from robot outcomes where possible.
5. No distribution monitoring. What goes wrong: the model silently degrades over weeks. Why: nobody watches input statistics. Detect: track input mean, std, and prediction confidence per shift. Fix: alert on drift beyond a threshold, and retrain on fresh data quarterly.
Key Takeaways
- Split by episode, not by frame. Correlated frames inflate validation accuracy by 5–15 points. This is the most common silent failure.
- Compute normalization statistics on the training split only. Using the full dataset is leakage.
- Share one preprocessing function between training and inference. Version it and pin its hash in the model artifact.
- Derive labels from robot outcomes when possible. It is cheaper, faster, and less biased than human annotation.
- Monitor input statistics in production. Drift detection catches lens dust and lighting changes before they cost a shift.
Tomorrow
We move from the pipeline to the models that run inside it: supervised learning for perception, covering classification, detection, and segmentation architectures with the robotics-specific trade-offs that decide which one you ship.