Week 11 — Machine Learning for Robotics Day 5 of 7
Why This Matters
You train a policy in simulation. It walks a quadruped across a flat plane at 1.2 m/s. The reward curve is clean. Then you deploy on the physical robot, and within 400 ms the legs chatter and the robot falls forward. Nothing in your training loop told you why.
This is the reality gap. Simulation is a cheap, fast, safe place to train. It is also wrong in dozens of small ways. Contact friction is simplified. Motor dynamics lag. Sensor noise is Gaussian when the real IMU has bias drift and quantization. Mass properties are estimated, not measured. A policy that exploits simulator quirks gets high simulated reward and zero real reward.
The cost of ignoring this is concrete. A single fall on a Boston Dynamics Spot-class platform can damage a $2,000 leg assembly. A poorly transferred manipulation policy on a UR5e moving a 2 kg payload can crush a fixture. You cannot afford thousands of real-world trial-and-error episodes. Sim-to-real transfer is how you get the sample efficiency of simulation and the validity of hardware.
Industry numbers make the case. Training a locomotion policy in Isaac Gym can run at roughly 100,000+ environment steps per second on one GPU across thousands of parallel environments. The same policy on hardware runs at the control loop rate, typically 200–1000 Hz. That is a gap of five to six orders of magnitude in data throughput. You must close the reality gap in simulation, because you cannot close it on the robot.
Core Concepts
Domain Randomization
The core idea: instead of training in one simulator, train across a distribution of simulators. You randomize the parameters you cannot measure precisely. The policy learns to be robust to the whole distribution. At deployment, the real world is just another sample.
Typical randomization ranges for a small quadruped:
| Parameter | Nominal | Randomization range | Why |
|---|---|---|---|
| Link mass | 0.8 kg | ±20% | CAD vs. real mass mismatch |
| Friction coefficient | 0.8 | 0.3–1.2 | Floor material varies |
| Motor strength | 1.0× | 0.85–1.15× | Thermal, battery sag |
| CoM offset | 0 mm | ±15 mm | Wiring, payload |
| Action latency | 0 ms | 0–20 ms | Bus and firmware delay |
| IMU noise | 0 | σ up to 0.05 rad/s | Sensor spec |
The trade-off is real. Too narrow a range and the policy overfits to the simulator. Too wide a range and the policy becomes conservative and sluggish, because it must handle a 0.3-friction ice world and a 1.2-friction rubber world with one controller. Practitioners often use curriculum randomization: start narrow, widen as training progresses. The OpenAI Dactyl work (2018–2019) used randomization over object mass, friction, and appearance to get a Shadow Hand to reorient a cube, a well-known case of dexterous manipulation transferring with no real-world fine-tuning.
System Identification and Residual Models
Randomization is a blunt tool. A sharper approach is to measure the real system and match it. System identification (sysID) fits simulator parameters to real trajectories. You command a known torque profile on a joint, record the response, and fit motor constants, damping, and inertia.
A common modern pattern is the residual model. You train a nominal policy in a sysID-matched simulator, then learn a small correction network that maps the gap. The real dynamics are modeled as:
$$\dot{x}{real} = f{sim}(x, u; \theta) + g_\phi(x, u)$$
where $f_{sim}$ is the simulator with identified parameters $\theta$, and $g_\phi$ is a learned residual with small parameters $\phi$. Because $g_\phi$ only needs to capture the mismatch, it trains from far less real data than a full model. This is the logic behind many recent agile-locomotion papers.
Observation and Action Space Matching
A third, often-overlooked source of gap is interface mismatch. If your policy expects joint velocity from an ideal encoder but the real encoder is differentiated position with 2 ms delay and quantization, the policy sees inputs it never trained on. Fix this by injecting the same noise and delay into the simulated observation pipeline. Add a first-order filter with the measured time constant. Quantize to the real encoder resolution (e.g., 12-bit over the joint range). This costs nothing and removes a whole class of failures.
Mathematical Foundation
The formal framing is a Partially Observable Markov Decision Process (POMDP) with a dynamics mismatch. The simulated transition is $P_{sim}(s’ \mid s, a)$ and the real transition is $P_{real}(s’ \mid s, a)$. The reality gap is a bound on their divergence.
Domain randomization treats the transition as drawn from a distribution over parameters $\xi \sim p(\xi)$:
$$P_{DR}(s’ \mid s, a) = \int P(s’ \mid s, a, \xi), p(\xi), d\xi$$
The training objective maximizes expected return over this mixture:
$$J(\pi) = \mathbb{E}{\xi \sim p(\xi)}\left[\mathbb{E}{\tau \sim \pi, P_\xi}\left[\sum_{t=0}^{T} \gamma^t r(s_t, a_t)\right]\right]$$
Symbols: $\pi$ is the policy, $\tau$ a trajectory, $\gamma \in (0,1)$ the discount, $r$ the reward, $T$ the horizon, $\xi$ the randomized parameter vector.
Worked example. Suppose your reward is $r = 1$ per step while upright and $-10$ on a fall, horizon $T=500$, $\gamma = 0.99$. Under a nominal simulator the policy succeeds 95% of episodes, expected return $\approx 0.95 \times 500 = 475$ (ignoring discount). Now add a 20 ms latency that the nominal policy never saw. On hardware, fall probability jumps to 60%. Real return $\approx 0.4 \times 500 - 0.6 \times 10 = 194$. The 281-point drop is the reality gap in reward terms.
Now randomize latency uniformly in $[0, 20]$ ms during training. The policy learns a slightly more cautious gait. Simulated success drops to 85%, but hardware success rises to 90%. The policy gave up 10% simulated return to gain 50% real return. That trade is the entire point.
A useful robustness bound comes from considering the Lipschitz constant $L$ of the policy with respect to observations. If the observation perturbation from the gap is bounded by $\epsilon$, the action perturbation is at most $L\epsilon$. Keeping $L$ small (through regularization or noise injection) bounds how badly the gap can hurt you.
Practical Example
Scenario. A 12 kg quadruped with 12 joints (3 per leg), torque-controlled, 500 Hz control loop. Task: trot at 0.8 m/s over mixed office flooring (carpet, tile, threshold strips). Tolerance: no falls in a 10-minute continuous run.
Step 1 — Identify what matters. Run a sensitivity sweep in simulation. Perturb each parameter 10% and measure reward change. Results:
| Parameter | Reward change at +10% |
|---|---|
| Friction | −18% |
| Action latency | −22% |
| Motor strength | −9% |
| Link mass | −4% |
| IMU noise | −2% |
Friction and latency dominate. Spend your randomization budget there.
Step 2 — Measure the real values. SysID on the real robot: measure friction by dragging a foot on each floor type (carpet $\mu \approx 0.6$, tile $\mu \approx 0.85$). Measure latency by timestamping a commanded step and the encoder response: 12 ms mean, 4 ms jitter. Set friction range $[0.4, 1.0]$ and latency $[4, 20]$ ms.
Step 3 — Train with randomization plus noise injection. Add observation noise matching encoder quantization (0.0015 rad) and IMU bias drift. Use 4,096 parallel environments in Isaac Gym. Train 300M steps, roughly 3 hours on one A100.
Step 4 — Evaluate the transfer. Deploy. Expect 2–3 falls in the first hour from unmodeled effects (e.g., the threshold strips). Those are your new randomization targets. Add a 10 mm step-height perturbation and retrain. Second deployment: zero falls in 10 minutes.
Step 5 — Hold out. Keep one floor type (e.g., the carpet) out of your sysID set and use it only for final testing. If the policy passes there, your randomization generalizes rather than memorizes.
Common Pitfalls
1. Randomizing everything uniformly. What goes wrong: the policy becomes timid and slow, and real performance is worse than a nominal policy. Why: the mixture includes physically impossible combinations (ice friction with heavy mass) that force conservative behavior. Detect: compare real speed against the nominal sim speed; a large drop signals over-randomization. Fix: use sensitivity analysis to randomize only high-impact parameters, and use curriculum widening.
2. Ignoring actuator delay. What goes wrong: high-frequency gaits that are stable in sim oscillate and fall on hardware. Why: the simulator applies actions instantly, so the policy exploits zero-latency feedback that does not exist. Detect: measure step-response latency with a scope or timestamped logs; if it exceeds 5 ms and is unmodeled, you have this bug. Fix: add a delay buffer of 0–20 ms to the action pipeline in sim.
3. Mismatched observation scaling. What goes wrong: the policy behaves erratically on hardware but fine in sim. Why: sim normalizes joint positions by a range that differs from the real encoder range, so inputs are shifted. Detect: log the min/max of each observation channel on both sim and real; compare distributions. Fix: use identical normalization constants derived from real hardware specs.
4. Training on a single terrain. What goes wrong: the robot handles the lab floor and fails on the first threshold. Why: the policy never saw contact discontinuities, so it has no recovery behavior. Detect: test on a held-out terrain set. Fix: add terrain randomization (step heights 0–15 mm, slopes 0–10°) and randomized push disturbances (50–150 N for 0.1 s).
5. Evaluating only on average reward. What goes wrong: you ship a policy with good mean return but a fat tail of catastrophic falls. Why: the mean hides rare high-cost events that dominate real-world risk. Detect: plot the return distribution and the fall rate, not just the mean. Fix: use a risk-sensitive objective, such as CVaR over the lower 5% of episode returns, or add a hard penalty on falls.
Key Takeaways
- The reality gap is a distribution mismatch, not a single bug. Treat it as a distribution over parameters and train across it.
- Spend your effort where sensitivity is highest. A 10% perturbation sweep tells you which three parameters matter; randomize those, not all twenty.
- Measure latency and friction on real hardware. Guessing these two values accounts for most transfer failures.
- Match the observation and action interfaces exactly: noise, delay, quantization, and normalization constants. This is free and removes a whole class of failures.
- Hold out a terrain or condition for final testing. If your policy only passes where you tuned, you have not closed the gap.
Tomorrow
Tomorrow we move from theory to code: you will build and train an RL agent with Stable Baselines3 in Python, including the wrappers you need to inject the randomization and delay discussed today.