Week 11 — Machine Learning for Robotics Day 4 of 7
Why This Matters
A quadruped walking on ice, a 7-DOF arm inserting a peg into a 20 mm hole, a warehouse robot recovering after a human kicks it — these are control problems where hand-tuned PID loops fall apart. A PID controller for a leg joint has maybe 3 gains. A whole-body walking controller has 12–24 coupled joints, contact switching, and a 500 Hz control loop. Writing that by hand takes a team months and still fails on new terrain.
Reinforcement learning (RL) replaces the hand-tuned controller with a policy $\pi_\theta(a \mid s)$ trained in simulation. The payoff is concrete. In 2019, Hwangbo et al. (ANYmal, Science Robotics) trained a single policy that controlled all 12 leg joints at 200 Hz on a real quadruped, without a separate state estimator or trajectory planner. The policy ran on the robot’s onboard CPU. That is the kind of result that changes how you staff a controls team.
Without RL, three things break in the real world:
- Contact-rich tasks. Peg insertion, valve turning, and foot placement have discontinuous dynamics. Gradient-based trajectory optimization struggles because the contact force is non-smooth. RL handles this by sampling and averaging over contact events.
- Robustness to perturbation. A model-based controller tuned on flat ground fails on gravel. RL policies trained with randomized disturbances recover — the ANYmal policy kept walking after being pushed and after motor failures.
- Scale of tuning. Tuning 18 joint PD gains plus a gait scheduler by hand is weeks of work per robot. One RL training run in Isaac Gym (4096 parallel environments) takes ~20 minutes on a single GPU.
The cost is real too: sample inefficiency, reward hacking, and the sim-to-real gap. This lesson shows you how to manage those costs for control tasks.
Core Concepts
Locomotion as a Markov Decision Process
You frame the controller as an MDP: state $s_t$, action $a_t$, reward $r_t$, transition $p(s_{t+1}\mid s_t,a_t)$, discount $\gamma$.
For a quadruped (Unitree A1, 12 joints), a standard formulation from Rudin et al. (2021, “Learning to Walk in Minutes”) is:
| Item | Value |
|---|---|
| Control frequency | 50 Hz (policy), 200 Hz (PD joint loop) |
| Action | target joint angles, 12-dim, scaled to ±0.25 rad around nominal |
| Observation | base linear velocity (3), angular velocity (3), gravity vector in body frame (3), joint positions (12), joint velocities (12), previous action (12) = 45 dims |
| Episode length | 20 s (1000 steps) |
| Parallel envs | 4096 |
| PPO minibatch | 4 × 4096 = 16384 samples |
The action is not torque. It is a target angle fed to a PD controller at 200 Hz. This is a deliberate design choice. The PD loop absorbs high-frequency motor dynamics, and the policy learns at a slower, more stable rate. Torque-level control is harder to train and transfers worse.
The reward is a weighted sum. A typical quadruped reward:
$$r_t = 1.0 \cdot r_{\text{lin vel}} + 0.5 \cdot r_{\text{yaw}} - 0.01 \cdot |\tau|^2 - 0.5 \cdot \mathbb{1}{\text{fall}} - 0.05 \cdot |a_t - a{t-1}|^2$$
Each term matters. The velocity tracking term rewards forward speed. The torque penalty keeps energy down. The fall penalty ends the episode. The action-smoothness term prevents jitter that destroys real motors.
Manipulation: Sparse Rewards and HER
For a Franka Emika Panda arm doing pick-and-place, the reward is often sparse: +1 when the object is within 5 cm of the goal, 0 otherwise. Sparse rewards make exploration nearly impossible — the probability of random success is tiny.
Two fixes dominate:
Hindsight Experience Replay (HER) (Andrychowicz et al., 2017). After a failed episode, relabel the goal as the achieved state. If the arm reached position $p$ instead of goal $g$, store the transition with goal $p$. Now the agent learns “how to reach where I ended up,” which is dense signal. HER typically cuts sample complexity by 10–50× on goal-conditioned tasks.
Reward shaping with potential functions. Add a dense term $r’ = r + \gamma \Phi(s’) - \Phi(s)$ where $\Phi$ is a distance-to-goal potential. This preserves the optimal policy (Ng et al., 1999) if $\Phi$ is a true potential. A common choice: $\Phi(s) = -|p_{\text{ee}} - p_{\text{goal}}|_2$.
PPO: The Default Algorithm
Proximal Policy Optimization (Schulman et al., 2017) is the workhorse for robot control. It is on-policy, so it needs fresh samples, but it is stable and easy to tune. The clipped surrogate objective:
$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta)\hat{A}_t,\ \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t\right)\right]$$
where $r_t(\theta) = \pi_\theta(a_t\mid s_t)/\pi_{\theta_{\text{old}}}(a_t\mid s_t)$ is the probability ratio, $\hat{A}_t$ is the advantage estimate, and $\epsilon = 0.2$ is the clip range. The clip stops the policy from moving too far in one update, which is what makes PPO stable enough to run unattended.
Advantages come from Generalized Advantage Estimation (GAE):
$$\hat{A}t = \sum{l=0}^{T-t}(\gamma\lambda)^l \delta_{t+l}, \quad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$
with $\gamma = 0.99$ and $\lambda = 0.95$ typical for locomotion.
Mathematical Foundation
The policy gradient theorem gives the gradient of expected return:
$$\nabla_\theta J(\theta) = \mathbb{E}{\tau \sim \pi\theta}\left[\sum_{t=0}^{T}\nabla_\theta \log \pi_\theta(a_t\mid s_t),\hat{A}_t\right]$$
Symbols:
- $J(\theta)$: expected discounted return $\mathbb{E}[\sum_t \gamma^t r_t]$
- $\tau$: a trajectory $(s_0,a_0,r_0,\dots)$
- $\pi_\theta(a_t\mid s_t)$: the policy, here a Gaussian with mean from a 2-layer MLP (256×256, ELU) and learned log-std
- $\hat{A}_t$: advantage — how much better action $a_t$ was than the average action at $s_t$
- $\nabla_\theta \log \pi_\theta$: the score function; it increases the probability of actions with positive advantage
Worked numeric example. Suppose the policy is Gaussian with mean $\mu = 0.1$ and std $\sigma = 0.3$ for a 1-D action. The agent samples $a = 0.4$. The log-probability is:
$$\log \pi(a\mid s) = -\frac{(a-\mu)^2}{2\sigma^2} - \log(\sigma\sqrt{2\pi}) = -\frac{0.09}{0.18} - \log(0.752) = -0.5 + 0.285 = -0.215$$
The score is $\nabla_\mu \log\pi = (a-\mu)/\sigma^2 = 0.3/0.09 = 3.33$. If GAE gives $\hat{A} = 0.8$, the policy-gradient update pushes $\mu$ up by $\alpha \cdot 3.33 \cdot 0.8 = 2.66\alpha$. With learning rate $\alpha = 3\times10^{-4}$, $\mu$ moves by about $8\times10^{-4}$ — a small step. This is why PPO uses 4–10 epochs per batch and 4096 parallel envs: many small steps in the right direction.
Discount and horizon. With $\gamma = 0.99$ at 50 Hz, the effective horizon is $1/(1-\gamma) = 100$ steps = 2 s. That is enough for a step cycle (~0.5 s) but not for navigation planning. For navigation tasks you either raise $\gamma$ to 0.995 (4 s horizon) or feed a goal vector into the observation.
Practical Example
Scenario: Train a velocity-tracking policy for a Unitree Go1 quadruped. Target: walk forward at 1.0 m/s, follow yaw commands up to 0.5 rad/s, survive 1000-step episodes, deploy on the real robot within a week.
Step 1 — Simulation. Use Isaac Gym with 4096 envs, dt = 0.02 s (50 Hz policy). Terrain: flat plane plus 30% of envs on randomized friction (0.4–1.2) and small height noise (±2 cm). This randomization is the seed of sim-to-real robustness.
Step 2 — Observation and action. Observation is the 45-dim vector above. Action is 12 target joint angles, scaled to ±0.25 rad, plus a fixed nominal pose (the standing pose). PD gains: $K_p = 20$, $K_d = 0.5$ per joint.
Step 3 — Reward weights. Start with $r_{\text{lin vel}}$ weight = 1.0 and yaw weight = 0.5. Add penalties: torque $|\tau|^2$ weight 2.5e-5, action rate weight 0.01, feet air time 0.02 (encourages a trot). Add a fall penalty of −50 and terminate the episode.
Step 4 — Train. PPO with 4 epochs, minibatch 16384, learning rate 3e-4 with adaptive KL target 0.01. Expect ~1500 iterations, ~20–25 min on an RTX 3090. Watch the mean episode reward: it should climb from ~−20 (falling immediately) to ~+35 (stable trot) within 300 iterations.
Step 5 — Diagnose. If the robot shuffles with a low step height, check the foot clearance reward. If it falls when yawing, check that yaw commands are sampled uniformly in [−0.5, 0.5] rad/s during training. If it jitters, the action-rate penalty is too low.
Step 6 — Deploy. Export the policy as an ONNX graph. Run it at 50 Hz on the robot’s onboard computer, with the 200 Hz PD loop underneath. Measure torque tracking error: if it exceeds 10%, the sim motor model is wrong and the policy will limp.
This pipeline (Rudin et al. 2021) is what most quadruped companies use today. It is not magic — it is a disciplined sim setup plus a well-tuned PPO.
Common Pitfalls
1. Reward hacking. The policy finds a way to get reward without doing the task. Example: a locomotion policy that learns to hop on two legs because the velocity term rewards forward motion and the fall penalty never fires if it never falls. Why: the reward is a proxy, not the goal. Fix: inspect the trained policy in video every 100 iterations. Add a pose penalty or a foot-contact pattern reward. Check that the reward terms you intended are actually the ones dominating the gradient.
2. Sim-to-real gap from unmodeled latency. The real robot has 5–20 ms of control latency that the sim lacks. A policy trained with zero latency becomes unstable. Why: the policy learned to react to state it no longer has. Fix: randomize latency in sim (0–20 ms) and add an action history of 3 steps to the observation. Detect it if the robot oscillates at deployment but not in sim.
3. Observation normalization mismatch. Training normalizes observations with running mean/std. At deployment, if you forget to apply the same statistics, the policy sees inputs 10× too large. Why: the network’s first layer was tuned to the normalized scale. Fix: export the normalization constants with the policy and apply them in the deployment wrapper. This is the single most common deployment bug.
4. Sparse rewards with no shaping. Training a manipulation policy with only a success bonus. Why: the gradient signal is near zero for most of training. Fix: use HER or a potential-based shaping term. Verify by logging how often the random policy succeeds — if it is under 1%, you need shaping.
5. Too few parallel environments. Training PPO with 16 envs instead of 4096. Why: PPO needs low-variance advantage estimates; small batches give noisy gradients and the policy collapses. Fix: use at least 1024 envs. If GPU memory is tight, reduce network size (128×128) rather than env count.
6. Ignoring contact force limits. The policy commands a gait that looks fine in sim but exceeds the real motor’s 33 Nm peak torque. Why: sim actuators are often modeled as ideal. Fix: clamp torque in sim and add a penalty when the commanded torque saturates.
Key Takeaways
- Frame control as an MDP with a slow policy and fast PD loop. 50 Hz policy over 200 Hz joint control is the standard split for legged robots and works for arms too.
- PPO with 4096 parallel envs is the default. It trains a quadruped policy in ~20 minutes. Use $\gamma=0.99$, $\lambda=0.95$, $\epsilon=0.2$, learning rate 3e-4.
- Sparse rewards need HER or potential-based shaping. Without them, manipulation policies never leave the random-exploration regime.
- Randomize what you cannot model. Friction, latency, payload, and motor strength randomization in sim is what makes the policy survive on the real robot.
- The deployment bug is usually normalization or latency, not the policy. Export normalization statistics with the ONNX graph and test the deployment wrapper against the training wrapper on identical inputs.
Tomorrow
Tomorrow we tackle the gap between simulation and the real robot: Sim-to-Real Transfer — domain randomization, system identification, and the specific tricks that make a policy trained in Isaac Gym survive on hardware.