Week 11 — Machine Learning for Robotics Day 3 of 7
Why This Matters
A robot arm that has been taught by imitation only knows how to do what it saw. A warehouse AMR trained on a fixed map fails the moment a pallet is moved. Hand-tuned PID gains work for one payload and oscillate for another. In each case, the engineer cannot write down the right behavior in advance. The environment is too variable, and the correct action depends on the current state in ways no rule list captures.
Reinforcement learning (RL) addresses this by letting the robot improve from its own experience. The agent acts, observes a reward, and adjusts. This is how DeepMind’s DQN reached human-level play on 49 Atari games in 2015, and how later work learned quadruped walking policies that transfer to real hardware. For robotics, the appeal is direct: a policy can be trained in simulation for millions of episodes, then deployed.
What breaks without RL? Consider a 6-DoF arm doing bin picking. The object pose is uncertain to ±3 mm. A supervised policy trained on 10,000 demonstrations fails on the 11,000th object because it never saw that geometry. An RL agent trained with domain randomization on 50 million simulated grasps learns a recovery behavior: approach, detect slip, re-grip. That recovery is not in any demonstration. It emerges from reward.
RL is not a replacement for classical control. It is a tool for problems where the policy is high-dimensional and the objective is a scalar. Today you will learn the three foundations: MDPs, Q-learning, and policy gradients.
Core Concepts
Markov Decision Processes: The Problem Statement
Every RL problem is a Markov Decision Process (MDP). It is a tuple $(S, A, P, R, \gamma)$:
- $S$: set of states. For a robot arm, $s \in \mathbb{R}^{14}$ (7 joint angles, 7 joint velocities).
- $A$: set of actions. For position control, $a \in \mathbb{R}^{7}$ (joint position deltas).
- $P(s’ \mid s, a)$: transition probability. In simulation this is the physics engine. On hardware it is unknown.
- $R(s, a)$: scalar reward. Example: $+1$ for a successful grasp, $-0.01$ per timestep to encourage speed.
- $\gamma \in [0,1)$: discount factor. $\gamma = 0.99$ means a reward 100 steps away counts 37% as much ($0.99^{100} \approx 0.366$).
The Markov property says the next state depends only on the current state and action. This is a strong assumption. If your state omits velocity, the robot is not Markov. The policy will fail because two different physical situations look identical. This is why state design matters more than algorithm choice in most robotics projects.
A trajectory is $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots)$. The return is $G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k}$. The goal is a policy $\pi(a \mid s)$ that maximizes expected return.
Q-Learning: Learning Values Without a Model
Q-learning learns the action-value function $Q(s,a)$: the expected return from taking action $a$ in state $s$ and then acting optimally. The update is:
$$Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma \max_{a’} Q(s’,a’) - Q(s,a) \right]$$
Here $\alpha$ is the learning rate. The bracket is the temporal-difference (TD) error. Q-learning is off-policy: it learns the optimal policy while following an exploratory one. This is powerful because you can explore with $\epsilon$-greedy ($\epsilon = 0.1$ means 10% random actions) and still learn the greedy optimum.
For continuous robot actions, tabular Q-learning fails because $A = \mathbb{R}^7$ is infinite. Two fixes exist. First, discretize: 7 joints × 5 bins = 78,125 cells, which is manageable. Second, use a neural network to approximate $Q_\theta(s,a)$, which is Deep Q-Network (DQN). DQN uses a replay buffer and a target network to stabilize training. It works well for discrete actions, like choosing among 8 grasp orientations.
Q-learning has a known failure mode: overestimation. The $\max$ operator is biased upward when Q-values are noisy. Double Q-learning fixes this by decoupling action selection from evaluation. In practice, always use the double variant.
Policy Gradients: Direct Policy Search
Policy gradient methods skip the value function and directly adjust the policy parameters $\theta$. The core result is the policy gradient theorem:
$$\nabla_\theta J(\theta) = \mathbb{E}{\tau \sim \pi\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t) , G_t \right]$$
Intuition: increase the probability of actions that led to high return. The variance of this estimator is large, so two standard tricks reduce it. First, subtract a baseline $b(s_t)$ (often the value function) to get the advantage $A_t = G_t - b(s_t)$. Second, use Generalized Advantage Estimation (GAE) with $\lambda = 0.95$ to trade bias for variance.
Proximal Policy Optimization (PPO) is the workhorse. It clips the policy ratio to $[1-\epsilon, 1+\epsilon]$ with $\epsilon = 0.2$, preventing destructively large updates. PPO trains stably on 8 parallel simulated environments and is what most robotics teams use for continuous control. Soft Actor-Critic (SAC) is an alternative that adds an entropy bonus, which encourages exploration and often improves sample efficiency.
The tradeoff is clear. Q-learning is sample-efficient but discrete. Policy gradients handle continuous actions but need more samples. For a 7-DoF arm reaching task, expect 1–5 million environment steps with PPO in simulation. At 1,000 steps per second per environment on 8 cores, that is 2–10 minutes of wall-clock time.
Mathematical Foundation
The Bellman optimality equation defines the optimal value function:
$$V^(s) = \max_{a} \left[ R(s,a) + \gamma \sum_{s’} P(s’ \mid s, a) V^(s’) \right]$$
Symbols: $V^*(s)$ is the maximum expected return from state $s$. $R(s,a)$ is the immediate reward. The sum is the expected discounted future value over all possible next states. This equation is the fixed point that Q-learning converges to.
Worked numeric example. Consider a 2D grid robot with 4 actions (up, down, left, right). Reward is $-0.04$ per step (cost of moving). Reaching the goal gives $+1.0$. $\gamma = 0.9$, $\alpha = 0.5$.
Suppose the robot is at state $s$, takes action “right,” lands in $s’$, and receives $r = -0.04$. Current estimates: $Q(s, \text{right}) = 0.10$ and $\max_{a’} Q(s’, a’) = 0.80$.
TD target: $r + \gamma \max_{a’} Q(s’,a’) = -0.04 + 0.9 \times 0.80 = -0.04 + 0.72 = 0.68$.
TD error: $0.68 - 0.10 = 0.58$.
Update: $Q(s, \text{right}) \leftarrow 0.10 + 0.5 \times 0.58 = 0.39$.
The estimate jumped from 0.10 to 0.39 because the next state looks promising. Over many visits, Q-values propagate backward from the goal. After roughly 100 episodes on this 10×10 grid, the greedy policy (always pick the max Q action) becomes optimal. This is the same mechanism that lets A* find shortest paths, except Q-learning learns the cost-to-go without a map.
Practical Example
Scenario: a UR5e arm must insert a 20 mm peg into a hole with 1 mm clearance. The peg is held in the gripper. The hole position is known to ±5 mm. Force/torque sensor at 500 Hz. Tolerance for success: peg seated within 0.5 mm depth and 0.3 N lateral force.
Why RL here? Classical impedance control works, but tuning the gains for every peg-hole pair is slow. A learned policy can adapt the insertion force based on measured contact.
Design:
| Element | Choice | Reason |
|---|---|---|
| State | 6 joint angles, 6 velocities, 6-axis F/T, peg pose error (18 dims) | Markov: includes velocities and contact |
| Action | 6 joint position deltas, clipped to ±0.01 rad | Continuous, small steps |
| Reward | $-f_\text{lateral}$ during approach, $+10$ on seat, $-1$ on jam | Shaped reward |
| Algorithm | SAC, $\gamma=0.99$, $\tau=0.005$ | Continuous actions, entropy for exploration |
| Environment | PyBullet, 16 parallel workers | ~2,000 steps/s aggregate |
Step by step. First, train in simulation for 2 million steps. The reward shaping is critical: without the jam penalty, the agent learns to push hard and jam. Second, check the learning curve. If mean episode return plateaus below 50% success, the state is likely missing contact information. Third, evaluate on 1,000 randomized hole positions. Success rate should exceed 90% before sim-to-real. Fourth, deploy with a safety envelope: if lateral force exceeds 5 N, the policy is overridden by a hard stop.
A common result: SAC reaches 95% success in simulation after 3 million steps (about 25 minutes on 16 cores). On hardware, expect a 10–20% drop without domain randomization. Day 5 covers closing that gap.
Common Pitfalls
1. Reward hacking. What goes wrong: the agent finds a shortcut. Example: a reward for “peg near hole” gets gamed by hovering. Why: the reward is a proxy, not the true objective. Detect: inspect trajectories of high-reward episodes manually. Fix: add penalties for undesired behavior (e.g., time cost) and test the reward on a held-out task.
2. Non-Markov state. What goes wrong: the policy oscillates or behaves randomly in deployment. Why: the state omits velocity or contact, so identical observations map to different dynamics. Detect: check if adding the last action or velocity changes performance. Fix: include $s_{t-1}$, $a_{t-1}$, and sensor history in the state.
3. Sparse rewards and no learning signal. What goes wrong: the agent never succeeds, so return is always 0 and gradients vanish. Why: random exploration rarely reaches a 1 mm target. Detect: plot success rate; if it stays at 0 for 100k steps, the reward is too sparse. Fix: use reward shaping, hindsight experience replay (HER), or curriculum learning starting with a 10 mm clearance.
4. Ignoring action limits and safety. What goes wrong: the policy commands 0.5 rad jumps and trips the robot’s safety stop. Why: the action space was not clipped during training. Detect: log action magnitudes; compare training and deployment distributions. Fix: clip actions in the environment, add a low-pass filter, and wrap the policy in a classical safety controller.
5. Overfitting to one seed. What goes wrong: one run succeeds, the next fails. Why: RL has high variance across seeds. Detect: run 5 seeds and report mean and standard deviation. Fix: report confidence intervals, use more parallel environments, and tune entropy regularization.
Key Takeaways
- Frame the problem as an MDP first. Get the state right (include velocities and history) before choosing an algorithm. A bad state breaks every method.
- Use Q-learning for discrete action spaces and small state spaces. Use PPO or SAC for continuous robot control. Do not force one tool onto every problem.
- Reward design is the real engineering work. Shape rewards, add time penalties, and verify with trajectory inspection. A reward bug looks exactly like an algorithm bug.
- Expect 1–5 million environment steps for a 7-DoF manipulation task. Budget compute and parallel workers accordingly.
- Always evaluate across multiple seeds and with held-out initial conditions. A single lucky run is not evidence.
Tomorrow
We take these foundations and apply them to real robot control: reward shaping for contact tasks, action space design, and how to structure a training loop that produces policies you can actually deploy.