Training Robots: Reinforcement Learning, Domain Randomization and Sim-to-Real

Why simulation first

Reinforcement learning needs millions of trials, and physical robots break, wear and take wall-clock time. In a GPU-parallel simulator (Isaac Lab, MuJoCo), 4,096 robot instances practice simultaneously at faster-than-realtime — a year of falling down happens overnight, and nothing needs repair.

The reality gap and domain randomization

A policy trained in one pristine simulation memorizes that simulation's quirks. The fix is brutal and effective: randomize everything — friction, masses, motor strength, sensor latency, floor tilt, lighting. The policy that survives a thousand slightly-wrong worlds treats the real world as just one more variant.

# Minimal PPO walking-policy loop (Gymnasium-style pseudocode)
import torch
from torch.distributions import Normal

policy = ActorCritic(obs_dim=48, act_dim=12)        # 12 leg joints
opt = torch.optim.Adam(policy.parameters(), 3e-4)

for it in range(5_000):
    # --- collect rollouts from 4096 parallel, randomized sims ---
    obs = envs.reset(randomize=dict(
        friction=(0.4, 1.2), mass_scale=(0.8, 1.2),
        motor_strength=(0.85, 1.1), push_force=(0, 60)))   # N
    for t in range(24):
        mu, std, value = policy(obs)
        act = Normal(mu, std).sample()
        obs, rew, done, info = envs.step(act)
        buffer.add(obs, act, rew, value, done)

    # reward = forward velocity − energy − joint limits − falls
    adv = buffer.gae(gamma=0.99, lam=0.95)

    # --- PPO clipped update ---
    for batch in buffer.minibatches(8):
        ratio = (policy.logp(batch) - batch.logp_old).exp()
        loss = -torch.min(ratio * batch.adv,
                ratio.clamp(0.8, 1.2) * batch.adv).mean() \
               + 0.5 * policy.value_loss(batch)
        opt.zero_grad(); loss.backward(); opt.step()

Crossing over: the sim-to-real checklist

Match the action latency (insert the real system's delay into sim), train with observation noise, add random pushes, and deploy first with conservative torque limits. Teams that skip latency modeling get policies that oscillate on real hardware — the network learned to exploit a reaction time that doesn't exist.

Where world models change this

Hand-built simulators struggle with deformables, liquids, clutter. World models learned from video generate those hard scenarios directly — the training distribution stops being what an engineer could code and becomes what cameras have seen. Same PPO loop, radically richer worlds.