Sim2Real

Train in simulation, deploy in real world (with real-time adaptation)

Why simulators for robot learning?

• Most RL-based algos are very sample inefficient
• They are cheap/fast/scalable/safe/labeled

Problems of Sim2Real

• Non-parametric mismatches (simulator doesn’t consider some effects at all)
  • complex aerodynamics, fluid dynamics, tire dynamics, etc
• Parametric mismatches (simulator uses different parameters than real)
  • robot mass/friction,etc

Domain Randomization

• Randomize $e$ in $x_{t+1}=f(x_t,u_t,e)$
• Train a single RL policy $\pi (x)$ that works for the whole distribution of $e$
  • Approximation of robust control
• What is randomized?
  • Physics parameters (mass, gravity, friction, etc)
  • Sensor noise (camera blur, pixel noise, quantization, etc)
  • Rendering (lighting, textures, backgrounds)

Learning to Adapt (via Privileged Information)

• Randomize $e$ in $x_{t+1}=f(x_t,u_t,e)$
• Train an adaptive RL policy $\pi (x,e)$ that works for many $e$
  • approximation of adaptive control
• Issue! $e$ is often unknown in real world
  • Solution! Learning from a privileged teacher
    • Sim: First Train a teacher policy with privileged information $\pi (x,e)$
    • Sim: Student policy ${\pi }_s(x,\text{available info in the real})$ learns from $\pi (x,e)$
    • Real: Deploy student policy ${\pi }_s(x,\text{available info in the real})$
  • Basically becomes an Imitation Learning problem