Sim2Real

Train in simulation, deploy in real world (with real-time adaptation)

Why simulators for robot learning?

• most RL-based algos are very sample inefficient
• They are cheap/fast/scalable

Problems of Sim2Real

• non-parametric mismatches (simulator doesn’t consider some effects at all)
  • complex aerodynamics, fluid dynamics, tire dynamics, etc
• Parametric mismatches (simulator uses different parameters than real)
  • robot mass/friction,etc

Domain Randomization

• Randomize $e$ in $x_{t+1}=f(x_t,u_t,e)$
• Train a single RL policy $\pi (x)$ that works for many $e$
  • Approximation of robust control

Learning to Adapt

• Randomize $e$ in $x_{t+1}=f(x_t,u_t,e)$
• Train an adaptive RL policy $\pi (x,e)$ that works for many $e$
  • approximation of adaptive control
• Issue! $e$ is often unknown in real
  • Solution! Learning from a privileged teacher
    • Sim: First Train a teacher policy with privileged information $\pi (x,e)$
    • Sim: Student policy ${\pi }_s(x,\text{available info in the real})$ learns from $\pi (x,e)$
    • Real: Deploy student policy ${\pi }_s(x,\text{available info in the real})$
  • Basically an Imitation Learning problem