Imitation Learning

MDP (Markov Decision process)

Definitions:

• $S$: State space, $s_t\in S$: state at time $t$
• $A$: Action space, $a_t\in A$: action at time $t$
• $p$: Transition probability, $s_{t+1}\sim p(\cdot |s_t,a_t)$
• $r$: Reward function, $r:S\times A\to \mathbb{R}$

Goal: Learn a policy $\pi (a_t|s_t)$.

POMDP (Partially Observed MDP)

Additional Definitions:

• $O$: Observation space, $o_t\in O$: observation at time $t$
• $h$: Observation model, $o_t\sim h(\cdot |s_t)$

Goal: Learn a policy $\pi (a_t|o_t)$.

Imitation Learning

Idea

• collect expert data (observation/state and action pairs)
• Train a function to map observations/states to actions

Dataset Aggregation (DAgger)

1. Process:

   • Start with expert demonstrations.
   • Train policy ${\pi }_1$ via supervised learning.
   • Run ${\pi }_1$, query the expert to correct mistakes, and collect new data.
   • Aggregate new and old data, retrain to create ${\pi }_2$.
   • Repeat the process iteratively.

2. Advantages:

   • Reduces cascading errors.
   • Provides theoretical regret guarantees.

3. Limitations:

   • Requires frequent expert queries.

IL with Privileged Teachers

• It can be hard to directly learn the policy ${\pi }_{\theta }(a_t,o_t)$ especially if $o_t$ is high-dimensional

Obtain a "privileged" teacher ${\pi }_p(a_t,o_t)$

• $p_t$ contains “ground truth” information that is not available to the “students”
• Then use ${\pi }_p(a_t,o_t)$ to generate demonstrations for ${\pi }_{\theta }(a_t,o_t)$

Example

• Stage 1: learn a “privileged agent” from expert
  • It knows ground truth state (traffic light, other vehicles’ pos/vel, etc)
• Stage 2: a sensorimotor student learns from this trained privileged agent

This is especially useful in simulation, because we know every variable’s value in sim. So the privileged teacher learns from that, but the student only learns from stuff it can directly see/measure.

• privileged teacher is usually trained by PPO

Variants

• Student learning in the latent space: Adapting Rapid Motor Adaptation for Bipedal Robots
• Student learning to predict rays: [2401.17583] Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion

Deep Imitation Learning with Generative Modeling

What is the problem posed by generative modeling?

• Learn: learn a distribution $p_{\theta }$ that matches $p_{d}ata$
• Sample: Generate novel data so that $x_{new}\sim p_{\theta }$

For robotics, we want our $p_{data}$ to be from experts. There are three leading approaches:

GAN + Imitation Learning $\Rightarrow$ Generative Adversarial Imitation Learning (GAIL)

• Sample trajectory from students
• Update the discriminator, which is aimed at classifying the teacher and the student
• Train the student policy which aims to minimize the discriminator’s accuracy

VAE + IL $\Rightarrow$ Action Chunking with Transformers

• Based on CVAE (conditional VAE)
• Encoder: expert action sequence + observation → latent
• Decoder: latent + more observation → action sequence prediction
• Key: action chunking + temporal ensemble

Diffusion + IL $\Rightarrow$ Diffusion Policy

• Diffusion Policy