Proximal Policy Optimization (PPO)

---

Proximal Policy Optimization

Proximal Policy Optimization (PPO)

Proximal Policy Optimization (PPO) is an on-policy reinforcement learning algorithm designed to improve the stability and performance of policy gradient methods. PPO addresses key issues like high variance and large policy updates, which can destabilize training. The core idea is to limit the size of policy updates to ensure smooth and safe exploration of the policy space.

Algorithm

• The objective function in PPO is designed to maximize the expected cumulative reward while ensuring that the new policy does not deviate too far from the old one. The update rule is: $$\underset{\theta }{\max }{\mathbb{E}}_{{\pi }_{\theta }}\left[\min \left(r_t(\theta )\hat{A},\text{clip}(r_t(\theta ),1-ϵ,1+ϵ)\hat{A}\right)\right].$$

  • $r_t(\theta )=\frac{{\pi }_{\theta }(a|s)}{{\pi }_{\text{old}}(a|s)}$: Ratio of new to old policy probabilities at timestep $t$. This ratio quantifies how much the policy has changed.
  • $\hat{A}$: Advantage estimate, typically computed using a Generalized Advantage Estimation (GAE) to reduce bias and variance.
  • $ϵ$: A small hyperparameter that controls the extent to which the policy is allowed to change. Typical values range from 0.1 to 0.3.

Why is clipping used in PPO?

• Clipping is a key component of PPO that ensures stable updates by limiting the likelihood ratio $r_t(\theta )$, preventing overly large updates that could destabilize training.
• By clipping $r_t(\theta )$ within the range $[1-ϵ,1+ϵ]$, PPO avoids situations where the policy change is too large. This means that when the ratio is outside this range, the objective function will not continue to increase, thus maintaining stability.

  • This clip ensures that updates only happen when the ratio between the old and new policies is within a small, safe range.

PPO Update Rule Explained

• The objective function in PPO has two terms:
  1. The original objective: $r_t(\theta )\hat{A}$ — where we multiply the ratio by the advantage estimate. This encourages actions that lead to higher advantages (i.e., higher returns compared to the average).
  2. The clipped objective: $\text{clip}(r_t(\theta ),1-ϵ,1+ϵ)\hat{A}$ — where the ratio is clipped to lie within the range $[1-ϵ,1+ϵ]$. If the ratio exceeds this range, the objective function remains constant and does not increase further, preventing large policy updates.
• The min operator ensures that the final objective function is the smaller of the two terms, effectively ensuring that the policy updates do not deviate drastically from the previous policy.

Why does PPO work well?

• Efficiency: PPO strikes a balance between the sample efficiency of methods like TRPO (Trust Region Policy Optimization) and the simplicity of simpler policy gradient methods like REINFORCE.
• Stability: By clipping the probability ratios, PPO ensures that updates are stable, avoiding issues with overly large policy changes.
• Scalability: PPO can be applied to large-scale problems with high-dimensional state and action spaces, making it suitable for environments like robotics, games, and simulated environments.

PPO vs. Other Policy Optimization Methods

• TRPO (Trust Region Policy Optimization):

  • TRPO uses a constraint on the KL divergence between the old and new policies to ensure safe policy updates. However, TRPO is computationally expensive because it involves solving a constrained optimization problem.
  • PPO simplifies this by using clipping rather than a trust region, making it easier to implement and more efficient without sacrificing much performance.

• A3C:

  • While A3C uses multiple parallel workers to asynchronously update a global network, PPO is a more sample-efficient method that does not require parallelism and can achieve similar performance on many tasks.

• REINFORCE:

  • PPO outperforms REINFORCE because REINFORCE has higher variance and lacks the stability that comes with PPO’s clipping mechanism.

Key Components of PPO

1. Advantage Estimation:

   • Typically, a Generalized Advantage Estimation (GAE) is used to compute the advantage function, which combines Monte Carlo returns and temporal difference methods to reduce bias and variance.

2. Policy Updates:

   • The policy is updated by taking steps based on the gradient of the objective function. The key difference from basic policy gradients is the use of the clipped objective function, which provides stability during updates.

3. Entropy Regularization (Optional):

   • To encourage exploration and prevent premature convergence, PPO often adds an entropy term to the objective function. The entropy term penalizes overly deterministic policies, promoting exploration.
   • The entropy regularization is typically added as:

   $${\mathbb{E}}_{{\pi }_{\theta }}[\log {\pi }_{\theta }(a|s)].$$

Hyperparameters for PPO

• $ϵ$ (Clipping Parameter):

  • Controls how much the new policy can deviate from the old policy. A typical range is between 0.1 and 0.3.

• Learning Rate:

  • The step size used in the optimizer, which controls how much the policy is updated at each iteration.

• Batch Size:

  • The number of experience samples used in each update. PPO often uses mini-batch optimization with experience replay.

• Entropy Coefficient:

  • The weight of the entropy term in the objective function, which controls the exploration-exploitation tradeoff.