Policy Gradient

Policy-based methods

• Directly learn to approximate ${\pi }^{\ast }$ without having to learn a value function
• So we parameterize the policy using a neural network.
• $\boxed{{\pi }_{\theta }(s)=P(A|s;\theta )}$

Goal

Maximize the performance of the parameterized policy using gradient ascent

Pros and Cons of Policy Gradient Methods

Pros

• Directly estimate policy without storing additional data
• Can learn a stochastic policy
  • Don’t need to implement an exploration/expolitation tradeoff
• Much more effective in high-dimensional (continuous) action spaces
• Better convergence properties Cons
• They converge to a local max instead of a global max
• Can take longer to train
• Can have higher variance

Policy Gradient Algorithm

1. Initialize the policy parameters $\theta$ randomly.

   • Define the policy ${\pi }_{\theta }(s)$ as a parameterized probability distribution over actions.

2. Generate an episode by following the current policy ${\pi }_{\theta }$:

   • Observe states $s_t$, take actions $a_t$, and receive rewards $r_t$ for $t=1,2,\dots ,T$.

3. Calculate the cumulative reward for each step:

   $$G_t=\sum _{k=t}^T{\gamma }^{k-t}{r}_k,$$

   where $\gamma$ is the discount factor.

4. Compute the policy gradient:

   $${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(s,a)G_t\right].$$

   • Estimate this gradient by sampling episodes and using Monte Carlo methods.

5. Update the policy parameters using gradient ascent:

   $$\theta \leftarrow \theta +\alpha {\nabla }_{\theta }J(\theta ),$$

   where $\alpha$ is the learning rate.

6. Repeat steps 2–5 until convergence or for a predefined number of iterations.

Why does the policy gradient work?

• The gradient ${\nabla }_{\theta }J(\theta )$ pushes the policy to assign higher probabilities to actions that lead to higher rewards.
• This allows the policy to improve iteratively by maximizing the expected cumulative reward.

Variants of Policy Gradient

• REINFORCE: A simple implementation using Monte Carlo estimates for $G_t$.
• Actor-Critic: Combines a parameterized policy (actor) with a value function (critic) to reduce variance in gradient estimation.
• Proximal Policy Optimization (PPO): Regularizes policy updates to ensure stability and prevent large changes in the policy.

Policy Gradient Theorem

Policy Gradient Theorem

1. Objective Function:

   • The goal is to maximize the expected cumulative reward under a parameterized policy ${\pi }_{\theta }$:

   $$J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right].$$

   • Here, $\gamma \in [0,1]$ is the discount factor.

2. Policy Gradient Theorem:

   • The gradient of $J(\theta )$ is given by:

   $${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(s,a)Q^{{\pi }_{\theta }}(s,a)\right].$$

   • This connects the expected reward to the gradient of the policy’s log-probability, weighted by the action-value function $Q^{{\pi }_{\theta }}(s,a)$.

3. Simplified Gradient Estimate:

   • Using the cumulative reward $G_t$ as an estimate of $Q^{{\pi }_{\theta }}(s,a)$:

   $${\nabla }_{\theta }J(\theta )\approx {\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(s,a)G_t\right].$$

Key Advantages of Policy Gradient Theorem

• Directly optimizes the policy without needing a value function.
• Works well for stochastic or continuous action spaces.
• Facilitates modeling of complex, parameterized policies.

Variants and Improvements

• Baseline Subtraction:

  • Reduce variance by subtracting a baseline $b(s)$ from $G_t$:

  $${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}\left[{\nabla }_{\theta }\log {\pi }_{\theta }(s,a)(G_t-b(s))\right].$$

  • Common choice for $b(s)$ is the value function $V^{{\pi }_{\theta }}(s)$.

• Actor-Critic Methods:

  • Use a critic to estimate $Q^{{\pi }_{\theta }}(s,a)$ or $G_t$ to reduce variance and stabilize training.