PPO & Baseline Implementation

Policy Gradient Methods

Policy gradient methods are a class of policy-based reinforcement learning algorithms that aim to learn a policy directly by optimizing a performance objective.

They use gradient ascent to update the policy parameters in the direction of the gradient of the expected reward.

The policy gradient theorem provides an unbiased estimate of the gradient of the expected reward, allowing us to use gradient-based optimization methods to learn the policy.

A key challenge in policy gradient methods is that they can be very sensitive to hyperparameter choices. Small changes in hyperparameters can lead to large changes in performance, making them difficult to tune.

Proximal Policy Optimization (PPO)

PPO is a policy gradient algorithm that aims to address the instability and sensitivity to hyperparameter choices of standard policy gradient methods.

It achieves this by constraining the policy updates to stay close to the previous policy, preventing large and potentially destructive updates.

The main idea behind PPO is to use a surrogate objective function that is easier to optimize than the original objective function. This surrogate objective function is designed to be "close" to the original objective function, ensuring that policy updates stay within a reasonable range.

How PPO Works

To achieve this, PPO uses a clipped surrogate objective function. This function penalizes large policy updates, encouraging the policy to stay close to the previous policy.

The algorithm works as follows:

1. Sample trajectories from the current policy.
2. Compute the advantages for each state-action pair.
3. Compute the clipped surrogate objective function.
4. Optimize the objective function using gradient ascent.
5. Update the policy using the optimized objective function.

Continual-Bench is a unified benchmark for continual learning. It provides a set of robotic tasks with varying levels of difficulty and complexity, making it suitable for studying different aspects of continual learning. The benchmark allows researchers to evaluate their continual learning algorithms on a standardized set of tasks, enabling fair comparison between different methods.

The benchmark provides different tasks for evaluating continual learning algorithms. Some of the key tasks include:

• Open Faucet: Opening a faucet by adjusting the handle.
• Adjust Volume: Adjusting the volume of a speaker by turning the knob.
• Turn Knob: Turning a knob to control a mechanism.
• Flip Lid: Flipping a lid open or closed.

Each task has different characteristics, such as:

• Observation space: The size and type of observations vary across tasks.
• Action space: The action space also varies across tasks, with some tasks having continuous actions and others having discrete actions.
• Reward function: The reward function is designed to encourage the agent to achieve the task goal.

Overall, Continual-Bench provides a comprehensive platform for studying continual learning in robotics, allowing researchers to evaluate their algorithms on a standardized set of tasks and compare their performance on different aspects of continual learning.