BCS Club · Summer 2026

Recall-E
|

Exploring continual learning in reinforcement learning for robotic control by teaching agents to remember while they learn.

View Timeline → Read Paper

// Single-Task Baseline

Independent PPO Models

Three different models, each trained independently on a single specific task.

Door opening task

Door Opening

PPO agent trained exclusively on door opening

Peg insertion task

Peg Removal

PPO agent trained exclusively on peg removal

Window opening task

Window Opening

PPO agent trained exclusively on window sliding

// The Solution

Continual Learning Agent

One single model trained sequentially across multiple tasks without catastrophic forgetting.

Sequential Multi-Task Mastery

Our unified agent performing multiple tasks continuously, leveraging Shallow World Models.

// What is this?

About The Project

Bridging the gap between robotic control and lifelong learning

Robotic Controls

Moving beyond traditional physics-based control systems to Reinforcement Learning, where algorithms discover optimal policies through exploration and interaction with the environment.

Continual Learning

Addressing catastrophic forgetting, which is the tendency of agents to lose previous knowledge when learning new tasks. Our goal is to build agents that remember everything they've learned.

Shallow World Models

Using model-based approaches with Model Predictive Control (MPC) and Cross-Entropy Method (CEM) to learn environment physics and plan optimal action sequences.

PPO Baseline

Implementing Proximal Policy Optimization as our model-free baseline, training on sequential tasks to demonstrate catastrophic forgetting before comparing with our approach.

Benchmarking

Using the Continual-Bench environment with MuJoCo physics to rigorously evaluate retention across sequential tasks and measure how well agents preserve old skills.

Project Goals

Demonstrate that shallow world models with MPC achieve significantly better task retention than PPO when trained sequentially on multiple robotic control tasks.

// Learning Material

Current Resources

All the resources covered till now

Inspiration Paper: ICML 2025

"Shallow World Models for Continual Reinforcement Learning" is the foundation of our approach to solving catastrophic forgetting.

Read on arXiv →

Reference Books

For almost every concept in Reinforcement Learning, these books are a must read. I'd suggest you to go through Grokking Deep Reinforcement Learning first as it's more intuitive and then move on to Sutton and Barto for a more mathematical understanding.

Grokking Deep Reinforcement Learning → Sutton and Barto →
Week 0

Python & ML Foundations

Full resource list: Python basics, NumPy, Matplotlib, Regression, Neural Networks, PyTorch.

 Week 1

Reinforcement Learning Foundations

Full resource list: Markov Decision Process, Q-Learning, and setting up RL environments.

 Week 2

Deep Q-Networks & Tuning

Full resource list: Deep Q-Learning, DQN, Double DQN, and advanced variants including Rainbow DQN.

 Week 3

PPO & Baseline Implementation

Full resource list: Proximal Policy Optimization (PPO) theory, setting up Continual-Bench & MuJoCo, and single-task arm training.

 Week 5-8

Continual Learning & World Models

Topics covered: World Models, Shallow World Models, Cross-Entropy Method (CEM), and Follow the Leader method.

// Insights

Student Blogs

Weekly deep dives into RL concepts written by the team

Week 1

Teaching Machines to Learn from Experience: MDPs and Q-Learning

An intuitive walkthrough of Reinforcement Learning, Markov Decision Processes, the Bellman equation, and Q-Learning — with code along the way.

By Tejas, Saurav May 23, 2026
Week 2

Week 2 Blog: DQN, DDQN, and PER

A deep dive into modern value-based Reinforcement Learning, exploring DQN, Double DQN, and Prioritized Experience Replay.

By Prakhar, Priyanka, Harpal, Hemendra June 08, 2026
Paper Summary

RECALL-E: Continual Reinforcement Learning via Online World Models

A technical summary of the RECALL-E framework paper, laying the foundation for our upcoming implementation and final report.

By Chaitanya, Nishant, Adrija August 12, 2026

// Week by Week

Project Timeline

9 weeks of progressive learning and implementation (Week 0–8)

0
Week 0: Prerequisites ✓ Completed

Python & ML Foundations

Getting up to speed with Python, NumPy, and fundamental ML before diving into RL.

  • Python basics, classes, OOPs
  • NumPy, Matplotlib
  • Linear & Logistic Regression
  • Neural Networks & PyTorch
View Resources →
1
Week 1 ✓ Completed

Foundations of RL & Q-Learning

Understanding MDPs, exploration strategies, and tabular methods.

  • Markov Decision Processes
  • Epsilon-greedy and exploration strategies
  • Tabular Q-Learning and its limitations
  • Setting up Gymnasium environments
View Resources → View Assignment →
2
Week 2 ✓ Completed

Deep Q-Networks & Hyperparameter Tuning

Introducing model-free deep RL with function approximation.

  • Introduction to DQN (value-based method)
  • Double DQN improvements
  • Hyperparameter tuning strategies
  • Implementing DQN for game-playing agents
View Resources → View Assignment →
3
Week 3 ✓ Completed

PPO & Baseline Implementation

Policy gradient methods and first simulation results.

  • Proximal Policy Optimization theory
  • Setting up Continual-Bench & MuJoCo
  • Training arm for single task
  • First simulation run
View Resources → View Assignment →
4
Week 4: Mid Eval ✓ Completed

Introduction to Continual Learning

The core problem: catastrophic forgetting in sequential task learning.

  • The Continual Learning problem
  • Multi-task PPO demonstrating forgetting

Mid Evaluation

Q-Learning, DQN, PPO implementation complete. Single-task arm simulation. Catastrophic forgetting demonstrated.

5
Week 5 ✓ Completed

Model-Based Algorithms

Shifting from model-free to model-based RL approaches.

  • Introduction to model-based RL
  • Deep World Models
  • Shallow World Model architecture
6
Week 6 ✓ Completed

Model Predictive Controls

Planning and control with learned dynamics models.

  • MPC + Cross-Entropy Method
  • The main algorithm architecture
  • Shallow world models with MPC for single task
7
Week 7 ✓ Completed

Continual Learning with Shallow World Models

The main event: comparing retention across approaches.

  • Extending to multi-task sequential learning
  • Measuring and comparing retention
  • Shallow world models vs PPO analysis
8
Week 8 ✓ Completed

Final Analysis & Experimentation

Hyperparameter experiments and final report.

  • Final report formulation

End Evaluation

Multi-task arm simulation with shallow world models. Demonstrated better retention. Full analysis report.

// The People

Mentors

Brain & Cognitive Science Club, IIT Kanpur

Vishesh Kumar Singh

Vishesh Kumar Singh

Mentor

Aditi Mishra

Aditi Mishra

Mentor

Aayush Gajeshwar

Aayush Gajeshwar

Mentor

Kushagra Chandra

Kushagra Chandra

Mentor

// Findings

Results & Report

Experimental results and final analysis

Coming Soon

Results, analysis, and the final report will be published here once the project concludes. In the meantime, check out our technical summary of the inspiration paper!

Read the Paper Summary →