Introducing model-free deep RL with function approximation.
Till the last week, we depended a lot on videos and blogs to learn things, but from this week onwards we would push you towards actually reading research papers.
Q-Learning also has an important limitation.
The algorithm stores values in a table. This works fine for small environments like GridWorld or FrozenLake, but becomes impossible for large state spaces.
Imagine trying to store Q-values for:
The table would become unimaginably large.
This limitation eventually leads us toward one of the biggest breakthroughs in RL:
Where neural networks are used to approximate Q-values instead of storing them explicitly in a table.
Instead of maintaining a huge lookup table, DQN uses a neural network which takes the current state as input and predicts the Q-values for all possible actions. This allows the agent to generalize between similar states instead of memorizing every possible situation independently.
However, training neural networks inside RL is far more unstable than standard supervised learning. Unlike image classification where the dataset remains fixed, RL agents continuously generate their own training data while learning. This makes the targets constantly change during training.
To stabilize learning, DQN introduced two major ideas:
These simple but powerful ideas allowed Deep RL to scale to much more complex environments and eventually led to major breakthroughs in modern AI.
DQN became historically important after DeepMind showed that an agent could learn directly from raw game pixels and achieve human-level performance on several Atari games.
This was one of the first major demonstrations that deep neural networks and Reinforcement Learning could work together successfully.
Even though the core idea still comes from Bellman’s Equation and Q-Learning, replacing tables with neural networks allowed RL to move beyond small toy environments into large high-dimensional problems.
This breakthrough laid the foundation for much of modern Deep Reinforcement Learning.
You can refer to the actual paper: https://arxiv.org/pdf/1312.5602 or refer to Deep Q-Networks Explained https://www.youtube.com/watch?v=x83WmvbRa2I!
There were many advanced made in standard DQN for example:
Combining these 3 we get Rainbow DQN.
You may find the following blog interesting which covers all of these topics along with Q-Learning and DQN, written by one of our mentors Vishesh https://medium.com/@thevishesh16/q-learning-to-rainbow-dqn-the-complete-evolution-of-q-learning-complete-theory-with-8f25cfb31b01
We would not expect that you'd be able to master all of these topics by the end of this week, but I would expect you to atleast learn DQN, DDQN and Prioritized Experience Replay and should be having a basic intuition of the others, you shoul atleast be able to tell their functionalities and the problems they are trying to solve.