Assignment Instructions
Recommendation: Submit your assignment after forking the repository vishesh-kumar-singh/recall-e-assignments. In the week-1 folder, create a folder named [name]-[roll] (for example, vishesh-241176). In that folder, put task-1 as a PDF, task-2 python files, and a README.md with instructions to run your code along with your analysis section. Plots should be rendered in the README.
1. Formulating your world as MDP
- Design an MDP: Choose a simple daily scenario (e.g., deciding whether to study, sleep, or hang out; or navigating between specific campus buildings). Formally define the State space ($\mathcal{S}$), Action space ($\mathcal{A}$), Transition probabilities ($\mathcal{P}$), and Rewards ($\mathcal{R}$).
- Manual Calculation: For a highly simplified 3-state version of your scenario, perform Value Iteration by hand for exactly two steps, assuming a discount factor $\gamma = 0.9$.
2. The Epsilon-Decay Challenge
- Implementation: Implement tabular Q-learning on the
CliffWalking-v0 environment in Gymnasium.
- Experimentation: Train three separate agents, each using a different exploration strategy:
- Constant high exploration ($\epsilon = 0.5$).
- Constant low exploration ($\epsilon = 0.05$).
- Decaying exploration (start at $\epsilon = 1.0$ and decay linearly or exponentially to $0.01$).
- Analysis: Plot the sum of rewards per episode for all three agents on a single graph. Which agent learns a safe path the fastest? Which one ultimately finds the most optimal path, and why is there a difference?