Questions: Model-Based Reinforcement Learning

Question 1

An agent trained with model-based RL achieves near-perfect performance in simulation but fails catastrophically when deployed in the real environment. What is the most likely cause?

• A. The agent's policy network was too small to generalize beyond the training distribution
• B. Model-free updates are incompatible with simulated experience
• C. The learned world model contains errors that the agent exploited, producing a policy tuned to the model's mistakes rather than the real environment
• D. The agent did not collect enough real interactions before beginning to plan

Answer: (C)

This is the model exploitation problem — the defining failure mode of model-based RL. Planning algorithms are very good at finding optimal policies relative to whatever model they are given. If the model is imperfect, planning will find policies that look optimal in simulation by taking advantage of the model's inaccuracies. The agent becomes an expert at the simulation, not the real task. This is why uncertainty-aware models and hybrid approaches that periodically check real-world data are essential.

Question 2

Why is Dyna-Q more sample-efficient than pure Q-learning on the same task?

• A. Dyna-Q uses a larger neural network that generalizes better across states
• B. Dyna-Q skips the Q-learning update after real environment steps to save computation
• C. Each real interaction updates a world model, which then generates n additional simulated transitions used for Q-learning updates without further real-world steps
• D. Dyna-Q applies model-free updates exclusively on high-reward trajectories, filtering out uninformative experiences

Answer: (C)

Dyna-Q's insight is simple but powerful: after each real step, update the model with the observed transition, then run n additional Q-learning updates using transitions sampled from the model. Each real experience thus produces n+1 learning updates instead of one. With n=50, the agent extracts 51 policy improvements per real step. The model acts as a multiplier on data efficiency — the key mechanism behind model-based RL's sample efficiency advantage.

Question 3

Model-based RL is generally preferable to model-free RL because it learns from fewer real environment interactions.

• T. True
• F. False

Answer: False

Sample efficiency is the primary advantage of model-based RL, but it comes with a critical cost: model error. A learned world model is never perfectly accurate, and planning over an imperfect model can lead to model exploitation — policies optimized for the simulation rather than reality. Model-free methods are more robust because they learn directly from real experience and cannot exploit a model they don't have. The right choice depends on whether sample efficiency or robustness is more important for the task.

Question 4

In model-based RL, the agent can improve its policy by planning over simulated trajectories generated from a learned world model, without additional real-world interactions during the planning phase.

• T. True
• F. False

Answer: True

This is the core mechanism and main advantage of model-based RL. Once the world model has been learned from some initial real interactions, the agent can run arbitrarily many planning steps in simulation — dynamic programming, Monte Carlo rollouts, or model-free updates on imagined trajectories — all without touching the real environment. Real interactions are only needed to update the model itself, not for every policy improvement step.

Question 5

What is the central tension in model-based reinforcement learning, and how do modern approaches try to manage it?

Model answer: The central tension is between sample efficiency and model accuracy. A world model enables policy improvement through planning without costly real interactions, but any model errors can be exploited by the planner, yielding policies that fail in the real environment. Modern approaches address this through uncertainty-aware models (that refuse to plan confidently in unfamiliar regions), ensemble methods (that act conservatively where multiple models disagree), and hybrid architectures like Dreamer that ground predictions in periodic real-world data.

Understanding this tension is key to understanding the entire model-based RL literature. Every design choice — how often to collect real data, how to represent uncertainty, how many planning steps to perform — is a response to this tradeoff. The tradeoff also explains why model-free methods remain competitive despite lower sample efficiency: in complex environments, model accuracy is hard to guarantee, and robustness sometimes matters more than efficiency.