Questions: Temporal Difference Learning

Question 1

An agent in state s takes an action, receives reward r = -5, and transitions to state s'. The current value estimates are V(s) = 10 and V(s') = 8, with discount γ = 0.9 and learning rate α = 0.1. What is the TD(0) update to V(s)?

• A. V(s) ← 10 + 0.1 × (−5 + 0.9 × 8 − 10) = 10 + 0.1 × (−7.8) = 9.22
• B. V(s) ← 10 + 0.1 × (−5 − 10) = 8.5
• C. V(s) ← −5 + 0.9 × 8 = 2.2
• D. V(s) ← 10 − 0.1 × 8 = 9.2

Answer: (A)

The TD error is δ = r + γV(s') − V(s) = −5 + 0.9×8 − 10 = −5 + 7.2 − 10 = −7.8. The update is V(s) ← V(s) + α·δ = 10 + 0.1×(−7.8) = 9.22. The key insight is that δ measures the gap between what the agent expected (V(s) = 10) and what it now believes based on one step of observation (r + γV(s') = 2.2). The large negative TD error nudges the estimate downward, but only by α = 10% of the gap.

Question 2

Compared to Monte Carlo methods, TD(0) has lower variance but higher bias in its value estimates. What is the source of this bias?

• A. TD(0) uses a smaller learning rate, which causes estimates to converge to a slightly different value
• B. TD(0) updates use V(s') — itself just an estimate — rather than a true observed return, so errors in V(s') propagate into the update for V(s)
• C. TD(0) discounts future rewards with γ < 1, which systematically undervalues long-horizon states
• D. TD(0) only updates the immediately visited state, missing contributions from earlier states in the trajectory

Answer: (B)

This is the bootstrapping bias. Monte Carlo methods use the actual observed return (unbiased, because it is a real sample of the true value) but have high variance (because the full return is noisy). TD(0) uses r + γV(s') as the update target, but V(s') is itself an estimate — possibly wrong. If V(s') is too high, the update for V(s) will be pulled too high as well. This error in the estimate propagates through the value function. The tradeoff is intentional: bootstrapping reduces variance (you only need one step of experience) at the cost of introducing this estimation bias.

Question 3

TD learning can update value estimates after every single step, without waiting for an episode to complete.

• T. True
• F. False

Answer: True

This is the defining advantage of TD methods over Monte Carlo. The TD update only requires the immediate reward and the estimated value of the next state — both available after a single step. Monte Carlo methods must wait until the episode ends to observe the true return. For continuing tasks (no natural episode end) or long episodes, this online learning capability makes TD methods far more practical. The cost is bootstrapping bias, but the benefit is immediate learning from every transition.

Question 4

TD(λ) with λ = 1 is equivalent to TD(0) because eligibility traces decay to zero after one step when λ = 1.

• T. True
• F. False

Answer: False

This reverses the relationship. TD(λ) with λ = 0 gives TD(0) — updates use only the immediately next state's value, with no trace memory. TD(λ) with λ = 1 gives Monte Carlo — eligibility traces decay by λ = 1 per step (no decay), so all previously visited states receive full credit, equivalent to using the complete episode return. The λ parameter interpolates between TD(0) (λ=0, most bootstrapping, lowest variance, highest bias) and Monte Carlo (λ=1, no bootstrapping, zero bias, highest variance).

Question 5

What is bootstrapping in the context of TD learning, and why does it allow TD methods to learn online while also introducing bias?

Model answer: Bootstrapping means updating one value estimate using another value estimate rather than using a true observed outcome. In TD learning, V(s) is updated using r + γV(s') as the target, where V(s') is itself an estimate of the state value — not a real return from completing the episode. Because V(s') is immediately available after one step (unlike the true return, which requires the episode to finish), bootstrapping enables online updates after every transition. The bias arises because V(s') may be inaccurate; if the value function is poorly estimated early in training, these errors propagate into updates for V(s). As training progresses and V(s') improves, the bias decreases.

The analogy is to self-referential learning: you are updating your belief using your own beliefs as evidence, rather than waiting for external ground truth. This is efficient but initially unreliable. Monte Carlo avoids bootstrapping by always waiting for real ground truth (the full return), getting unbiased but high-variance estimates. Most practical RL algorithms — Q-learning, SARSA, DQN — use TD bootstrapping because online learning and sample efficiency outweigh the manageable bias.