Questions: Algorithms for Computerized Adaptive Testing

Question 1

After 5 items, a CAT algorithm estimates an examinee's ability at θ = 1.5. Which item should it select next?

• A. The item with the highest difficulty parameter in the bank, regardless of θ
• B. The item whose information function peaks nearest θ = 1.5
• C. A randomly selected item to prevent systematic bias
• D. The item that was most informative for a previous examinee with similar ability

Answer: (B)

Maximum information item selection works by identifying the item that provides the most statistical information at the current ability estimate. Each item's information function peaks at the ability level where the item is most discriminating — where a small difference in ability produces the largest difference in the probability of a correct response. Selecting the item whose peak is at θ = 1.5 means asking the most diagnostic possible question given what the test knows so far. Option A (highest difficulty) ignores where the item is informative; a very hard item has its information peak at a high θ and provides almost nothing at θ = 1.5.

Question 2

Why does a pure maximum-information algorithm create practical problems for real-world CAT programs?

• A. It makes real-time ability estimation computationally intractable on modern hardware
• B. It repeatedly selects the same small set of highly informative items, enabling memorization and score inflation
• C. It systematically ignores the Bayesian prior over the ability distribution
• D. It consistently underestimates ability at the high and low ends of the scale

Answer: (B)

The maximum information algorithm, applied without constraints, will repeatedly select whichever items are most informative — and since item information functions are stable, this tends to be the same items across examinees. Examinees who take the test on different days share experiences, enabling item content to spread and allowing coached candidates to inflate scores. Real CAT programs therefore impose exposure controls (limiting how often any item is administered) and content constraints (ensuring coverage of specified topics). This transforms item selection into a constrained optimization problem, not pure information maximization.

Question 3

A CAT system that selects items purely by maximum information — with no content or exposure constraints — is fully optimized for operational testing use.

• T. True
• F. False

Answer: False

Pure maximum-information selection is theoretically elegant but operationally flawed. Without constraints, a small set of highly informative items gets selected repeatedly, making those items vulnerable to memorization and enabling score inflation. Content constraints are also needed to ensure the test covers the full domain as specified by the test blueprint — a test that happens to measure only a subset of content is not a valid measure of the full construct, regardless of how efficient its item selection is. Real CAT systems solve a constrained optimization problem that balances information, content representation, and item security.

Question 4

A well-calibrated Bayesian prior over the ability distribution can improve early item selection in a CAT by preventing the algorithm from committing fully to a badly wrong initial ability estimate.

• T. True
• F. False

Answer: True

When a CAT begins, the algorithm has no response data to work with and must start with some ability estimate. A Bayesian approach incorporates a prior — typically based on the population distribution — that pulls early estimates toward the typical range. This prevents the algorithm from chasing a fluke first response down an extreme ability level, which would select very easy or very hard items that contribute little useful information for typical examinees. As responses accumulate, the likelihood from the data comes to dominate the prior, and the two approaches converge. The prior is most valuable in the first few items.

Question 5

Explain why an item provides maximum statistical information at the ability level where its characteristic curve is steepest. How does the CAT algorithm exploit this to achieve measurement efficiency?

Model answer: An item's information function peaks where its characteristic curve is steepest because that is the region where a small difference in ability produces the largest difference in the probability of a correct response — making each response maximally diagnostic about the examinee's true ability. A CAT exploits this by maintaining a running ability estimate and always selecting the item whose information peaks nearest that estimate, effectively asking the most discriminating question possible at each step. Because every item is targeted to the individual's current estimated ability, a CAT achieves the same measurement precision as a much longer fixed-form test.

Technically, information I(θ) = P'(θ)² / [P(θ)(1−P(θ))], which is maximized where the slope P'(θ) is large relative to response uncertainty. The CAT's efficiency gain comes from targeting: a fixed test must be designed for the 'average' examinee, so it provides sub-optimal information for people far from the mean. A CAT dynamically adjusts to each person, keeping them at the most informative region of their personal item bank at every step — which is why 20 adaptive items can outperform 60 fixed items in measurement precision.