Questions: Distractor Analysis and Item Optimization

Question 1

In a distractor frequency table, one incorrect option is chosen by 38% of the bottom quartile, 36% of the middle quartile, and 34% of the top quartile. What does this pattern indicate?

• A. A highly functioning distractor — it attracts respondents at all ability levels equally
• B. A non-functioning distractor — it fails to discriminate between ability levels and should be revised
• C. An inverse distractor — it attracts high-ability respondents more than low-ability ones
• D. An ideal distractor — equal selection rates mean it is neither too easy nor too hard to resist

Answer: (B)

A functioning distractor should show a gradient: chosen most by the bottom quartile, less by the middle, rarely by the top. Flat selection across ability groups — even if the option is chosen frequently — means the distractor is not discriminating. It could represent a concept that confuses everyone, an ambiguous option, or something unrelated to ability. This distractor is not 'working' to separate knowers from non-knowers and should be revised. Option A sounds tempting (it is attracting responses) but misses the point: the goal is differential attraction by ability level, not total attraction.

Question 2

A test developer finds that a distractor on a pharmacology exam is never chosen by any respondent — not even students in the bottom quartile. What is the most appropriate next step?

• A. Leave it — a low-chosen distractor proves the item is very discriminating
• B. Delete it and run a three-option item, since it is adding no information
• C. Revise it to represent a plausible misconception or common error that students with incomplete knowledge would make
• D. Lower the difficulty of the item by changing the correct answer to a more obvious option

Answer: (C)

A never-chosen distractor is a 'transparent foil' — everyone, regardless of ability, can immediately see it is wrong. It contributes nothing to the item's discriminating power. Deleting it (option B) is statistically defensible, but the better fix is revision (option C): replace it with a distractor that represents a genuine misconception or likely error, which requires content expertise. Simply running a three-option item (option B) reduces guessing probability but doesn't address the root issue if the remaining distractors are also weak. Changing the correct answer (option D) is never appropriate.

Question 3

A good set of distractors should be chosen equally often by high- and low-ability test takers, since equal selection rates prove the item is unbiased.

• T. True
• F. False

Answer: False

Equal selection rates are the hallmark of a *non-functioning* distractor, not an ideal one. A functioning distractor should attract low-ability respondents far more than high-ability ones — this differential is exactly what gives the item its discriminating power. An item where high-ability respondents choose wrong options at the same rate as low-ability respondents is either flawed (misleading to knowers) or measuring something other than the intended construct. 'Unbiased' in measurement means fair across demographic groups, not equal wrong-answer rates across ability levels.

Question 4

Revising a non-functioning distractor requires both statistical evidence that it is not working and content expertise to understand why and what to replace it with.

• T. True
• F. False

Answer: True

Statistics reveal *that* a distractor isn't functioning — the frequency table shows a flat or inverse gradient. But statistics cannot tell you what the distractor should say instead. Effective replacement requires knowing what misconceptions, common errors, or partially-correct ideas students actually hold about the tested content. This is where content expertise is irreplaceable: reviewing open-ended responses to similar questions, surveying students about what confuses them, or consulting subject-matter experts identifies the genuine traps that will discriminate knowers from non-knowers.

Question 5

Why can't statistical distractor analysis alone fix a non-functioning distractor — what role does content expertise play?

Model answer: Statistical analysis identifies that a distractor is not functioning (the frequency table shows it fails to attract low-ability respondents differentially), but it cannot identify what the distractor should say. Content expertise is required to diagnose why the distractor fails and to generate a replacement anchored in real learner misconceptions or errors. A plausible distractor must represent something a non-master would reasonably believe — and knowing what that is requires deep understanding of the construct and how students typically mislearn it.

The interaction between statistical feedback and content knowledge is the core of distractor revision. Statistics provide diagnostic signal (this option isn't working), while content expertise provides generative capacity (here's what would actually trap a non-master). Replacing a transparent foil with another random wrong answer doesn't help; it must represent a genuine conceptual error. This is why good test development requires domain experts, not just psychometricians, and why distractor revision is described as among the highest-leverage activities in improving test quality.