Questions: Within-Subjects Design Implementation and Counterbalancing

Question 1

A researcher completes a within-subjects study using full counterbalancing and concludes: 'Order effects are no longer a concern — counterbalancing takes care of them.' A colleague pushes back. Who is right?

• A. The researcher — counterbalancing distributes conditions evenly, eliminating order effects
• B. The colleague — counterbalancing prevents order effects from creating a confound between conditions, but they remain in the data and must still be modeled
• C. Both are partially right — counterbalancing eliminates carryover effects but not practice effects
• D. Neither — order effects cannot be addressed statistically and require a between-subjects design instead

Answer: (B)

Counterbalancing ensures order effects are distributed equally across conditions, so that no single condition always benefits from being first (or suffers from being last). This prevents order effects from creating a systematic confound. However, counterbalancing does not make order effects disappear: if practice improves performance regardless of condition, scores in later positions are genuinely elevated for everyone. Researchers should include ordinal position as a variable in the analysis to estimate and account for this — saying 'order effects are no longer a concern' overstates what counterbalancing accomplishes.

Question 2

Why do within-subjects designs typically require fewer participants than between-subjects designs to achieve equivalent statistical power?

• A. Within-subjects studies collect data from participants more quickly, reducing recruitment burden
• B. Within-subjects designs use more lenient statistical significance thresholds
• C. Individual differences that would inflate error variance in a between-subjects comparison cancel out because each participant appears in all conditions
• D. Within-subjects designs are only used for small effects, which require fewer participants to detect

Answer: (C)

The power advantage comes from error variance reduction, not from a statistical trick. In a between-subjects design, each condition contains a different group of people — the groups may differ in baseline ability, motivation, or any trait unrelated to the treatment, adding noise to the comparison. In a within-subjects design, the same people appear in every condition. Because individual differences affect all conditions equally, they cancel out of the within-condition comparison and are excluded from the error term entirely. Lower error variance means a given effect is easier to detect, so the same power is achievable with fewer participants.

Question 3

A practice effect — where performance improves simply from doing a task repeatedly — can occur in a within-subjects study even when conditions are presented in completely random order.

• T. True
• F. False

Answer: True

Practice effects are a type of order effect tied to experience accumulated over time, not to the specific sequence of conditions. Even with random ordering, a participant doing their 4th condition has more practice with the general task than a participant doing their 1st condition. Randomization prevents any one condition from systematically occupying a particular position, but it does not undo the general improvement that comes from doing the task multiple times. This is why simply randomizing order is insufficient — counterbalancing and explicit modeling of ordinal position are needed.

Question 4

Once counterbalancing is implemented in a within-subjects design, the researcher no longer needs to model or analyze ordinal position as a variable, since counterbalancing has already controlled for it.

• T. True
• F. False

Answer: False

Counterbalancing controls for systematic confounding — it ensures no condition always comes first or last — but it does not quantify or remove order effects from the data. Including ordinal position as a covariate in the statistical model lets researchers estimate how large the order effects are, verify that they are not interacting with the treatment, and produce cleaner estimates of the true treatment effect. Skipping this step means order effects are still present in the data; they have simply been balanced across conditions rather than accounted for statistically.

Question 5

How does a within-subjects design reduce error variance, and why doesn't counterbalancing fully solve the statistical problem of order effects?

Model answer: A within-subjects design uses each participant as their own control: individual differences in baseline ability, personality, and trait-level performance affect all conditions equally and therefore drop out of the within-person comparison, reducing error variance substantially. Counterbalancing ensures that systematic order advantages (e.g., being in position 1 vs. position 4) are distributed equally across conditions, preventing any condition from being artificially inflated or deflated. But counterbalancing does not make order effects disappear statistically — practice improvement and fatigue still alter scores across positions for all participants. To fully address order effects, researchers must include ordinal position as a variable in the analysis and estimate its magnitude directly.

The key distinction is between confounding (which counterbalancing addresses) and statistical contamination (which requires modeling). Counterbalancing prevents order effects from systematically favoring one condition; it doesn't remove their influence on score magnitudes. A researcher who counterbalances but doesn't model position has prevented a biased comparison between conditions, but has not measured how large the order effects are or verified that they don't interact with treatment — both of which are scientifically important questions.