A topic in the Open Knowledge Graph — a free, open map of 15,290 topics and the order to learn them in.
Generalizability theory extends classical reliability by examining how scores generalize across multiple measurement facets (items, raters, occasions, contexts). It decomposes variance into components from persons, each facet, and their interactions, providing nuanced reliability estimates for different testing conditions.
Design and conduct a simple G-study identifying facets and collecting data, then use results to conduct D-studies examining how test design decisions affect generalizability.
Generalizability theory replaces classical reliability entirely. Both frameworks are useful depending on context. Confusing G-coefficients with traditional reliability indices; g-coefficients address specific generalization questions.
Classical test theory (CTT) gives you one number for reliability: the ratio of true-score variance to observed-score variance. That number is powerful but opaque. It tells you how reproducible scores are, but not *why* they vary or *across what circumstances* they generalize. When a teacher rates students' essays, scores vary because students differ in writing ability — but they also vary because raters use the rubric differently, because some essay prompts are harder than others, and because all of these factors interact. CTT lumps all of this into a single "error" bucket. Generalizability theory (G-theory) opens that bucket.
The core move of G-theory is to treat the measurement situation as a design in the ANOVA sense, which you know from your work with one-way ANOVA. Just as ANOVA partitions total variance into between-groups variance and within-groups variance, G-theory partitions total score variance into components attributable to persons (the object of measurement), each facet of the measurement design, and their interactions. A facet is any systematic source of variation in the measurement conditions — raters, items, occasions, testing sites, and so on. Running a G-study (generalizability study) means collecting data across multiple conditions of each facet and estimating the variance each source contributes.
Suppose you run a G-study where 50 students each write two essays, and two raters score all essays. Your ANOVA-like decomposition might show: 40% of variance is attributable to persons (good — this is the signal), 15% to items (one prompt is harder than the other), 10% to raters (one rater scores more harshly), 20% to the person × item interaction (some students are relatively better on one prompt type), and 15% to the person × rater interaction (raters rank students inconsistently). Now you can ask targeted questions: which facet contributes most to measurement noise? How much can you reduce error by adding more raters vs. more items?
That targeted question is answered by the D-study (decision study). A D-study uses the variance components from the G-study to forecast how reliability — expressed as a generalizability coefficient, G — would change under different testing conditions. If you doubled the number of raters from 2 to 4, how much would G improve? If you added 3 more essay prompts? The D-study lets you optimize test design before actually running the test. The generalizability coefficient is analogous to Cronbach's alpha, but it is specific to the facets and number of conditions you are generalizing across — which is why G-coefficients answer specific generalization questions rather than providing a single context-free reliability number.
The practical takeaway is that CTT and G-theory are complementary tools. CTT is simpler and sufficient when you only care about overall score reproducibility and have a single source of error (items). G-theory becomes indispensable when your measurement involves multiple facets — any time raters, occasions, or varying contexts are part of the design — because only G-theory can tell you which facet is the bottleneck limiting reliability, and what redesigning the test around that bottleneck would cost or save.