How Generalisability Theory Works | Separate Person, Task, Rater and Occasion Variance Before Trusting a Score

eduKateSG Learning Node Series · 0249

A student writes one essay. Two teachers score it. One gives 14/20. The other gives 17/20.

That looks like a rater problem. But now give the student a second essay prompt. Both teachers score it lower. Give the same student a third prompt next week and the score rises again. Suddenly the question is larger than whether two teachers agree.

How much of the variation belongs to the learner? How much belongs to the particular task? How much belongs to the rater? How much belongs to the occasion? And how much appears only because a particular learner met a particular task, rater or moment?

Generalisability Theory—often written Generalizability Theory in American sources and abbreviated G Theory—is a measurement framework for answering exactly that kind of question. Instead of treating all unwanted variation as one undifferentiated lump called error, it decomposes observed score variation into the sources that a measurement design makes visible.

That changes the practical question from “Is this test reliable?” to something more useful: Reliable for what decision, over which tasks, raters, occasions and conditions—and what measurement design would make the intended interpretation more dependable?

Quick answer

Generalisability Theory works in two linked stages. A G study estimates how much observed score variation comes from the object of measurement—usually learners—and from measurement facets such as items, tasks, raters or occasions, including their interactions. A D study then uses those estimated variance components to ask how dependable scores would be under a proposed operational design: more tasks, fewer raters, another occasion, a different crossed or nested structure, or a decision focused on ranking versus an absolute standard.

The result is not one universal reliability number. It is a model of where measurement uncertainty comes from and how design choices change it.

The owned reader job

This Learning Node owns the question: when a score can vary because of several measurement conditions at once, how do we separate those sources and design a more dependable measurement procedure?

It does not replace How Scoring Rubric Validation Works, which focuses on whether criteria, descriptors and raters support an intended interpretation of performance. It does not replace How Score Reports Work, which owns communication of results and uncertainty, or How Score Interpretation and Use Arguments Work, which owns the broader claim chain from observation to decision.

Generalisability Theory sits underneath those uses when the measurement procedure itself contains multiple sources of variation that matter.

Why ordinary reliability can hide the problem

Classical approaches often begin with a useful abstraction:

Observed score = true score + error.

That is powerful when one main error source dominates or when the purpose is narrow. But educational measurement frequently has more structure. A classroom oral examination may depend on learner, question, examiner and day. A writing assessment may depend on learner, prompt and rater. A practical laboratory assessment may depend on learner, station, assessor and equipment conditions. A school observation may depend on teacher, lesson, observer and occasion.

If all those sources are collapsed into one error term, the overall reliability coefficient tells us that uncertainty exists but not where it comes from or which design change would reduce it.

Generalisability Theory makes the structure explicit.

Objects of measurement and facets

The first design decision is conceptual. What is the object of measurement? In a student assessment, it is usually the learner. In another study, it could be a school, teacher, programme, lesson or product.

Then identify the facets: conditions across which the measurement could vary and across which we may wish to generalise.

Facet	Example question
Items or tasks	Would the learner look equally strong on another defensible sample of questions?
Raters	Would another qualified scorer reach a similar judgement?
Occasions	Would the score remain similar on another appropriate day?
Methods	Would another response format represent the same capability similarly?
Stations or contexts	Would performance travel to another sampled setting?

A facet is not automatically “noise”. Task difficulty can be a legitimate part of the assessment universe. Rater severity can sometimes be adjusted or controlled. Occasion variation may be relevant if the intended claim concerns stable capability, but less problematic if the claim explicitly concerns performance on that occasion.

Measurement error is therefore defined relative to the intended interpretation.

Variance components: where the score variation lives

Imagine every learner answers every task and every response is scored by every rater. A simple person × task × rater design can estimate several kinds of variance:

Person variance: stable differences among learners across the sampled measurement conditions.
Task variance: some tasks are generally easier or harder than others.
Rater variance: some raters are generally more severe or lenient.
Person × task variance: particular learners perform unusually well or badly on particular tasks.
Person × rater variance: a particular rater scores particular learners differently from what general severity alone would predict.
Task × rater variance: some raters react differently to particular tasks.
Person × task × rater plus residual: remaining variation not separated further by the design.

This decomposition is the core advantage. Two assessment systems can have the same headline reliability while needing very different repairs. One may need more tasks because person-by-task variation dominates. Another may need stronger rater training or more raters. A third may need repeated occasions because day-to-day instability is large.

The G study: diagnose the measurement design

A generalisability study, or G study, estimates the relevant variance components from data collected under a defined measurement design. The goal is diagnosis rather than simply issuing a pass/fail verdict on reliability.

Suppose an essay assessment finds the following illustrative pattern:

Source	Illustrative variance share	Interpretation
Learner	40%	There are meaningful differences among learners.
Prompt	5%	Some prompts are generally easier.
Rater	3%	General severity differences are modest.
Learner × prompt	30%	Learner ranking changes substantially across prompts.
Learner × rater and residual	22%	Other conditional variation remains.

These numbers are deliberately illustrative, not empirical results from a specific programme. Their purpose is to show the decision logic. If learner-by-prompt variation is large, adding a second rater to the same single prompt may do less for dependability than sampling more prompts. The bottleneck is task sampling, not primarily scorer severity.

The D study: redesign before collecting everything again

The second stage is the decision study, or D study. Once variance components have been estimated, the analyst can model how the measurement procedure would behave under different operational designs.

For example:

one prompt scored by two raters;
two prompts scored by one rater each;
three prompts with one common rater;
two occasions with two prompts each;
a shorter test with more diverse item sampling;
a design where every learner sees the same tasks;
a design where learners receive different sampled tasks.

The D study does not magically make the original data richer. It uses the estimated structure to forecast the consequences of changing the number and arrangement of facet conditions. This turns reliability analysis into design engineering.

Relative decisions and absolute decisions need different error

One of G Theory’s most important distinctions is between relative and absolute decisions.

A relative decision asks where one learner stands compared with others. If every learner is scored slightly higher by the same lenient rater, rankings may barely change. General rater severity can therefore matter less for a relative decision.

An absolute decision asks whether a learner meets a defined standard, such as mastery, certification or a cut score. Now a generally lenient rater can shift learners across the threshold. Facet main effects that do not alter rank can still matter.

Generalisability Theory therefore distinguishes a generalisability coefficient for relative interpretations from a dependability coefficient for absolute interpretations. They can differ because the definition of relevant error differs.

This is why “the reliability is .85” is incomplete. The reader needs to know the design, the universe of generalisation and the decision being supported.

Crossed and nested designs are not notation trivia

In a crossed design, every object of measurement encounters every condition of a facet. If every learner answers the same ten tasks, learners and tasks are crossed. If every response is scored by all raters, raters are crossed too.

In a nested design, some facet conditions occur only within another condition. If every learner receives a unique set of randomly generated questions, items may be nested within learners. If different classrooms are observed by different local observers, observer can be nested within school.

The distinction determines which variance components can be separately estimated. A design may confound sources that another design can separate. That means the measurement plan should be chosen with the later inference in mind, not merely for logistical convenience.

A modern case: different AI-generated items for different learners

Flexible digital assessment makes G Theory newly relevant. A 2026 Journal of Educational Measurement paper by Won-Chan Lee, Stella Y. Kim and Seungwon Shin examined Generalizability Theory for randomly parallel testing, where examinees can receive different but psychometrically similar item sets generated from templates or AI-based systems.

That operational flexibility creates a measurement problem: if two learners receive different sampled variants, score precision depends not only on the learner but on the template and item-sampling design. The authors show how G Theory can handle crossed, nested and multivariate structures and estimate conditional measurement error for domain-referenced interpretations.

The important educational lesson is broader than AI testing. When measurement conditions become more flexible, the error structure usually becomes more important, not less. A large item bank does not guarantee comparable evidence. The sampling design, domain representation and intended interpretation still need to be made explicit.

Worked example: should we add another rater or another task?

Imagine a speaking assessment with one interviewer, one speaking task and one scoring rubric. Results vary more than expected.

The instinctive response may be to add a second rater. But a G study reveals that rater main effects are small while learner-by-task variation is large. Some learners are much stronger on narrative prompts than on explanation prompts; others show the reverse.

A D study then compares designs. Two raters on one task produce only a modest improvement. Two tasks scored by one trained rater each produce a larger gain in dependability. Three tasks improve it further, but with increasing time and cost.

The conclusion is not that raters never matter. It is that the design should spend measurement resources on the dominant source of uncertainty. Generalisability Theory makes that trade-off visible.

More measurement is not automatically better measurement

Adding tasks, raters or occasions generally reduces some sampling error because scores are averaged over more observations. But the marginal benefit depends on the variance structure.

Ten almost identical tasks may add less useful domain coverage than four well-chosen tasks. More raters cannot repair construct underrepresentation. Repeated occasions can improve stability estimates but may introduce practice effects. A long test can raise conventional reliability while narrowing the construct if all items sample one easy-to-measure corner.

That is why Generalisability Theory belongs inside a larger validity argument rather than replacing one.

Reliability is conditional on the universe you claim

The phrase universe of admissible observations sounds technical, but the idea is ordinary: which tasks, raters, occasions and conditions count as legitimate replications of this measurement?

If an assessment is intended to support a claim about argumentative writing across many topics, the universe must include a defensible range of topics. If the score only supports writing on one prepared prompt, generalising across prompts may not be appropriate. If all scoring will be performed by one specifically trained rater, treating rater as a random facet may not match the operational design.

The reliability question therefore begins with the claim. You cannot estimate meaningful generalisability until you say where the score is supposed to travel.

What G Theory can diagnose—and what it cannot

G Theory is strong at decomposing variation under a specified design. It is not a detector of every measurement problem.

A high coefficient does not prove that the assessment measures the intended construct.
A precise score can still be biased.
A dependable narrow score can still underrepresent a broad capability.
Variance components depend on the sampled population and measurement conditions.
Poorly chosen facets produce a beautifully analysed version of the wrong design.
Small samples or sparse designs can make variance estimates unstable.
Confounded facets cannot always be separated after the fact.

Generalisability is therefore evidence about score dependability under a defined universe, not a universal certificate of validity or fairness.

A practical workflow for schools and assessment teams

Name the score interpretation. Is the decision relative, absolute, formative, placement-based or certification-based?
Name the object of measurement. Learner, teacher, class, school, programme or another entity?
List plausible facets. Tasks, raters, occasions, methods, contexts, stations or forms.
Decide the intended universe. Across which conditions should the score generalise?
Choose a design that can separate the sources that matter.
Run the G study. Estimate variance components and inspect interactions, not only a final coefficient.
Run D studies. Compare feasible operational designs.
Test cost and burden. More observations consume learner and staff time.
Check construct coverage. Do not optimise dependability by measuring a smaller thing.
Monitor after deployment. New raters, item-generation systems, populations or delivery modes can change the error structure.

For teachers: the concept matters even without running the model

A classroom teacher may never estimate a variance component formally, but the reasoning is still useful.

If a learner performs badly once, ask what changed. Was the difficulty tied to the learner, the task type, the marking condition, the day, the support available or the interaction among them? If important decisions depend on the result, collect evidence across enough defensible conditions to support the claim being made.

One difficult essay does not prove weak writing across all writing. One strong worked example does not prove independent problem solving. One oral response on one day does not automatically define language proficiency.

Generalisability reasoning is a formal version of a humane educational habit: do not turn one observation into a larger claim than the observation can carry.

For parents and learners

If an important decision rests on one score, ask what was sampled. How many tasks? How many kinds of task? Was judgement made by one rater or several? Is the score meant to represent performance on that exact test or capability across a broader domain?

This is not an argument for distrusting every assessment. It is an argument for matching confidence to evidence. Some tightly standardised tests are designed precisely to make a single administration highly informative. Other complex performances need broader sampling.

Failure modes

The one-number failure: reporting reliability without the design and decision context.
The wrong-facet failure: analysing raters when task sampling is the real source of instability.
The cheapest-design failure: choosing a nested or sparse design that cannot separate the sources later needed for interpretation.
The coefficient-maximisation failure: narrowing content simply to raise reliability.
The relative/absolute confusion: using rank-order reliability to justify threshold decisions.
The universal-population failure: assuming variance components estimated in one population automatically transfer to another.
The precision-equals-validity failure: treating dependable repetition as proof that the right construct was measured.

Sources and further reading

Return to the core idea: a score is not produced by the learner alone. It is produced by a learner meeting a measurement design. Generalisability Theory makes that meeting visible. It separates the variance the design can distinguish, asks which kinds of error matter for the intended decision, and lets us redesign the measurement procedure before false certainty hardens around one number.