Choose your reading route
eduKateSG · Why Mathematics? · Randolph’s Kappa, Free Margins and Uniform Category References
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 4–7
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 8–13
Check assumptions
See where design and calibration matter.Route 4 · Chapters 14–18
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 19–23
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when several reviewers are free to use categories without fixed quotas? Randolph’s free-marginal multirater kappa compares observed pair agreement with a uniform reference of 1/q, where q is the number of genuinely available categories. It can differ sharply from Fleiss’ fixed-marginal kappa when reviewers concentrate their classifications in one category.
This chapter works from an original ten-item, four-reviewer example with three categories. It derives the pair counts, obtains Randolph’s kappa of 11/20, and shows why Fleiss’ coefficient on the same data is −1/79. The contrast does not declare one method right. It asks which chance model fits the design and why the category set, sample and reviewer instructions matter.
Full chapter index
Chapters 1–3 · Understand the question
Chapters 4–7 · Reproduce the example
Chapters 8–13 · Check assumptions
Chapters 14–18 · Explore applications
Chapters 19–23 · Study and verify
Four reviewers independently classify each of ten items as A, B or C. They are not told to place a fixed number of items in each category. The three categories are all available, and C is used at least once.
That sounds like a free-marginal design: category totals emerge from decisions rather than quotas. Randolph proposed a multirater coefficient for this setting. Its expected agreement is 1/q, so the reference for three categories is 1/3.
The phrase “free marginal” describes the task design, not a guarantee that reviewers will use categories evenly. In our example they do not. Most ratings are A. The uniform reference is a model chosen because no category totals were constrained.
For each item, count how many of the four ratings fall in A, B and C. The ten rows are five copies of (4,0,0), three copies of (3,1,0), then (2,2,0) and (2,1,1).
Every row totals four. Across all 40 ratings, category totals are 33 for A, 6 for B and 1 for C. These pooled margins are highly uneven.
The item rows preserve more information than the totals. A row of (3,1,0) contains three agreeing reviewer pairs and three disagreeing pairs. A row of (2,1,1) contains only one agreeing pair. Agreement is computed within items before averaging.
| Items | A | B | C | Item agreement |
|---|---|---|---|---|
| 1–5 | 4 | 0 | 0 | 1 each |
| 6–8 | 3 | 1 | 0 | 1/2 each |
| 9 | 2 | 2 | 0 | 1/3 |
| 10 | 2 | 1 | 1 | 1/6 |
| Pooled ratings | 33 | 6 | 1 | Po = 7/10 |
Four reviewers form six unordered pairs because 4 choose 2 equals 6. For category count n, the number of agreeing pairs in that category is n choose 2, or n(n−1)/2.
Dividing the sum of agreeing pairs by six gives item agreement. A unanimous (4,0,0) row has 6/6 = 1. A (3,1,0) row has 3/6 = 1/2. A (2,2,0) row has 1 + 1 = 2 agreeing pairs, or 1/3. A (2,1,1) row has 1/6.
This combination step prevents double counting. Counting ordered pairs would create 12 comparisons, but both numerator and denominator would need the same convention.
The five unanimous rows contribute 5. The three (3,1,0) rows contribute 3/2. The (2,2,0) row contributes 1/3, and the (2,1,1) row contributes 1/6.
The sum is 5 + 3/2 + 1/3 + 1/6 = 7. Divide by ten items and observed pair agreement is 7/10.
This means that a randomly selected unordered reviewer pair agrees on a randomly selected item 70% of the time, under the equal-item averaging used here. It does not mean 70% of items are unanimous; only five of ten are.
With q = 3 categories, Randolph’s expected agreement is 1/3. The coefficient is (7/10 − 1/3)/(1 − 1/3).
The numerator is 11/30 and the denominator is 2/3, so the result is 11/20 = 0.55. Equivalently, the formula can be written (qPo − 1)/(q − 1).
The uniform reference asks how much the observed 0.70 exceeds the 0.333 benchmark. It does not use the pooled proportions 33/40, 6/40 and 1/40.
CHAPTER 6 OF 23 · Reproduce the example
6. Compare Fleiss’ fixed-marginal reference
Fleiss’ expected agreement uses squared pooled category proportions. Here it is (33/40)² + (6/40)² + (1/40)² = 1126/1600 = 563/800, or 0.70375.
Observed agreement is 0.70, slightly below that reference. Fleiss’ kappa is (7/10 − 563/800)/(1 − 563/800). The numerator is −3/800 and the denominator 237/800, so the result is −1/79, approximately −0.0127.
The same ratings therefore yield 0.55 and about −0.013. The gap comes from reference models, not arithmetic error.
| Measure | Observed | Expected | Coefficient |
|---|---|---|---|
| Randolph | 7/10 | 1/3 | 11/20 |
| Fleiss | 7/10 | 563/800 | -1/79 |
The pooled ratings are dominated by A. A fixed-marginal model treats high agreement as unsurprising when reviewers frequently use the same dominant category. The free-marginal model retains the uniform 1/3 reference because category totals were not restricted.
If the project’s main question concerns actual reviewer behaviour in this sample, the pooled concentration is informative. If the intended benchmark reflects free selection among three available categories, Randolph’s coefficient may be useful.
A responsible report can show both, explain the design and avoid choosing by numerical appeal. The 70% observed agreement remains a stable descriptive anchor.
If q changes, the free-marginal reference changes. With two categories it is 1/2; with three, 1/3; with four, 1/4.
An unused category should not be added casually to raise the coefficient. It must have been genuinely available, defined and plausible in the task. Conversely, omitting a rarely used but real category changes the decision space.
Our category C appears once, which confirms use but does not by itself prove the rubric is well designed. Document the full category set before ratings begin.
Did you know that pair agreement is not the same as the percentage of unanimous items? In our table, unanimity is 50%, while pair agreement is 70%.
The three 3-to-1 splits still contain half of all reviewer pairs in agreement. The 2-to-2 split contains one-third. Pair counting credits partial consensus in a transparent way.
This distinction matters whenever a report says “reviewers agreed on 70% of cases”. State whether that means unanimous items, majority agreement, average pair agreement or something else.
Reviewers were free to choose categories, yet 33 of 40 ratings are A. Freedom of choice and uniform realised frequencies are different claims.
The uniform reference is a hypothetical benchmark, not a prediction that each category will appear one-third of the time. When the observed margins are extremely concentrated, show them so readers understand why another coefficient differs.
Mathematics improves the discussion by separating the experimental instruction from the empirical distribution.
The coefficient assumes ratings were produced independently enough for agreement to be meaningful. Reviewers who discuss each item before recording decisions can manufacture consensus.
Shared training is appropriate before a validation set, but the scored items should be rated independently if the goal is independent agreement. Record whether reviewers saw one another’s labels, model suggestions or adjudication notes.
In AI labelling, a shared model suggestion can correlate errors. In education, a common answer key can create consistent interpretation without proving the rubric covers every valid response.
The formula averages item pair agreement equally. If some items come from the same student or document, observations may be clustered. If some items are intentionally difficult boundary cases, the sample may not represent routine work.
Report the sampling frame and consider uncertainty procedures that resample at the appropriate cluster. A coefficient cannot identify why an item was difficult.
A useful calibration report lists repeated disagreement patterns and examines them qualitatively after the independent calculation.
Our example is complete. With missing ratings, each item may have a different number of available reviewer pairs. Analysts must decide how to weight items and whether missingness is informative.
Do not treat “not rated” as category C. That creates false disagreements and changes q. Preserve missing values separately.
When designs are incomplete, use a method and software implementation that explicitly documents how available pairs, items and categories enter the calculation.
Randolph’s coefficient becomes negative when observed agreement is below 1/q. That says the data show less pair agreement than the uniform reference.
A negative value can arise from systematic opposition, category misunderstandings or a small unusual sample. It is not proof that reviewers deliberately disagree.
Inspect the item table before interpreting the sign. Coefficients summarise patterns; they do not diagnose causes.
Ten items make this a teaching example, not a precise study. A bootstrap can resample items and recalculate the coefficient, provided items are the independent units. Clustered items require cluster resampling.
Confidence intervals for different kappa variants may use different variance formulas. Software output should be checked against the design and version used.
Avoid rigid adjective bands as a substitute for uncertainty and consequences. A value of 0.55 can be adequate for exploratory coding and inadequate for a high-stakes decision.
Define categories and reviewer instructions first. Decide whether quotas or fixed totals exist. Collect an independent pilot, preserve raw ratings, and calculate the item-by-category counts.
Report observed pair agreement, unanimity, category totals, Randolph’s coefficient and at least one margin-sensitive comparison. Review disagreements, revise the rubric, then test on new items.
Do not revise the same labels until a target coefficient appears. That measures negotiation, not independent reproducibility.
Write six possible reviewer pairs for four reviewers. For each row, count matching pairs with combinations. Add the ten fractions and divide by ten.
Next calculate 1/3 and obtain 11/20. Pool all ratings, square the three proportions and obtain 563/800. Use that reference to verify −1/79.
The exercise joins combinations, fractions, probability and model comparison. It is a clear example of why mathematics education supports reasoning about evidence.
- Free margins do not mean realised categories are uniform.
- Randolph’s kappa is not automatically better than Fleiss’.
- Adding unused categories is not legitimate sensitivity analysis.
- Pair agreement is not the same as unanimity.
- Agreement is not accuracy or validity.
- A high value does not remove the need to inspect category totals.
The central misconception is that “chance correction” has one inevitable meaning. Every coefficient embeds a reference model.
Verify the item agreements 1, 1, 1, 1, 1, 1/2, 1/2, 1/2, 1/3 and 1/6. Confirm their mean is 7/10.
Then:
- derive the six pairs from 4 choose 2;
- calculate Randolph’s 11/20;
- confirm pooled totals 33, 6 and 1;
- calculate Fleiss’ 563/800 reference; and
- show the fixed-marginal coefficient is −1/79.
Finally, predict how adding a genuine fourth category changes only the free-marginal reference if the observed ratings stay the same. Explain why that hypothetical may or may not be legitimate.
Students can practise consensus questions with four friends classifying shapes, headlines or solution methods. Define categories before starting and record decisions independently.
Parents can ask a useful question: were reviewers free to choose, or required to fill a quota? That simple design fact changes the mathematics.
For projects, keep the raw grid and a short method note. Reproducible records are more valuable than an unexplained decimal.
When is Randolph’s kappa appropriate?
It is designed for multiple reviewers when category totals are not fixed and a uniform-category reference is defensible.
Why report Fleiss’ kappa too?
The comparison shows sensitivity to the observed pooled margins and prevents the free-marginal assumption from remaining hidden.
Can there be two reviewers?
Free-marginal binary coefficients exist, but Randolph’s contribution is commonly discussed as a multirater extension.
Does 0.55 mean 55% agreement?
No. Observed pair agreement here is 70%. The coefficient is agreement beyond the 1/3 reference, scaled by the available improvement.
Randolph’s primary conference paper is available through ERIC with the full paper. A later mathematical comparison of multirater kappas is Warrens, Inequalities between multi-rater kappas.
Compare Fleiss’ kappa, Brennan–Prediger and the Mathematics Learning Hub.
Before using the result, verify the reviewer panel, independence, genuine category set, absence or presence of quotas, item sampling, missing-data rule, pair denominator, pooled margins and uncertainty. Report the table and observed agreement beside the coefficient.
The importance of mathematics is its ability to make an invisible assumption visible: whether expected agreement should follow the observed margins or a free, uniform category model.
Extended practice laboratory
Practice case 1: Two categories
Setup: Keep Po at 0.70 and set q to two. Task: Calculate the free reference and coefficient.
Check: Pe is 1/2 and kappa-free is 0.40. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 2: Four categories
Setup: Keep Po at 0.70 with four real options. Task: Recalculate.
Check: Pe is 1/4 and the coefficient is 0.60. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 3: Unused option
Setup: Add a category nobody could select. Task: Decide whether q changes.
Check: It must not; only genuinely available categories belong in the model. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 4: Rare real option
Setup: One reviewer uses a defined category once. Task: Document q.
Check: The option remains part of the declared decision space despite rarity. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 5: Five reviewers
Setup: Count unordered pairs. Task: Compute the denominator.
Check: Five choose two gives ten pairs per complete item. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 6: Three-to-two split
Setup: Five reviewers divide 3 and 2. Task: Count agreeing pairs.
Check: Three choose two plus two choose two gives four of ten pairs. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 7: All different
Setup: Three reviewers use three different categories. Task: Find item agreement.
Check: No pair matches, so item agreement is zero. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 8: One abstention
Setup: A reviewer skips an item. Task: Choose the pair denominator.
Check: Use a documented available-pair method or a coefficient for incomplete panels; never invent a label. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 9: Quota instruction
Setup: Reviewers must use each category ten times. Task: Choose the reference family.
Check: Margins are constrained, so a free-marginal benchmark needs special justification. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 10: Open choice
Setup: No quotas exist. Task: State what remains uncertain.
Check: Free selection supports the design story, while observed margins still deserve reporting. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 11: Dominant category
Setup: Almost all ratings are A. Task: Compare models.
Check: Randolph can stay positive while a pooled-margin coefficient falls sharply. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 12: Uniform margins
Setup: All categories are used equally overall. Task: Predict convergence.
Check: Free and pooled expected agreements can coincide at 1/q. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 13: Unanimity rate
Setup: Half the items are unanimous. Task: Can Po be inferred?
Check: No; partial splits on remaining items determine pair agreement. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 14: Majority rate
Setup: Every item has a majority. Task: Does that imply high agreement?
Check: No; narrow majorities can contain many disagreeing pairs. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 15: Shared hints
Setup: Reviewers see a model suggestion. Task: Interpret independence.
Check: Common prompts can correlate labels and errors, so disclose them. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 16: Calibration reuse
Setup: Score the same items after discussion. Task: Name the risk.
Check: Improvement may reflect memory; validate on new items. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 17: Clustered documents
Setup: Several items come from one document. Task: Plan uncertainty.
Check: Resample or model at the document cluster when appropriate. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 18: Small sample
Setup: Only ten items are rated. Task: Describe precision.
Check: Treat the estimate as a teaching or pilot result and show uncertainty. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 19: Negative value
Setup: Po falls below 1/q. Task: Interpret cautiously.
Check: Agreement is below the uniform benchmark; the coefficient does not diagnose motive. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 20: Category merge
Setup: Combine B and C. Task: What changes?
Check: Both q and observed pair agreement can change, so recompute from raw ratings. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 21: Reviewer removal
Setup: Drop one reviewer after seeing results. Task: Assess the practice.
Check: Post-hoc removal biases the summary unless justified independently and disclosed. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 22: Equal item weights
Setup: One item is high stakes. Task: Can it receive extra weight silently?
Check: No; a weighted estimand requires a predeclared reason and formula. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 23: Accuracy question
Setup: A reference label exists. Task: Add suitable evidence.
Check: Agreement among reviewers does not replace comparison with the justified reference. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 24: Reproducibility file
Setup: Another analyst repeats the work. Task: List minimum inputs.
Check: Provide the raw grid, category set, missing rule, q, pair formula and software version. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.