Choose your reading route
eduKateSG · Why Mathematics? · Light’s Kappa, Pairwise Panels and Reviewer-Specific Margins
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 4–7
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 8–13
Check assumptions
See where design and calibration matter.Route 4 · Chapters 14–18
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 19–24
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when the same items are classified by three or more named reviewers? Light’s kappa summarises the pairwise Cohen kappas. It respects each reviewer pair’s separate margins, then averages the pair coefficients. That sounds simple, yet it raises useful questions about shared items, unequal denominators, dependence among pairs and what an average can hide.
This chapter uses twelve original binary classifications by three reviewers. Two reviewer pairs have kappa 2/3, one has 1/3, and Light’s average is 5/9. We compare the result with pooled-margin alternatives, inspect the weakest pair and show why a panel coefficient should support calibration rather than replace the underlying pair tables.
Full chapter index
Chapters 1–3 · Understand the question
Chapters 4–7 · Reproduce the example
Chapters 8–13 · Check assumptions
Chapters 14–18 · Explore applications
Chapters 19–24 · Study and verify
Three reviewers independently classify the same twelve items as A or B. Because the reviewers are named and fixed, each pair has a genuine 2 × 2 table and separate margins.
Reviewer 1 gives six A and six B ratings. Reviewer 2 gives four A and eight B. Reviewer 3 gives six A and six B. The full sequence is preserved in the worked table rather than reduced immediately to totals.
Light’s approach calculates Cohen kappa for pairs 1–2, 1–3 and 2–3, then takes their arithmetic mean. With three reviewers there are three pairs; with r reviewers there are r choose 2.
| Item | Reviewer 1 | Reviewer 2 | Reviewer 3 |
|---|---|---|---|
| 1 | A | A | A |
| 2 | A | A | A |
| 3 | A | A | A |
| 4 | A | A | B |
| 5 | A | B | A |
| 6 | A | B | A |
| 7 | B | B | A |
| 8 | B | B | B |
| 9 | B | B | B |
| 10 | B | B | B |
| 11 | B | B | B |
| 12 | B | B | B |
An overall percentage can hide which reviewers disagree. A panel may look acceptable because two pairs are strong while one pair is weak. Pair tables identify the relationship that needs calibration.
Pairwise Cohen kappa also uses each pair’s own margins. Reviewer 2’s greater use of B enters pairs 1–2 and 2–3, while pair 1–3 has balanced margins.
The method therefore keeps reviewer identity. That differs from statistics based only on item-by-category counts, where names can disappear after counts are pooled.
For reviewers 1 and 2, the table is [[4,2],[0,6]] when rows and columns follow A, B. They agree on ten of twelve items, so observed agreement is 5/6.
Reviewer 1 uses A with probability 1/2; reviewer 2 uses A with probability 1/3. Expected agreement is (1/2 × 1/3) + (1/2 × 2/3) = 1/2.
Kappa is (5/6 − 1/2)/(1 − 1/2) = (1/3)/(1/2) = 2/3. The off-diagonal direction also shows reviewer 2 assigns B where reviewer 1 assigns A twice, with no reverse cases.
For reviewers 1 and 3, the table is [[5,1],[1,5]]. Again observed agreement is 10/12 = 5/6.
Both margins are balanced, so expected agreement is 1/2. Kappa is again 2/3.
The coefficient matches pair 1–2, but the tables differ. Pair 1–3 has symmetric disagreement; pair 1–2 has a directional shift. A panel report should not treat equal kappas as proof that the reviewer relationships are identical.
For reviewers 2 and 3, the table is [[3,1],[3,5]]. They agree on eight of twelve items, giving observed agreement 2/3.
Reviewer 2 uses A one-third of the time and reviewer 3 one-half. Expected agreement remains 1/2. Kappa is (2/3 − 1/2)/(1/2) = 1/3.
This pair is clearly weaker. Their disagreement is also directional: reviewer 3 assigns A on three items that reviewer 2 labels B, while the reverse occurs once.
Average the three pair kappas: (2/3 + 2/3 + 1/3)/3. The sum is 5/3, so the average is 5/9, approximately 0.556.
This number describes the mean pairwise margin-corrected agreement for the fixed panel. It does not mean every pair has kappa 0.556. The range from 1/3 to 2/3 is operationally important.
Always report the pair coefficients or at least their distribution beside the mean. An average without spread can conceal a reviewer who interprets the rubric differently.
| Pair | Table [[AA,AB],[BA,BB]] | Po | Pe | Kappa |
|---|---|---|---|---|
| 1–2 | [[4,2],[0,6]] | 5/6 | 1/2 | 2/3 |
| 1–3 | [[5,1],[1,5]] | 5/6 | 1/2 | 2/3 |
| 2–3 | [[3,1],[3,5]] | 2/3 | 1/2 | 1/3 |
| Light mean | 7/9 | 5/9 |
The three observed agreements are 5/6, 5/6 and 2/3. Their mean is 7/9, approximately 0.778.
Because all three pair expected agreements happen to equal 1/2, applying the kappa transformation to the mean observed agreement also gives 5/9. That equality is special to this example.
In general, the average of ratios is not the ratio formed from average numerators and denominators. If pair expected agreements differ, compute each Cohen kappa first for Light’s definition.
Across 36 ratings, A appears 16 times and B 20 times. A pooled reference is (16/36)² + (20/36)² = 41/81.
Using mean pair agreement 7/9, a Fleiss-style coefficient is (7/9 − 41/81)/(1 − 41/81) = 11/20 = 0.55.
That is close to Light’s 5/9 but not equal. Similar decimals should not blur the difference: Light averages pairwise Cohen references; the pooled method uses a common category distribution.
Conger’s fixed-panel generalisation combines reviewer-specific margins into an overall observed and expected term. Light’s statistic averages the completed pairwise kappas.
When pair denominators 1 − Pe differ, simple averaging and denominator-weighted aggregation can diverge. This was visible in the earlier Conger example in the series.
Choose the statistic by definition and design, not by whichever returns a familiar number. Preserve pair results so another analyst can reproduce the aggregation.
Did you know the three pair kappas are not independent estimates? Pair 1–2 and pair 1–3 both contain reviewer 1 and the same twelve items.
Treating them as three independent studies would understate uncertainty. A confidence interval for the panel summary should respect shared reviewers and items, often through a method designed for the complete rating matrix or a suitable item-level bootstrap.
The arithmetic mean is easy; inference around it requires the data structure.
Our panel is complete, so every pair uses twelve items. If ratings are missing, one pair may use many more items than another.
A simple unweighted mean then gives a small pair the same influence as a large pair. Weighting by sample size is tempting, but it changes the estimand and still may not address informative missingness.
State the complete-case rule for each pair and consider a coefficient designed for incomplete panels. Do not silently fill missing ratings or treat abstention as B.
Keeping names allows targeted calibration. Pair 2–3 may disagree because reviewer 2 uses B more often, reviewer 3 interprets a boundary differently, or the two see different evidence.
Inspect the specific discordant items after independent scoring. Ask which rubric phrase or example led to each decision. Revise the system, then validate on new items.
The coefficient finds a pattern; conversation and domain evidence investigate its cause.
All three reviewers could share the same misconception. High pair kappas do not prove labels are correct, fair or useful.
If a justified reference standard exists, compare each reviewer with it using accuracy measures appropriate to the purpose. If no gold standard exists, adjudication can create a consensus label, but consensus should not be retrospectively presented as independent truth.
In school marking and AI evaluation, agreement is one part of quality assurance, alongside rubric validity, coverage and consequence.
This example is binary and unweighted. For ordered categories, each pair could use a declared weighted Cohen kappa before averaging.
Every pair must use the same category order and weight matrix for the panel mean to be interpretable. If one pair uses linear weights and another quadratic weights, the average combines different loss functions.
Report exact agreement and the weight definition beside any weighted Light-style summary. Near agreement should never be hidden behind a high weighted value.
All pairs rate the same items, so easy or difficult samples affect the whole panel. A set dominated by obvious cases may inflate agreement; a boundary-focused calibration set may reduce it.
Describe item selection and intended population. Do not compare panels that rated different samples as if the coefficients alone rank reviewer skill.
A shared representative validation set supports fairer comparison and clearer improvement over time.
An item-level bootstrap can resample the twelve rating rows, recalculate all pair tables and recompute their mean. That preserves reviewer identity and the dependence created by shared items.
If items cluster within students or documents, resample clusters. If reviewers are meant to represent a larger reviewer population, a fixed-panel interval may not generalise to new reviewers.
State which source of variation the interval covers: new items, new reviewers or both.
Define the rubric, select a common sample and collect independent ratings. Calculate every pair table, observed agreement, kappa and the Light mean.
Flag the weakest pairs and recurring discordant items. Discuss reasons, improve examples and test a new sample. Keep earlier results rather than overwriting them.
This cycle makes mathematics a learning system. The objective is stable, defensible decisions, not a decorative threshold.
Write the twelve triplets and form three 2 × 2 tables. Check each row and column total before calculating expected agreement.
Derive pair kappas 2/3, 2/3 and 1/3, then average them to 5/9. Pool all 36 ratings to obtain proportions 4/9 and 5/9, and verify the alternative coefficient 11/20.
Finally explain why 5/9 and 11/20 are close but conceptually different. This combines fractions, combinations, probability and careful language.
- Light’s kappa is not the kappa of one pooled table.
- The mean does not describe every reviewer pair.
- Pair estimates sharing reviewers are not independent.
- Similar panel coefficients can come from different pair patterns.
- Agreement does not prove accuracy.
- Missing ratings cannot be ignored without defining the estimand.
Another mistake is to report only the panel mean. Calibration needs the pair tables because they show direction and locate disagreement.
Using the worked ratings:
- build all three pair tables;
- verify observed agreements 5/6, 5/6 and 2/3;
- verify every pair reference equals 1/2;
- calculate kappas 2/3, 2/3 and 1/3;
- obtain Light’s 5/9;
- count pooled A and B ratings as 16 and 20; and
- verify the pooled-margin result 11/20.
Then change one reviewer-2 rating from B to A. Rebuild both pairs containing reviewer 2. Do not assume the panel mean rises: margins and observed agreement can move together.
Students can learn this method through three independent classifications of the same set of objects. The useful question is not “who is worst?” but “where do definitions lead to different decisions?”
Parents can reinforce a broad lesson: an average may hide variation. Ask to see the components before accepting a single score.
For project work, preserve identifiers, raw labels and the rule used to combine pairs. Those habits transfer to data science, engineering reviews, research coding and AI.
How many pair kappas are there?
For r reviewers, there are r(r−1)/2 unordered pairs.
Is the mean weighted by pair size?
The usual Light definition is an arithmetic mean of pairwise kappas. In incomplete designs, use a method that explicitly matches the intended estimand.
Why not report only Fleiss’ kappa?
Fleiss uses pooled margins and can be appropriate for exchangeable reviewer designs. Light preserves named pairwise Cohen references. The design determines relevance.
Can Light’s kappa be negative?
Yes. Negative pair kappas can pull the average below zero when agreement is worse than pair-specific references.
Light’s primary article is *Measures of Response Agreement for Qualitative Data: Some Generalizations and Alternatives*, DOI 10.1037/h0031643. Warrens’ inequalities between multi-rater kappas states the relationship among Light, Hubert, Fleiss and Randolph measures.
Continue with Conger’s kappa, Fleiss’ kappa and the Mathematics Learning Hub.
Verify the fixed named panel, common items, independent ratings, pair tables, category definitions, missing-data rules and aggregation. Show pair coefficients, observed agreements, margins and uncertainty. Avoid claims about truth that agreement cannot support.
The importance of mathematics here is that an average becomes transparent: every pair contributes a visible calculation, and every hidden disagreement can be traced back to a particular relationship and item.
Extended practice laboratory
Practice case 1: Four reviewers
Setup: Expand the panel from three to four. Task: Count pair kappas.
Check: Four choose two gives six pairwise coefficients. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 2: Five reviewers
Setup: Expand again. Task: Count pairs.
Check: Five choose two gives ten, showing why pair summaries scale quickly. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 3: One weak reviewer
Setup: One person disagrees with everyone. Task: Predict the pair pattern.
Check: Every pair containing that reviewer may be weak, which the mean alone can hide. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 4: One weak relationship
Setup: Only pair 2–3 is weak. Task: Plan calibration.
Check: Inspect their discordant items without assuming either reviewer is wrong. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 5: Equal kappas
Setup: Two pairs share the same coefficient. Task: Compare tables.
Check: Equal kappas can conceal different disagreement direction and margins. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 6: Different Pe
Setup: Pair expected agreements differ. Task: Test shortcut averaging.
Check: Average completed kappas; a ratio of average components is generally different. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 7: Missing block
Setup: Reviewer 3 skips four hard items. Task: Assess complete-case bias.
Check: Their pair kappas may describe an easier subset, so report pair samples. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 8: One shared item set
Setup: All pairs rate identical items. Task: Assess dependence.
Check: Pair estimates share items and reviewers and are not independent studies. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 9: Bootstrap rows
Setup: Resample the rating matrix. Task: Preserve structure.
Check: Resample whole item rows and recompute every pair before averaging. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 10: Student clusters
Setup: Several rows belong to each student. Task: Change resampling.
Check: Sample student clusters when within-student dependence matters. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 11: Ordered labels
Setup: Use four rubric levels. Task: Prepare a weighted panel mean.
Check: Declare one common order and weight matrix for every pair. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 12: Mixed weights
Setup: Pairs use different weight rules. Task: Can they be averaged plainly?
Check: No; they represent different loss functions and lack a common interpretation. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 13: Directional disagreement
Setup: Table b exceeds c. Task: What does kappa omit?
Check: The symmetric coefficient omits direction, so show the pair table. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 14: Balanced margins
Setup: Both reviewers use A and B equally. Task: Find Pe.
Check: For a binary pair, Pe is one-half. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 15: Extreme margins
Setup: Both mostly use A. Task: Predict Pe.
Check: Expected agreement rises, which can lower kappa despite high raw agreement. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 16: Gold standard
Setup: Reviewer 1 is a validated reference. Task: Change the analysis question.
Check: Accuracy summaries may be more useful than treating every pair symmetrically. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 17: Consensus meeting
Setup: Reviewers resolve all differences. Task: Is kappa now one?
Check: Consensus labels are not independent ratings and should not replace the original estimate. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 18: New validation set
Setup: Apply a revised rubric elsewhere. Task: What does it test?
Check: It tests transfer rather than memory of calibration examples. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 19: Pair-size weighting
Setup: One pair has twice as many complete items. Task: Can weights be improvised?
Check: Weighting changes the estimand and must be declared and justified. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 20: Reviewer population
Setup: The panel represents many possible reviewers. Task: Name a limitation.
Check: A fixed-panel statistic may not capture variation from selecting new reviewers. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 21: Panel drift
Setup: Repeat ratings next term. Task: Compare responsibly.
Check: Use a comparable item sample and preserve rubric versions before attributing change. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 22: High mean, wide range
Setup: Pair kappas are 0.9, 0.9 and 0.1. Task: Interpret the mean.
Check: The average is misleading without the range and pair identities. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 23: Similar methods
Setup: Light and Fleiss values nearly match. Task: Can one substitute for the other?
Check: No; similar decimals can arise from different expected-agreement models. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 24: Audit package
Setup: Software prints one panel score. Task: Verify its definition.
Check: Confirm whether it averages pairwise Cohen kappas or implements another multirater coefficient. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.