Choose your reading route
eduKateSG · Why Mathematics? · Weighted Kappa, Ordinal Distance and Declared Error Costs
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 4–7
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 8–13
Check assumptions
See where design and calibration matter.Route 4 · Chapters 14–18
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 19–26
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when two people use an ordered scale? Exact agreement treats every mismatch alike, but a one-step difference between “developing” and “secure” is not the same decision as a three-step difference between “beginning” and “advanced”. Weighted Cohen’s kappa makes that distinction explicit. It combines a contingency table, an order for the categories, a declared penalty pattern and a chance reference built from the two reviewers’ margins.
This is a useful example of mathematics in education, research, AI evaluation and quality control because the formula is only the final step. The deeper work is deciding whether the labels are genuinely ordered, whether their gaps have a defensible meaning and whether linear or quadratic weights answer the actual question. This chapter builds the calculation from an original synthetic example, checks it with exact fractions and shows why a larger weighted coefficient is not automatically evidence of better reviewing.
Full chapter index
Chapters 1–3 · Understand the question
Chapters 4–7 · Reproduce the example
Chapters 8–13 · Check assumptions
Chapters 14–18 · Explore applications
Chapters 19–26 · Study and verify
CHAPTER 1 OF 26 · Understand the question
1. Why ordered disagreement needs its own question
Suppose two teachers classify the same set of responses on four levels numbered 0, 1, 2 and 3. An unweighted agreement measure asks whether the numbers match exactly. That is often useful, yet it discards information about how far the two decisions are apart. A 2 versus 3 classification and a 0 versus 3 classification both become a single disagreement.
Weighted kappa asks a more specific question: after assigning partial credit to near agreement, how much weighted agreement remains beyond the amount expected from the two reviewers’ category use? The answer depends on the weight rule. That dependence is not a defect hidden inside the method. It is a modelling choice that should be visible in the plan and report.
Mathematics matters here because it forces the team to translate “close enough” into a reproducible table. Once the translation is explicit, another reader can audit both the arithmetic and the judgement behind it.
CHAPTER 2 OF 26 · Understand the question
2. The data table and what every cell means
Our synthetic study contains 35 items independently classified by reviewer A and reviewer B. Rows record A’s category and columns record B’s category. The diagonal cells are exact matches. Cells next to the diagonal differ by one category; more distant cells would represent larger disagreements.
The four row totals are 10, 10, 9 and 6. The column totals are 9, 10, 9 and 7. There are 26 exact matches: 8 + 7 + 6 + 5. Nine items are one step apart, and no item is two or three steps apart. These counts deliberately create a clean worked example; they are not claims about any school, marking team or student population.
Before computing a coefficient, check that the same labels mean the same thing to both reviewers and that every item occupies one cell. If reviewer A’s “2” means something different from reviewer B’s “2”, a neat square table can conceal a measurement problem rather than solve it.
| A / B | 0 | 1 | 2 | 3 | Row total |
|---|---|---|---|---|---|
| 0 | 8 | 2 | 0 | 0 | 10 |
| 1 | 1 | 7 | 2 | 0 | 10 |
| 2 | 0 | 1 | 6 | 2 | 9 |
| 3 | 0 | 0 | 1 | 5 | 6 |
| Column total | 9 | 10 | 9 | 7 | 35 |
The observed exact agreement is 26/35, approximately 0.7429. Cohen’s chance reference uses the two separate margins. For category 0, the product is (10/35)(9/35); the other products are (10/35)(10/35), (9/35)(9/35) and (6/35)(7/35).
Adding them gives 313/1225, approximately 0.2555. Unweighted kappa is therefore (26/35 − 313/1225) divided by (1 − 313/1225), which simplifies to 199/304, approximately 0.6546.
That number answers an exact-match question. It does not yet use the fact that every disagreement is only one step away. Keeping this baseline is valuable. If the weighted result later looks much larger, the reader can see that the difference came from the chosen treatment of distance rather than from a different dataset.
CHAPTER 4 OF 26 · Reproduce the example
4. Turn category distance into disagreement weights
For four ordered categories, label the positions 0, 1, 2 and 3. A common linear disagreement weight is the absolute distance divided by 3. Exact matches receive 0 disagreement; one-step differences receive 1/3; two-step differences receive 2/3; and a 0 versus 3 difference receives 1.
A common quadratic disagreement weight squares that normalised distance. The possible penalties become 0, 1/9, 4/9 and 1. Quadratic weights therefore forgive one-step differences more strongly while making the most distant difference retain the full penalty.
Some software presents agreement weights rather than disagreement weights. In that convention, the weight is one minus the penalty. Both conventions can lead to the same coefficient if used consistently. Mixing them can reverse the intended meaning. A careful report names the convention, gives the matrix or formula and states the category order.
| Distance | Linear | Quadratic |
|---|---|---|
| 0 | 0 | 0 |
| 1 | 1/3 | 1/9 |
| 2 | 2/3 | 4/9 |
| 3 | 1 | 1 |
CHAPTER 5 OF 26 · Reproduce the example
5. Calculate the observed linear disagreement
Every one of the nine off-diagonal items in our table lies one step from the diagonal. Each therefore contributes a linear penalty of 1/3. The total weighted penalty is 9 × 1/3 = 3.
Divide by all 35 items and the observed linear disagreement is 3/35, approximately 0.0857. Equivalently, the observed weighted agreement is 1 − 3/35 = 32/35. The disagreement form is especially convenient because weighted kappa can be written as one minus the ratio of observed weighted disagreement to expected weighted disagreement.
Notice what has changed. The underlying 35 classifications have not changed. We have changed the loss assigned to a mismatch. This is why a weighted coefficient should never be reported without the weights. “Weighted kappa was 0.78” is incomplete if the reader cannot tell what kind of disagreement the number discounted.
CHAPTER 6 OF 26 · Reproduce the example
6. Build the expected linear disagreement from margins
The expected table pairs reviewer A’s row proportions with reviewer B’s column proportions. For every possible cell, multiply the relevant row probability, column probability and distance penalty, then add the results.
For this example, the exact fraction is 487/1225, approximately 0.3976. That expected disagreement is much larger than the observed 3/35. Linear weighted kappa is 1 − (3/35)/(487/1225). Since 3/35 equals 105/1225, the ratio is 105/487 and the coefficient is 382/487, approximately 0.7844.
The calculation is a comparison of two weighted averages. The observed average uses actual paired classifications. The expected average uses all row-column combinations implied by separate marginal use. The coefficient is high because the observed pairs stay much closer than those independent combinations would usually be.
CHAPTER 7 OF 26 · Reproduce the example
7. Repeat the calculation with quadratic weights
With quadratic disagreement, each one-step mismatch contributes 1/9. Nine such mismatches therefore contribute a total penalty of 1. The observed quadratic disagreement is 1/35.
The margin-based expected quadratic disagreement is 401/1575, approximately 0.2546. Quadratic weighted kappa is 1 − (1/35)/(401/1575). Because 1/35 equals 45/1575, the ratio is 45/401 and the coefficient is 356/401, approximately 0.8878.
The quadratic result exceeds the linear result because all observed disagreements are small. That is a feature of this dataset and weight choice, not a universal ranking. If several classifications were three steps apart, both systems would assign those extreme cells full penalty. The comparison must be traced through the actual cells rather than memorised as “quadratic is always higher”.
The unweighted coefficient is about 0.655, the linear weighted coefficient about 0.784 and the quadratic weighted coefficient about 0.888. These are three answers to three related questions.
- Unweighted kappa asks about exact category equality.
- Linear weighting gives partial credit that falls evenly with each step.
- Quadratic weighting gives especially generous credit to near matches.
It would be misleading to select the largest coefficient after seeing the results and present it as the definitive score. The team should choose the loss structure before examining the coefficient, based on what one-, two- and three-step differences mean in the application.
A teacher calibration study may need exact category consistency because each level triggers a different action. A research coding task may care mainly about avoiding large ordinal disagreements. The mathematics stays the same; the decision purpose determines which version is useful.
CHAPTER 9 OF 26 · Check assumptions
9. Why category order is a scientific assumption
Weights require more than four labels. They require an order. “Low, medium, high, very high” has an obvious sequence, although its gaps may still be unequal. Colours, school houses or diagnosis names usually do not have a natural sequence. Assigning them numbers does not create one.
If a nominal set is arbitrarily coded 0, 1, 2 and 3, weighted kappa will treat categories with adjacent numbers as closer. A different coding order can change the coefficient while the actual judgements remain identical. That sensitivity is a warning that the method was applied to the wrong kind of scale.
Before using weights, write a plain-language sentence explaining what a one-step move means. Then ask whether a two-step move is twice as consequential, four times as consequential, or neither. If the answer is unclear, unweighted agreement or a domain-specific loss matrix may be more honest.
Linear weights assume each additional category step adds the same amount of disagreement. Moving from 0 to 1 has the same penalty as moving from 2 to 3, and a two-step gap costs twice a one-step gap.
That is defensible when the categories represent successive, similarly important thresholds. It can also be a useful transparent default because readers can explain the penalties without complicated language. Yet equal increments are still an assumption. Rubric levels may not be equally spaced in skill, risk or consequence.
A practical check is to list one real decision associated with each boundary. If crossing from 2 to 3 changes only feedback wording while crossing from 0 to 1 changes access to support, equal linear penalties may understate the more important boundary. Mathematics reveals this issue because it demands a number for every cell rather than leaving “near agreement” undefined.
CHAPTER 11 OF 26 · Check assumptions
11. Quadratic weights emphasise large separation
Quadratic weights square the normalised distance. A one-step gap in a four-category scale has penalty 1/9, while a two-step gap has 4/9. The two-step disagreement is therefore four times, not twice, the one-step disagreement.
This can make sense when large discrepancies are disproportionately concerning. It is also closely connected in some settings to squared-error ideas and intraclass correlation, though equivalence claims require matching assumptions and should not be stated casually.
The attractive feature of a high quadratic kappa is also its interpretive risk: many adjacent disagreements can receive little penalty. If exact equality matters operationally, report exact agreement and the unweighted table beside the quadratic coefficient. A reader should be able to see whether the strong score reflects true exact consistency or a decision to tolerate near misses.
Linear and quadratic matrices are conventional choices, not laws. A project may have a defensible custom matrix. For example, confusing levels 0 and 1 might be tolerable, while confusing level 2 with level 3 could matter greatly because a new action begins at level 3.
A custom matrix should be specified before analysis, symmetric unless direction genuinely matters, and accompanied by a reason for each unusual value. If the cost of A calling an item high when B calls it low differs from the reverse, the analysis is moving toward directional loss rather than a simple symmetric agreement summary.
Custom weights also make comparison across studies harder. Two teams can analyse the same table and obtain different values. That is acceptable when the decisions differ, but the matrix must travel with the coefficient. Reproducibility depends on preserving the category order, weight convention, table and code or arithmetic.
CHAPTER 13 OF 26 · Check assumptions
13. A Did You Know moment about adjacent cells
Did you know that the pattern of off-diagonal cells can matter more than the number of off-diagonal cells? Two tables can both have 26 exact matches out of 35 and therefore the same unweighted agreement. If one table puts all nine disagreements next to the diagonal while another puts them in opposite corners, weighted results can be very different.
This is why an agreement table should be shown rather than collapsed immediately into one number. The diagonal tells us how often reviewers match. The distance from the diagonal tells us how they differ. The margins show how each reviewer uses the scale.
Students can explore this by moving one count at a time while preserving the total. Moving an item from a one-step cell to a three-step cell leaves exact agreement unchanged but increases both linear and quadratic observed disagreement. The exercise turns a formula into a visible model of error severity.
Agreement compares reviewers with each other. Accuracy compares a classification with a justified reference or truth standard. Two reviewers can agree perfectly and both be wrong. They can also disagree because one uses new information that the other did not receive.
Weighted kappa therefore does not prove that a rubric is valid, that labels predict an outcome, or that a marking decision is fair. Those questions need evidence about the construct, the task, the reference process and possible group effects.
In an AI evaluation, human reviewers may share the same misunderstanding and generate a high coefficient. In school assessment, two markers may reproduce an ambiguous rubric consistently. Mathematics helps by separating claims: this statistic describes agreement under a declared weighting model; it does not establish truth or educational value.
Weighted Cohen kappa retains Cohen’s use of separate reviewer margins. If one reviewer uses high categories much more often, the expected weighted disagreement changes. This can affect the coefficient even when observed weighted agreement looks similar.
That behaviour is part of the statistic’s model. It should not automatically be called a flaw or corrected away. The margins may reveal genuine reviewer severity, different thresholds or different interpretations. They can also reflect the item sample.
Report the margins and inspect them. A simple stacked bar chart or the row and column totals may show a systematic shift that a single coefficient hides. If reviewer A is consistently one category higher than reviewer B, weighted agreement might remain strong because the gap is small, yet the directional shift could matter for decisions. Pair the coefficient with a signed difference summary when direction is important.
An agreement study containing mostly easy, obvious items may produce many diagonal counts. A study designed around difficult boundary cases may produce more near disagreements. The resulting coefficients describe those samples.
This does not mean agreement measures are useless. It means sampling is part of measurement. State how items were selected, what category opportunities were present and whether the sample represents the intended decision environment.
For educational use, avoid comparing two marking teams solely by their weighted kappa if they rated different sets of work. A fairer design gives teams a common, appropriately varied sample or uses a model that accounts for item difficulty. The coefficient cannot repair a sample that excludes the cases where judgement is most challenging.
The 35-item example is complete: both reviewers classify every item. Real projects may contain missing ratings. Simply dropping every item with any missing value can change the sample and the margins, especially if difficult items are more likely to be skipped.
Write the missingness rule before calculation. Record whether “cannot assess” is a substantive category or a missing observation. Those are different meanings. Turning missing data into an ordinary low score changes the scale and can create artificial disagreement.
Some software supports incomplete panels through other agreement coefficients or resampling procedures. The correct method depends on who rated which items and why ratings are absent. A useful mathematics habit is to draw the item-by-reviewer design before choosing a formula. The rectangular data structure often reveals assumptions that a tidy final coefficient would hide.
CHAPTER 18 OF 26 · Explore applications
18. Uncertainty belongs beside the estimate
A coefficient computed from 35 items is an estimate, not a population constant. A different sample can produce a different value. Confidence intervals or a carefully designed bootstrap can show sampling uncertainty, provided the resampling respects the unit of analysis.
Do not treat the 35 paired ratings as 70 independent observations. The pair belongs to one item. If items are clustered within students, classes or documents, a simple item-level interval may also be too optimistic.
The purpose of an interval is not to decorate the score. It is to show how precisely the study distinguishes plausible agreement levels. A wide interval may lead to more calibration data, clearer category descriptions or a larger common sample before a high-stakes decision.
Weighted kappa is most useful inside a learning process. Begin with a shared rubric and a small calibration set. Ask reviewers to classify independently, calculate the table and agreed summaries, then discuss cells that are far from the diagonal as well as repeated adjacent shifts.
Revise examples or boundary language, not past ratings merely to inflate the coefficient. Use a fresh validation set after calibration. Otherwise the team may measure memory of the discussion instead of transfer to new items.
Keep the raw table, weight matrix, category definitions and version of the rubric together. If agreement drifts later, the team can determine whether the items changed, the reviewers changed or the interpretation of a boundary moved. Mathematics becomes a feedback tool rather than a pass-fail badge.
Start with a 3 × 3 table before moving to four categories. Shade the diagonal, label each off-diagonal cell by its distance and compute exact agreement. Then replace every cell count by its count times the chosen penalty.
Next, calculate row and column proportions and build the expected table. Use fractions for a small example so rounding does not hide cancellations. Finally, compare unweighted, linear and quadratic results while keeping the data fixed.
The strongest learning step is explanation. Ask the student to describe one change that raises weighted kappa without changing exact agreement. Then ask whether that change represents better measurement in the real situation. This joins algebra, probability and judgement instead of treating the statistic as button pressing.
- A weighted coefficient is not automatically superior to an unweighted one.
- Numbered categories are not automatically ordered measurements.
- A high quadratic value does not prove exact consistency.
- The largest available coefficient is not the fairest choice.
- Agreement does not establish accuracy, validity or lack of bias.
- Software defaults do not replace a declared weight matrix.
Another misconception is that weights recover information about the true distance between categories. They do not discover distance; analysts supply it. The coefficient then applies that choice consistently.
Correcting these ideas early helps students transfer mathematics to AI labels, medical classifications, design reviews and school rubrics without overclaiming what one statistic can show.
Use the 35-item table to answer these questions before reading the checks.
- Verify all row totals, column totals and the 26 exact matches.
- Recalculate the unweighted reference 313/1225.
- Show why nine adjacent disagreements give linear observed disagreement 3/35.
- Reproduce the expected linear disagreement 487/1225.
- Confirm linear weighted kappa 382/487.
- Reproduce quadratic observed disagreement 1/35 and expected disagreement 401/1575.
- Confirm quadratic weighted kappa 356/401.
Now move one item from a one-step cell to a two-step cell without changing the diagonal. Predict which summaries stay fixed and which change. Exact agreement and unweighted kappa stay fixed if the margins also remain unchanged; weighted quantities may change. If the move alters margins, the expected terms can change too, so calculate rather than guess.
Parents do not need a kappa coefficient to support mathematical learning. The useful lesson is that a score depends on a definition of error. When a child says two answers are “almost the same”, ask what difference remains and whether that difference matters to the decision.
Students can practise with familiar ordered scales: temperature bands, confidence ratings or rubric levels. They should define the labels, justify the order and choose penalties before seeing the final statistic. This builds problem-solving skills, numeracy and the habit of separating data from assumptions.
For project work, encourage a short methods note: who classified what, independently or together, using which categories and which weights. Clear records make mathematics useful in careers because they allow another person to reproduce a decision rather than trust an unexplained number.
Can weighted kappa be negative?
Yes. In the disagreement form, it becomes negative when observed weighted disagreement exceeds the margin-based expected disagreement. That signals performance worse than this reference model, not a universal lower limit on review quality.
Should I use linear or quadratic weights?
Choose the rule that reflects the consequence of category distance and declare it before analysis. Report sensitivity to another defensible rule when the conclusion depends strongly on that choice.
Can I compare coefficients from different scales?
Only cautiously. Different category counts, samples, margins and weights can change the value. Compare the full design, not just the decimal.
Why show exact agreement too?
It tells readers how often labels match without partial credit. Together with the table, it keeps the weighted result interpretable.
The scikit-learn documentation for Cohen’s kappa documents unweighted, linear and quadratic options. The irrCAC kappa table documentation shows a supplied weight matrix, and its weighting overview lists standard and custom choices. Cohen’s original weighted-kappa paper is identified by DOI 10.1037/h0026256.
For the broader series, continue to Cohen’s kappa and marginal agreement and the Mathematics Learning Hub. Those pages provide the unweighted foundation and a route into more mathematics education, problem-solving and career applications.
Before publishing or using a weighted-kappa result, verify:
- the same items were classified independently by two reviewers;
- the category order is real and documented;
- the weight convention and complete matrix are preserved;
- exact agreement, margins and the contingency table are reported;
- the sample represents the intended use;
- uncertainty and missing ratings are handled at the correct unit;
- agreement is not described as accuracy or validity; and
- the conclusion remains sensible under another defensible weight rule.
The importance of mathematics here is practical. It converts “close agreement” into an inspectable model, then makes the cost of every modelling choice visible.
Extended practice laboratory
Practice case 1: Boundary shift
Setup: Move one count from cell 0–1 to 0–2 while holding the diagonal fixed. Task: Recalculate linear and quadratic observed disagreement.
Check: Exact agreement stays fixed; both weighted penalties rise, and quadratic change must be computed from squared distance. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 2: Order audit
Setup: Permute the labels 1 and 2 without changing cell counts. Task: Explain when the coefficient may change.
Check: A genuine ordinal scale cannot be permuted casually because distances encode meaning. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.