Why Mathematics? | Conger’s Kappa, Fixed Panels and Reviewer-Specific References

Imagine three reviewers independently classifying whether a written explanation contains a required justification. Each records A or B for every one of eight explanations, using the same codebook and the same version of the work. Neither category is assumed to be the objectively correct label without additional evidence.

For the elementary calculation here, the panel is fully crossed: the same three reviewers rate all eight objects. This makes each reviewer’s category proportions and each pair’s observed agreement directly comparable on the same set of objects. More complicated assignment or missing-data patterns require additional choices.

Specify the intended population of work and the role of the reviewers. Are these three particular reviewers the focus, or are they intended to represent a wider reviewer pool? The point estimate can be calculated from the matrix, but its broader interpretation depends on selection and assignment. A coefficient cannot turn a convenience panel into a representative sample merely by retaining its separate margins.

Contents · Next chapter

Use one row per explanation and one column per named reviewer. This layout retains who supplied each classification. Keep a stable object identifier so that sorting a file does not accidentally compare labels from different explanations.

Our eight rows are A–A–A, A–A–A, A–B–A, A–B–A, A–B–B, A–B–B, B–B–B and B–B–B. The repeated patterns are intentional and represent distinct synthetic objects. They make the later fractions transparent without implying that repeated copies of one object would create independent evidence.

An object-by-category count table would retain the number of A and B labels on each row, but would remove which reviewer supplied them. That is sufficient for observed pair agreement and some pooled-reference measures. It is insufficient for Conger’s reviewer-specific reference unless the reviewer margins are supplied separately. Keeping the raw matrix protects information that later questions may need.

Contents · Previous chapter · Next chapter

Three reviewers create three unordered pairs on each object: reviewers one and two, one and three, and two and three. A unanimous row gives three matching pairs. A two–one split gives one matching pair and two mismatching pairs.

There are four unanimous rows and four split rows in the example. The matching-pair total is therefore 4 × 3 + 4 × 1 = 16. The total number of available pairs is 8 × 3 = 24. Average observed pair agreement is Pₒ = 16/24 = 2/3.

These twenty-four comparisons are not twenty-four independent sampling units. Pairs share objects and reviewers. Pair counting defines the descriptive numerator and denominator; uncertainty requires a separate account of the sampling design. Confusing these two uses of a denominator can make an otherwise correct point estimate appear much more precise than the study actually supports.

Contents · Previous chapter · Next chapter

Reviewers one and two match on the first two and final two objects, giving four matches out of eight, or 1/2. Reviewers one and three match on six objects, giving 3/4. Reviewers two and three also match on six, giving 3/4.

Averaging these three pair agreement proportions gives (1/2 + 3/4 + 3/4)/3 = 2/3. That matches the within-object pair enumeration. The equivalence holds here because every reviewer pair rates the same number of objects in the fully crossed complete design.

This second view reveals information that the single panel proportion hides: one reviewer pair has lower observed agreement than the other two. That pattern may suggest where to inspect disagreements, but it does not identify a reviewer who is wrong. The codebook and the work must be examined before turning a pairwise difference into a claim about competence, strictness or educational quality.

Contents · Previous chapter · Next chapter

Reviewer one uses A on six objects and B on two, giving proportions 3/4 and 1/4. Reviewer two uses A on two and B on six, giving 1/4 and 3/4. Reviewer three uses each category four times, giving 1/2 and 1/2.

These distributions are deliberately different. Pooling all twenty-four labels would give twelve As and twelve Bs, but that balanced pooled distribution would conceal the contrast between the first two reviewers. Conger’s reference keeps their separate proportions available.

The proportions describe recorded category usage on this sample. They do not establish a stable psychological trait or a universal tendency of a reviewer. Case mix, unclear instructions and ordinary sampling variation can influence them. A useful report presents the counts and asks whether the differences persist on comparable new work rather than assigning a broad personal label from eight invented or observed examples.

Contents · Previous chapter · Next chapter

For reviewers one and two, multiply their corresponding category proportions and add: (3/4)(1/4) + (1/4)(3/4) = 3/8. For reviewers one and three, the reference is (3/4)(1/2) + (1/4)(1/2) = 1/2. The second–third reference is also 1/2.

Each reference is the matching probability for independent labels generated using that pair’s separate marginal distributions. The construction preserves reviewer-specific category usage rather than replacing it with a common pooled distribution.

This probability model does not claim that reviewers literally ignore the object or guess at those rates. It supplies a comparison for interpreting observed matching. Real ratings can reflect shared evidence, shared training and systematic judgement. Those processes need substantive explanation. The reference is useful only if its retained margins and intended interpretation fit the question the panel study was designed to investigate.

Contents · Previous chapter · Next chapter

Conger’s unweighted pairwise reference in this complete design is the mean of the pair-specific marginal-product agreements. In the example, Pₑ = (3/8 + 1/2 + 1/2)/3 = 11/24, approximately 0.4583.

The order of operations matters. First calculate a reference for each distinct reviewer pair using its separate margins, then average those references. Pooling all labels first and squaring the pooled proportions is a different construction. It gives one half for this dataset rather than 11/24.

Both approaches use category frequencies, but they preserve different information. The Conger reference records the fact that the first two reviewers have opposing marginal tendencies. A pooled reference remembers only their combined balance. Naming the retained information is more useful than calling one formula “more advanced”, because the appropriate choice depends on what the study wants to describe about the reviewers and their classification process.

Contents · Previous chapter · Next chapter

Conger’s kappa is κ_C = (Pₒ − Pₑ)/(1 − Pₑ). With Pₒ = 2/3 = 16/24 and Pₑ = 11/24, the numerator is 5/24 and the denominator is 13/24. The ratio is 5/13, approximately 0.3846.

The result expresses observed pair agreement above the reviewer-specific reference relative to the distance from that reference to perfect agreement. It is not the percentage of objects with unanimous labels. Unanimity is four of eight, or one half. Average pair agreement is two thirds.

Keep these quantities separate in a report. A panel may produce a majority decision on every binary object while still containing substantial pairwise disagreement. The fact that all eight rows have a majority does not mean that the reviewers all agreed or that the majority labels are valid. The coefficient summarises one defined relationship among the ratings, not every desirable property of the panel’s decisions.

Contents · Previous chapter · Next chapter

For reviewers one and two, Cohen’s kappa is (1/2 − 3/8)/(1 − 3/8) = 1/5. For one and three it is (3/4 − 1/2)/(1 − 1/2) = 1/2. The second–third pair also has kappa 1/2.

Their arithmetic mean is (1/5 + 1/2 + 1/2)/3 = 2/5 = 0.40. Conger’s panel coefficient is 5/13, approximately 0.3846. The difference is not rounding. Each pair has been normalised by a different remaining distance to perfection before those kappas were averaged.

The mean of ratios need not equal a ratio built from the mean numerators and denominators. This familiar algebraic lesson appears in many statistical summaries. Before averaging coefficients, ask whether the intended quantity is an average of pair-specific adjusted relationships or one adjustment to the panel’s average observed agreement. Those are distinct summaries, even when they are computed from the same rating matrix.

Contents · Previous chapter · Next chapter

Write κ_ab for a pair’s kappa and e_ab for its reference agreement. Then observed agreement minus reference for that pair is (1 − e_ab)κ_ab. Summing across pairs gives Conger’s coefficient as Σ[(1 − e_ab)κ_ab]/Σ(1 − e_ab), when the pairwise quantities are defined.

In the example, the weights are 5/8, 1/2 and 1/2. The weighted numerator is (5/8)(1/5) + (1/2)(1/2) + (1/2)(1/2) = 5/8. The denominator is 13/8. Their ratio is 5/13 again.

These are algebraic weights based on the reference denominator, not inverse-variance weights and not a judgement about which reviewers deserve more influence. Keeping that distinction clear prevents a useful identity from being misinterpreted as a statistical efficiency claim. It also explains exactly why the unweighted average of the three pairwise kappas gave a different answer.

Contents · Previous chapter · Next chapter

Across the panel there are twelve A and twelve B labels, so pooled proportions are one half and one half. A Fleiss reference is therefore 1/2. With the same observed pair agreement 2/3, Fleiss’ kappa is (2/3 − 1/2)/(1 − 1/2) = 1/3.

Conger’s value is 5/13, which is larger here because its reference 11/24 is smaller than the pooled reference 1/2. No rating changed between the calculations. The difference comes from whether reviewer-specific variation in category proportions is preserved in the benchmark.

The dedicated Fleiss kappa guide develops the pooled count-table mechanism. Comparing it with this example is useful when choosing a method, provided the report explains the design and reference. Selecting a larger coefficient after inspecting several methods does not itself justify changing the benchmark.

Contents · Previous chapter · Next chapter

Let p_aj be reviewer a’s proportion in category j and let p̄_j be the mean across r reviewers. Conger’s reference can also be written as Σp̄_j² − Σs_j²/r, where s_j² is the sample variance of the reviewer proportions for category j, using denominator r − 1.

For A, the proportions are 3/4, 1/4 and 1/2. Their sample variance is 1/16. The B proportions have the same variance. The pooled squared-proportion sum is 1/2, so the Conger reference is 1/2 − (1/16 + 1/16)/3 = 1/2 − 1/24 = 11/24.

This form makes the distinction from the pooled reference explicit. The subtraction accounts for variation among reviewer margins in the pairwise reference, not for sampling error in those margins. It does not estimate how much disagreement is caused by reviewer bias. A causal interpretation would require a design and evidence beyond this algebraic decomposition.

Contents · Previous chapter · Next chapter

If every reviewer has the same category proportions, the between-reviewer marginal variances are zero. Conger’s reference then equals the pooled squared-proportion reference. With the same observed agreement, the two coefficients coincide, provided the denominator is nonzero.

In the complete unweighted design, the pooled reference is at least Conger’s reference because it differs by a sum of nonnegative variance terms. For fixed observed agreement below one and nondegenerate references, the larger reference gives the smaller adjusted coefficient. This explains the ordering in our worked example.

The conclusion belongs to the stated definitions. Weighted versions, unequal assignment and missing ratings need their own examination. Do not extend a clean identity beyond its conditions merely because a software function uses the same surname. A useful mathematical habit is to attach every comparison theorem to the data structure and formulas under which it was established.

Contents · Previous chapter · Next chapter

With only two reviewers, there is exactly one distinct reviewer pair. Conger’s pairwise reference is then that pair’s separate marginal-product reference, and observed pair agreement is its usual diagonal agreement. The coefficient reduces to Cohen’s kappa.

This is one reason to distinguish Conger’s generalisation from a pooled-reference multi-rater coefficient. With two reviewers, the pooled approach corresponds to a shared category distribution, while the separate-margin approach preserves the pair’s individual frequencies.

The reduction is a useful implementation check. Supply a complete two-column rating matrix, form its square contingency table, and verify that the Conger calculation and the ordinary Cohen calculation agree. This checks the meaning of the reference, not just the syntax of an input file. It also helps students see how a general formula should behave when its number of reviewers is reduced to the simplest nontrivial case.

Contents · Previous chapter · Next chapter

Suppose the four split objects retain their A/B counts, but the minority label is reassigned among reviewers. Average pair agreement remains two thirds because the within-object patterns are unchanged. The individual reviewer margins may change, however, which can alter Conger’s reference.

An aggregated object-by-category table cannot tell us which reassignment occurred. Different raw rating matrices can therefore share that table while giving different reviewer-specific benchmarks. If an analysis needs Conger’s coefficient, preserve identities from the start rather than hoping to reconstruct them from category totals later.

This is a broader lesson about data compression. A summary may be sufficient for one statistic and insufficient for another. The decision about what to retain should follow the questions likely to matter, including reviewer-specific diagnostics and missingness. A compact table is convenient, but convenience does not make discarded information irrelevant to every later analysis.

Contents · Previous chapter · Next chapter

Our derivation assumes that every reviewer rates every object. If labels are missing, different reviewer pairs may be observed on different sets of objects. A pair’s raw agreement can then reflect its case mix as well as the relationship between its reviewers.

Implementations can define extensions using available labels and separate reviewer proportions, but their weighting rules need to be read explicitly. Averaging pair-specific agreements without regard to unequal coverage is not automatically the same as averaging object-level pair agreement. Filling missing labels from a majority decision would invent observations.

Report which reviewers rated which objects and why ratings are absent. If difficult explanations are more likely to be skipped, the available comparisons may be selected. A function that accepts missing values does not establish that the data support the intended generalisation. The simple complete example provides a reference point from which to understand exactly what an extension changes.

Contents · Previous chapter · Next chapter

The A/B example uses exact nominal matching. With more categories, the same pair-specific marginal-product idea extends by summing over corresponding category proportions. The categories must have aligned meanings for every reviewer.

If the task uses ordered levels, a weighted agreement model can give partial credit for nearby judgements. The weighting rule changes observed and reference agreement and must be justified before interpreting the coefficient. A numerical code is not automatically a measurement scale with equal gaps.

State the weight matrix or convention when reporting a weighted result. Explain what an adjacent mismatch means in the application and why it should receive the chosen credit. The fact that a particular set of weights increases the coefficient is not sufficient justification. The educational meaning of disagreement should determine the arithmetic, especially when the result will influence a consequential decision.

Contents · Previous chapter · Next chapter

All three reviewers could apply the same mistaken interpretation and agree perfectly. Conger’s kappa could then be one when the reference is nondegenerate, while the shared classification still failed to capture the intended reasoning skill.

Validation asks whether the categories and evidence support the inference being made. It may involve inspecting the codebook, comparing with an appropriate independent reference or evaluating performance on different tasks. Agreement contributes evidence about reproducibility, but it does not supply the entire validity argument.

For a parent, a useful question is what decision the labels support. Are they used to identify a topic for discussion, allocate practice or make a placement decision? Those uses have different consequences of disagreement. A coefficient should be interpreted alongside those consequences and the original counts, rather than treated as a universal certificate of quality for the reviewers or the assessment system.

Contents · Previous chapter · Next chapter

Eight objects are enough to teach the arithmetic but not to support a precise general claim about a panel’s agreement. Another set of explanations can change observed matching and each reviewer’s marginal proportions. Both changes affect the coefficient.

An interval method should account for the structure of the study. The three pairs share reviewers, and each object contributes to all three pairwise summaries. Treating the pairwise kappas as independent estimates and applying a standard formula for three unrelated numbers would ignore that dependence.

When objects are sampled independently, resampling whole object rows is one way to preserve the within-object rating structure, though the exact interval procedure and its assumptions still need justification. If responses are clustered within students or documents, that clustering also matters. A precise-looking interval derived from the wrong independent unit can be less useful than a transparent descriptive report that honestly states the limits of the available sample.

Contents · Previous chapter · Next chapter

The first and second reviewers differ on the middle four objects in our example. The third reviewer matches one on two of those objects and the other on two. This pattern suggests inspecting which evidence separates those middle cases, rather than immediately assigning one reviewer as the standard.

After the independent round, compare explanations and identify the exact codebook clause each reviewer used. A clarification might specify whether an implied reason counts or whether the justification must be explicit. Record both the original labels and the revised instruction so that the change is reviewable.

Test the revision on fresh work when generalisation matters. Improved matching on previously discussed objects can show successful calibration, but memory and shared discussion are part of that training situation. The practical aim is a clearer classification process that works on new evidence, with a coefficient serving as one summary of the resulting observations.

Contents · Previous chapter · Next chapter

Keep an object identifier and named reviewer columns. For each reviewer, count every category and divide by that reviewer’s number of rated objects. In the complete example those denominators all equal eight. Check that each reviewer’s proportions sum to one.

Create a separate table of distinct reviewer pairs. Calculate observed matching and the marginal-product reference for each pair, then average the observed values and references before forming Conger’s ratio. Independently calculate the reference from pooled proportions minus the marginal-variance correction.

Verify that reordering objects or renaming category symbols changes nothing. Renaming the reviewer columns also changes nothing if their full data move with them. Reassigning individual labels among reviewers can change the reference even if each object’s category counts remain fixed. That distinction tests whether the spreadsheet has preserved reviewer identity rather than accidentally collapsing the data into a pooled count analysis.

Contents · Previous chapter · Next chapter

Reproduce the three pair-specific observed agreements: one half, three quarters and three quarters. Their mean is two thirds. Then reproduce the three references: three eighths, one half and one half. Their mean is eleven twenty-fourths, giving Conger’s kappa five thirteenths.

Calculate each pair’s Cohen kappa. The answers are one fifth, one half and one half. Their arithmetic mean is two fifths. Explain why this differs from five thirteenths by identifying the three denominator weights, five eighths, one half and one half.

For a final exercise, keep only reviewer columns one and two. Conger’s formula now has one pair and must give one fifth, matching Cohen’s kappa for those two reviewers. This reduction is a structural check on the method. It is more informative than expecting every coefficient with the word kappa in its name to return the same result on a two-reviewer dataset.

Contents · Previous chapter · Next chapter

A parent reading a panel report can ask whether the same reviewers assessed every object and whether their category frequencies were kept separate. Ask what “panel agreement” means: unanimous decisions, majority decisions or pairwise matching. The answers clarify what the reported coefficient actually summarises.

A student can use the example to study averages of ratios. Calculate the three pairwise kappas, average them, and then compare that result with a ratio built from averaged observed and reference agreement. The difference exposes the role of each normalising denominator.

The transferable idea is that aggregation should preserve the information needed by the question. Reviewer identities are essential for one benchmark and unnecessary for another. Learning to notice that distinction builds stronger judgement about grouped data in many subjects, including education, quality control and collaborative classification work, without suggesting that one statistic can replace an explanation of the underlying process.

Contents · Previous chapter · Next chapter

Is Conger’s kappa the arithmetic mean of Cohen kappas? Generally no. In the complete example it is a denominator-weighted mean, giving five thirteenths instead of two fifths. Does it reduce to Cohen’s kappa for two reviewers? Yes for the complete unweighted pairwise construction shown here.

Why does it differ from Fleiss’ kappa? Conger retains reviewer-specific marginal proportions. Fleiss uses pooled category proportions. They coincide when the relevant reviewer margins are equal, under the stated complete-data formulas.

Can I calculate it from category counts alone? Not generally without reviewer-margin information. Does the marginal correction prove reviewer bias caused disagreement? No. It is an algebraic reference adjustment, not a causal decomposition. Can missing ratings be ignored? Their pattern and the implementation’s weighting rules need examination because pairs may then concern different sets of objects.

Contents · Previous chapter · Next chapter

The official irrCAC Conger documentation describes the raw-rating interface. The package source exposes its reviewer-specific category proportions and marginal-variance reference. Conger’s foundational paper is Integration and Generalization of Kappas for Multiple Raters.

The eight-object matrix and all calculations in this guide are synthetic worked analysis. The pair enumeration, separate-margin reference and equivalent variance form provide independent ways to inspect the result. They are not empirical claims about a real panel or estimates of a school’s marking quality.

Continue through the Mathematics Learning Hub and rearrange minority labels among reviewers while keeping each object’s category counts fixed. Predict which summaries must remain unchanged and which can change, then calculate. Explaining that distinction is the central learning step behind the panel coefficient.

Contents · Previous chapter · Mathematics Learning Hub