Why Mathematics? | Scott’s Pi, Pooled Margins and Shared Category Models

Imagine two reviewers reading the same thirty short mathematical explanations. Each assigns one nominal code: A, B or C. These codes might represent three mutually exclusive types of reasoning in a carefully written classification scheme. For this example, the letters are category names, with no numerical distance or ranking.

The thirty explanations are the objects. The two reviewers supply paired labels on each object. An object identifier connects the labels, even if the files are sorted differently. That pairing is essential because agreement concerns whether the reviewers classify the same explanation in the same way.

Before collecting labels, define each category and explain how a borderline response should be handled. If the categories overlap, a reviewer may make a defensible choice that differs from another reviewer’s. The resulting disagreement can reveal a codebook problem, but the coefficient cannot identify that problem without inspecting the work and the classification instructions that generated the labels.

Contents · Next chapter

Observed agreement is the fraction of objects receiving matching labels. It answers a direct counting question. A chance-adjusted agreement measure adds a comparison model, asking how the observed matching proportion relates to a specified baseline constructed from category probabilities.

The word “chance” does not establish that reviewers guessed or that a known fraction of labels was produced carelessly. It identifies a mathematical reference. Real reviewers may share training, notice the same misleading clue or interpret a criterion in different ways. Those mechanisms need substantive investigation.

Scott’s pi uses one pooled category distribution for both draws in its reference. Cohen’s kappa instead retains the two reviewers’ separate category distributions. Neither formula can be understood fully by looking only at a final decimal. State the benchmark in words, retain the raw table, and explain which rating process the study intends to describe. That combination makes the adjusted result open to meaningful scrutiny.

Contents · Previous chapter · Next chapter

Use reviewer A’s labels for rows and reviewer B’s labels for columns. The three rows of synthetic counts are (12, 3, 0), (1, 6, 2) and (0, 1, 5). There are thirty objects altogether. The row totals are fifteen, nine and six; the column totals are thirteen, ten and seven.

The cell containing three means A assigned category A while B assigned category B on three explanations. It is a count of paired outcomes, not a judgement that three explanations were objectively incorrect. Neither reviewer has been appointed as an error-free standard.

The diagonal counts are twelve, six and five. They give twenty-three matching classifications and seven mismatches. Reading the table before calculating reveals both the amount of matching and the direction of particular disagreements. A single coefficient will compress these distinctions, so the table should remain available if the report is intended to guide a change in the classification procedure.

Contents · Previous chapter · Next chapter

The diagonal sum is 12 + 6 + 5 = 23. Dividing by thirty gives Pₒ = 23/30, approximately 0.7667. This is the directly observed proportion of matching labels, with one contribution per object.

It is useful to check the complement. The off-diagonal cells sum to 3 + 1 + 2 + 1 = 7, and 23 + 7 = 30. The mismatch proportion is therefore 7/30. These elementary checks are often more effective than inspecting a complex formula after an input error has already propagated through it.

Do not interpret 23/30 as accuracy. If both reviewers systematically misclassify the same kind of explanation, their labels can match while their shared interpretation remains questionable. Conversely, a well-defined but difficult boundary can produce genuine disagreement. Observed agreement describes matching under the stated task; validity requires evidence that the task and categories represent the intended educational distinction.

Contents · Previous chapter · Next chapter

There are sixty labels because each of the thirty objects has two ratings. Category A appears fifteen times in the row totals and thirteen times in the column totals, giving twenty-eight labels. Categories B and C appear nineteen and thirteen times respectively.

The pooled proportions are p_A = 28/60 = 7/15, p_B = 19/60 and p_C = 13/60. They sum to one. Each pooled proportion is the average of the two reviewers’ marginal proportions for that category because the reviewers classified the same number of objects.

This averaging is a modelling choice. It preserves overall category usage while discarding the difference between the reviewers’ separate frequencies. A pooled A proportion of one half could arise because both reviewers use A half the time, or because one uses A much more often than the other. Those situations can have different substantive meanings even though they contribute the same pooled probability to Scott’s reference.

Contents · Previous chapter · Next chapter

Scott’s expected agreement is Pₑ = Σpⱼ², the sum of squared pooled category proportions. It is the matching probability for two independent draws from that pooled distribution, using a with-replacement probability construction.

For our example, Pₑ = (28/60)² + (19/60)² + (13/60)². The squared counts sum to 784 + 361 + 169 = 1,314. Dividing by 3,600 gives 73/200 = 0.365. Keeping exact fractions makes the later simplification easy to audit.

This reference does not shuffle the original thirty paired rows while preserving each reviewer’s margin separately. Such a construction would correspond to a different benchmark. Nor does it estimate the proportion of actual labels that arose from guessing. Its role is specific: provide a shared-distribution matching baseline against which observed agreement can be normalised. The usefulness of that baseline depends on the intended interpretation of the coding process.

Contents · Previous chapter · Next chapter

Scott’s pi is π = (Pₒ − Pₑ)/(1 − Pₑ), provided Pₑ is below one. Substituting the checked values gives π = (23/30 − 73/200)/(1 − 73/200). The numerator is 241/600 and the denominator is 381/600.

The ratio is 241/381, approximately 0.6325. This result describes observed agreement above the pooled reference as a fraction of the distance from that reference to perfect agreement. It is not the percentage of explanations classified correctly or the percentage on which both reviewers matched.

The raw matching proportion remains approximately 0.7667. The difference between that proportion and pi is produced by the reference and normalisation, not by deleting some of the twenty-three matches. Keeping the original counts alongside the adjusted result prevents the formula from being interpreted as a mysterious correction to what was actually observed.

Contents · Previous chapter · Next chapter

Let N be the number of paired objects, D the diagonal count, and sⱼ the sum of the row and column totals for category j. Then the pooled probability is sⱼ/(2N), and Scott’s pi can be written as [4ND − Σsⱼ²]/[4N² − Σsⱼ²].

Here 4ND = 4 × 30 × 23 = 2,760. The squared pooled-count sum is 1,314. The numerator is 1,446 and the denominator is 3,600 − 1,314 = 2,286. Reducing 1,446/2,286 gives 241/381 again.

This is an independent organisation of the same arithmetic. It is useful in a spreadsheet because it avoids rounding pooled proportions before squaring them. Retain both the original count table and the intermediate totals. If the two forms disagree materially, investigate the table layout, the factor of two in pooled labels and the selected diagonal before assuming that the discrepancy is a harmless software rounding issue.

Contents · Previous chapter · Next chapter

For the same table, Cohen’s reference is the sum of corresponding row-proportion and column-proportion products. It is [15 × 13 + 9 × 10 + 6 × 7]/30² = 327/900 = 109/300, approximately 0.3633.

Cohen’s kappa is therefore (23/30 − 109/300)/(1 − 109/300) = 121/191, approximately 0.6335. Scott’s pi was about 0.6325. The values are close because the two reviewer margins are close: fifteen versus thirteen, nine versus ten, and six versus seven.

The numerical closeness should not obscure the conceptual difference. Cohen retains reviewer-specific category frequencies; Scott pools them. A different dataset can make the distinction much more visible. The relevant question is which reference matches the study’s intended interpretation, rather than whether the difference happens to be small enough to disappear when both coefficients are rounded to one decimal place.

Contents · Previous chapter · Next chapter

For one category, write the two reviewer proportions as a and b. Scott contributes [(a + b)/2]² to its reference, whereas Cohen contributes ab. Their difference is (a − b)²/4. Summing across categories gives Pₑ, Scott − Pₑ, Cohen = Σ(aⱼ − bⱼ)²/4.

Every squared term is nonnegative, so Scott’s pooled reference is at least Cohen’s separate-margin reference. Equality holds when the reviewer proportions match in every category. This statement concerns the unweighted two-reviewer formulas used here, with the same complete paired dataset.

For our table, the proportion differences are 2/30, −1/30 and −1/30. Their squared sum divided by four is 1/600. Indeed, 73/200 − 109/300 = 1/600. The derivation turns a small numerical discrepancy into an explainable measure of marginal asymmetry. It also provides a useful check that goes beyond rerunning either coefficient formula.

Contents · Previous chapter · Next chapter

For fixed observed agreement Pₒ below one, the function f(e) = (Pₒ − e)/(1 − e) decreases as e increases, provided e is below one. Its derivative is (Pₒ − 1)/(1 − e)², which is negative under those conditions.

Combining this fact with the squared-difference identity shows that Scott’s pi is no greater than Cohen’s kappa for the same unweighted complete paired table when both are defined. Equal reviewer margins give equal references and equal coefficients. Perfect observed agreement also requires matching margins and gives both coefficients one when the reference is not degenerate.

This ordering is an algebraic relationship, not a ranking of the methods’ usefulness. A smaller coefficient can reflect a different benchmark rather than a better diagnosis. Explain why pooling is appropriate, and do not treat a reviewer’s marginal tendency as either an error to remove or a feature to preserve without considering what the rating design is meant to measure.

Contents · Previous chapter · Next chapter

Consider a second synthetic table with rows (6, 8) and (0, 6). There are twenty objects and twelve matches, so observed agreement is 0.60. Reviewer A uses the first category fourteen times, while reviewer B uses it six times. The pooled proportions are one half and one half.

Scott’s reference is 0.50, giving pi (0.60 − 0.50)/(1 − 0.50) = 0.20. Cohen’s reference is 0.70 × 0.30 + 0.30 × 0.70 = 0.42, giving kappa 0.18/0.58 = 9/29, approximately 0.3103.

The difference is larger than in the thirty-object example because the reviewer margins differ more. Inspecting the eight mismatches also reveals a completely directional pattern: A uses the first category where B uses the second. A pooled coefficient alone cannot explain that pattern. The table and reviewer-specific frequencies are needed to investigate whether the category boundary or reviewer instruction should change.

Contents · Previous chapter · Next chapter

Pooling treats the category probabilities in the reference as common to both reviewers. This can be useful when reviewers are regarded as drawing on a shared coding process. It can be less informative when the purpose is to retain stable differences among a fixed pair of reviewers.

The fact that a pooled distribution can be calculated does not establish that the reviewers are interchangeable. Selection, training, task allocation and exposure to different evidence all matter. If one reviewer handles mainly difficult examples while another sees a different set, a simple paired analysis may not even be available without redesigning the collection of ratings.

Write the intended interpretation in a sentence before selecting the coefficient. Is the study describing a common classification process, comparing two named reviewers, or checking an operational decision rule? Those purposes can lead to different reference choices. The arithmetic should follow the question rather than silently deciding it on the researcher’s behalf.

Contents · Previous chapter · Next chapter

Each object supplies two dependent labels because both labels refer to the same object. That pairing is necessary for observed agreement. It does not conflict with using independent draws in a reference probability model; the reference is a comparison construction, not a claim that the actual labels ignore the object.

For uncertainty, a separate issue arises: are the objects independent sampling units? Several explanations from the same student may share language or reasoning habits. Extracts from the same document may share context. Treating all such objects as unrelated can make an interval look more precise than the design supports.

Record the sampling unit and the intended population. A descriptive exercise on thirty invented objects needs no population claim. A real evaluation intended to generalise to new work needs a defensible sampling argument and an interval method that respects any clustering. Mathematical clarity includes knowing which independence statement belongs to the benchmark and which belongs to inference.

Contents · Previous chapter · Next chapter

The simple two-reviewer table used here contains only objects with both labels. If one reviewer’s label is absent, the object cannot be assigned to a matching or mismatching cell without inventing information. A complete-pair calculation therefore describes the retained pairs.

That selection may matter. Reviewers might skip unreadable work or explanations they find difficult to classify. Removing those objects can leave an easier task and raise apparent agreement. A blank is not automatically equivalent to a substantive “uncertain” category, which would need to have been part of the actual coding instructions.

Report intended objects, complete pairs and the reasons for missing labels. For a more complex incomplete design, a framework such as Krippendorff’s alpha makes explicit choices about pairable values and weights. Its ability to use incomplete matrices does not make its reference identical to Scott’s, nor does any available coefficient establish that missingness is harmless.

Contents · Previous chapter · Next chapter

The pooled reference rises when category use becomes more concentrated. With q categories, the squared-proportion sum is smallest at the uniform distribution, where it equals 1/q. If almost every label uses one category, the reference can approach one.

That behaviour explains why a high raw matching rate can coexist with a modest pi. It also means that comparisons across different case mixes need care. A sample deliberately balanced across categories answers a different empirical question from a representative sample in which one label is very common.

Report both the natural category frequencies and any deliberate sampling choices. Balancing cases may be useful for reviewer training or studying rare boundaries, but the resulting coefficient should not automatically be presented as the agreement expected in ordinary use. A well-labelled study can be valuable for a narrow purpose without claiming to represent a broader population that it was not designed to sample.

Contents · Previous chapter · Next chapter

The unweighted calculation treats every mismatch equally. If labels describe ordered levels, a researcher may want to give partial credit for nearby classifications. That introduces a weight matrix and changes the definition of agreement as well as the reference calculation.

Do not infer distances from category numbers alone. Codes one, two and three might be mere storage labels, or they might denote ordered categories with unequal substantive gaps. A weighting choice should be justified using the task, with its convention stated clearly enough for another analyst to reproduce.

The squared-margin identity and coefficient ordering derived earlier concern the unweighted formulas in this guide. They should not be carried over casually to every weighted implementation. When a model changes, recheck its algebra and assumptions. This is a useful general habit: a theorem belongs to the definitions under which it was proved, not to every statistic that happens to share a similar name.

Contents · Previous chapter · Next chapter

If both reviewers use only category A on every object, observed agreement is one and the pooled reference is also one. Scott’s ratio then has a zero denominator and is undefined. Describe the data directly instead of forcing an adjective onto a nonexistent estimate.

If one reviewer always uses A and the other always uses B, observed agreement is zero and the pooled proportions are balanced. Scott’s pi is −1. Cohen’s separate-margin reference is zero in that particular table, giving kappa zero. The contrast reflects their different benchmarks, not disagreement about the raw labels.

Such extreme cases are useful checks because they reveal what the reference preserves. A shared pooled distribution regards the opposite constant labels differently from a model preserving each reviewer’s constant habit. Neither result makes the classifications operationally acceptable. The direct observation remains that the reviewers never matched, which is often the most important fact for the intended use.

Contents · Previous chapter · Next chapter

Consider a balanced binary table with no diagonal matches and ten objects in each off-diagonal cell. Each reviewer uses each category ten times, but their labels are opposite on every object. Scott’s pooled reference is one half and pi is −1. The observed matching rate is zero.

There is nevertheless a perfectly predictable relationship between the labels: knowing one reviewer’s category tells us the other reviewer’s category exactly. With ordinary zero–one coding, their correlation is −1. Perfect predictability therefore does not establish exact agreement. An association measure can recognise a deterministic relationship that an agreement measure correctly records as systematic mismatch.

If the apparent reversal is a data-coding error, correcting the documented mapping may restore the intended alignment. If the categories really mean the same things for both reviewers, relabelling only one axis merely to create a diagonal changes the question and should not be treated as a harmless cosmetic step. The meaning of the category dictionary determines which action is justified.

This example is useful whenever a report uses the word “consistent” without saying whether it means matching labels, similar ordering or predictability after a transformation. Those properties can diverge. Decide which relationship the intended use needs, preserve the original coding and choose the summary accordingly. A high absolute correlation cannot replace evidence that two reviewers will make the same operational classification.

Contents · Previous chapter · Next chapter

The exact fraction 241/381 is an exact result for our synthetic table, not a precise estimate of a population agreement level. Another sample can change the diagonal count and the pooled proportions. Confidence intervals need to account for both sources of variation.

A binomial interval for twenty-three matches out of thirty concerns raw agreement under its assumptions. It is not automatically an interval for pi. An appropriate estimator-specific method or a properly designed resampling procedure is needed for the adjusted coefficient, with clustering respected where relevant.

A test against pi equal to zero addresses the pooled reference. It does not show that the agreement is sufficient for a consequential decision. Define the practical standard from the use of the labels and the consequences of disagreement. A large sample can detect a small excess above reference that remains inadequate, while a small sample can leave substantial uncertainty around a promising observed pattern.

Contents · Previous chapter · Next chapter

The thirty-object table has three A–B mismatches but only one B–A mismatch. It also contains two B–C and one C–B mismatches. Those counts suggest which category boundaries are worth inspecting, without proving why the reviewers differed.

After the independent rating round, review the original explanations and identify the exact evidence each reviewer used. Perhaps one requires an explicit justification while another accepts an implied step. A codebook revision can make that distinction clearer. Record the original ratings before discussion so that the initial agreement remains auditable.

Evaluate a revised codebook on fresh objects when generalisation is the goal. Re-rating the same discussed examples can be useful training, but memory of the discussion can increase matching. The useful improvement is a clearer and more defensible classification procedure. A larger coefficient is evidence to interpret within that process, not a replacement for explaining what changed and why it matters.

Contents · Previous chapter · Next chapter

Reproduce the pooled category counts in the main example. They are twenty-eight, nineteen and thirteen. Explain why their total is sixty rather than thirty: the table contains thirty objects but two labels per object. Then calculate the squared-count reference, 1,314/3,600 = 73/200.

For a balanced binary table with cells 7, 3, 3 and 7, observed agreement is 0.70 and the pooled reference is 0.50. Scott’s pi is 0.40. The two reviewer margins are equal, so Cohen’s kappa is also 0.40. State the condition that makes the two benchmarks coincide.

For the asymmetric table with cells 6, 8, 0 and 6, verify pi 0.20 and kappa 9/29. Use the squared-margin difference to check the references: (0.40² + (−0.40)²)/4 = 0.08, exactly the gap between 0.50 and 0.42. This checks the mechanism as well as the final arithmetic.

Contents · Previous chapter · Next chapter

A parent reading a reliability report can ask whether the reviewers’ category proportions were pooled or kept separate. Ask to see the raw agreement and the table, especially when the adjusted coefficient seems surprising. These questions often reveal more than a verbal band such as “good” or “moderate”.

A student can connect Scott’s pi to fractions, independent probability, averaging and a simple squared-difference identity. The advanced learning lies in interpreting the pooling step. The same average can conceal very different individual patterns, and a summary based on that average cannot recover the information it discarded.

Use invented labels and small tables while learning. There is no need to turn classmates’ work into an informal ranking of reviewers. The valuable habit is to ask what a statistic preserves, what it pools and which benchmark it uses. Those questions transfer to many other comparisons in education, science and everyday data reporting.

Contents · Previous chapter · Next chapter

Is Scott’s pi the same number as raw agreement? No. It compares the observed matching proportion with a pooled-category reference. The worked table has raw agreement 23/30 and pi 241/381. When does it equal Cohen’s kappa? Equal reviewer category proportions give the same reference under the complete unweighted formulas shown here.

Does pooling establish that reviewers are interchangeable? No. That interpretation needs a study-design argument; pooling merely performs the specified calculation. Can pi be negative? Yes, when observed matching is below the pooled reference. Why can a perfectly predictable reversal have poor agreement? Predictability and matching are different relationships when category meanings stay aligned.

Can I ignore an absent label? Complete-pair analysis changes the analysed set, so missingness and exclusions should be reported. Does high agreement establish correct labels? Shared mistakes can produce matching labels. Accuracy and the validity of the coding scheme require additional evidence beyond an agreement coefficient.

Contents · Previous chapter · Next chapter

The official irrCAC documentation for Scott’s coefficient describes square-table input and weighting options. The package implementation exposes the pooled proportions and reference. Scott’s original paper is Reliability of Content Analysis: The Case of Nominal Scale Coding. The original tables and algebraic comparisons in this guide are synthetic worked analysis.

For a neighbouring method, the dedicated Cohen’s kappa guide retains separate reviewer margins. Read the two references side by side and explain their probability constructions before comparing the reported coefficients.

Continue through the Mathematics Learning Hub and construct two paired tables with the same observed matching rate but different marginal asymmetry. Predict which benchmark gap will be larger, then verify it using Σ(aⱼ − bⱼ)²/4. The prediction is a stronger demonstration of understanding than calculating two unexplained decimals.

Contents · Previous chapter · Mathematics Learning Hub