Choose your reading route
eduKateSG · Why Mathematics? · PABAK, Prevalence, Bias and Balanced Agreement References
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 4–7
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 8–13
Check assumptions
See where design and calibration matter.Route 4 · Chapters 14–18
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 19–24
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when two reviewers agree often but Cohen’s kappa looks unexpectedly modest? In a binary table, the prevalence-adjusted bias-adjusted kappa, usually shortened to PABAK, replaces the observed margins with a balanced reference and simplifies to twice observed agreement minus one. That can be illuminating, but it does not make prevalence disappear from the study or turn a convenient benchmark into truth.
This chapter uses an original 100-item table to separate observed agreement, reviewer bias, category prevalence and the reference model. It explains the Byrt–Bishop–Carlin identity, reproduces each fraction, and shows why good reporting places PABAK beside the original table and ordinary kappa. The goal is careful reasoning for education, research, quality review and AI classification rather than a search for the largest coefficient.
Full chapter index
Chapters 1–3 · Understand the question
Chapters 4–7 · Reproduce the example
Chapters 8–13 · Check assumptions
Chapters 14–18 · Explore applications
Chapters 19–24 · Study and verify
Imagine two reviewers classify 100 items as positive or negative. They agree on 88 items, which sounds strong. Yet ordinary Cohen kappa is only about 0.434 in our example. Nothing is arithmetically wrong. The reviewers use the positive category very frequently, so the margin-based chance reference is already high.
PABAK asks what the coefficient would be under a different reference: balanced positive and negative use with no marginal bias. It gives 0.760. The gap between 0.434 and 0.760 is not a contest with one correct winner. It reveals how much the reference model matters.
Mathematics helps because the two numbers can be decomposed. Instead of calling one “paradoxical”, we inspect the four cells, observed agreement, two margins, prevalence index and bias index.
The synthetic cells are a = 82, b = 8, c = 4 and d = 6. Cell a contains positive-positive agreements; d contains negative-negative agreements. Cells b and c are the two directions of disagreement. The total is 100.
Reviewer A calls 90 items positive and 10 negative. Reviewer B calls 86 positive and 14 negative. The reviewers agree on 82 + 6 = 88 items. They disagree on 12, with eight A-positive/B-negative cases and four A-negative/B-positive cases.
Always preserve this orientation. Swapping labels or reviewers can change the signs of descriptive indices even when kappa is unchanged. A table is more informative than a coefficient because it shows whether agreement is concentrated in one category and whether disagreements point mainly in one direction.
| Reviewer A / B | Positive | Negative | Row total |
|---|---|---|---|
| Positive | 82 | 8 | 90 |
| Negative | 4 | 6 | 10 |
| Column total | 86 | 14 | 100 |
Observed agreement is 88/100 = 0.88. The separate-margin reference is (0.90 × 0.86) + (0.10 × 0.14) = 0.774 + 0.014 = 0.788.
Cohen kappa is (0.88 − 0.788)/(1 − 0.788). The numerator is 0.092 and the denominator is 0.212, giving 23/53, approximately 0.434.
The coefficient is modest because the margin model predicts agreement of 78.8% before paired information is used. That does not prove most observed agreement is meaningless. It states what remains after comparison with this particular independent-margin benchmark. The table contains the evidence; kappa is one summary of it.
For a binary table, PABAK is 2Po − 1, where Po is observed agreement. With Po = 0.88, the result is 1.76 − 1 = 0.76.
The same value arises from a chance-corrected form with expected agreement fixed at 0.5: (0.88 − 0.5)/(1 − 0.5) = 0.38/0.5 = 0.76. This balanced reference corresponds to two equally likely categories and no reviewer marginal difference.
The calculation is easy; interpretation is the real work. PABAK is a re-expression of observed agreement under a uniform binary reference. It does not estimate how common the real condition should be, and it does not show that reviewers were unbiased.
Byrt, Bishop and Carlin define a prevalence index from the diagonal imbalance. In our orientation its magnitude is |a − d|/N = |82 − 6|/100 = 0.76.
This large value tells us that positive-positive agreements greatly outnumber negative-negative agreements. The word “prevalence” here belongs to the agreement-table decomposition. If neither reviewer is a gold standard, it should not be described automatically as the true population prevalence of a condition.
The index is useful because two studies can have the same observed agreement but distribute it differently across the two agreement cells. A balanced diagonal produces a small prevalence index; a diagonal dominated by one category produces a large one.
The bias index measures the difference between the off-diagonal cells or, equivalently, the difference in reviewers’ positive proportions. Its magnitude is |b − c|/N = |8 − 4|/100 = 0.04.
Reviewer A calls four percentage points more items positive than reviewer B. That is a descriptive directional difference. It does not by itself identify which reviewer is correct.
A small bias index can coexist with poor agreement if disagreements are balanced in direction. A large bias index can coexist with high agreement when most items match. This is why prevalence, bias and agreement should be reported separately rather than compressed into one moral judgement about reviewers.
For the binary table, the relationship can be written as kappa = (PABAK − PI² + BI²)/(1 − PI² + BI²), using signed or squared prevalence and bias indices consistently.
Substitute PABAK = 0.76, PI = 0.76 and BI = 0.04. The numerator is 0.76 − 0.5776 + 0.0016 = 0.184. The denominator is 1 − 0.5776 + 0.0016 = 0.424. The ratio is 0.184/0.424 = 23/53.
This exactly reproduces ordinary kappa. The identity shows why the two coefficients differ: their reference structures differ. It does not say that PABAK corrects a mistake in kappa. It provides a transparent sensitivity analysis around prevalence and bias.
Construct a second 100-item table with a = 44, b = 6, c = 6 and d = 44. Observed agreement remains 88%. The reviewers each call 50 items positive and 50 negative.
Now expected agreement under separate margins is 0.5. Ordinary kappa is (0.88 − 0.5)/0.5 = 0.76, exactly equal to PABAK. Both prevalence and bias indices are zero.
The comparison is a powerful lesson. The same 88% observed agreement can yield different ordinary kappas because the margins differ. PABAK remains 0.76 because it depends only on observed agreement. Neither fact should be hidden. Readers need both the level and distribution of agreement.
| Table | a | b | c | d | Po | Cohen kappa | PABAK |
|---|---|---|---|---|---|---|---|
| Concentrated | 82 | 8 | 4 | 6 | 0.88 | 23/53 | 0.76 |
| Balanced | 44 | 6 | 6 | 44 | 0.88 | 0.76 | 0.76 |
If a study deliberately samples mostly positive cases, the table reflects that design. Replacing the reference with 0.5 does not make the sample representative of a naturally occurring population.
Likewise, if the target environment truly contains rare positives, a balanced reference may answer a hypothetical question rather than the operational one. That can still be useful, especially for comparing classification consistency across samples, but the hypothetical nature must be stated.
Mathematics cannot rescue an unclear sampling frame. Record how items entered the study, whether category frequencies were fixed by design, and which population the conclusion concerns.
When one method is a justified reference standard, sensitivity, specificity and predictive values may answer more direct questions than symmetric agreement. Kappa and PABAK treat the two reviewers symmetrically.
Two reviewers can agree on 88% of items and both miss the same positive cases. Conversely, a new method can disagree with an imperfect reference because it detects something real. Agreement alone cannot decide.
In education, two markers can consistently apply an unclear rubric. In AI, two annotators can share the same assumption. Report agreement as agreement, then evaluate validity or accuracy with suitable evidence.
Ordinary kappa uses the observed separate margins. PABAK uses a uniform 0.5 expected agreement. Brennan–Prediger and other free-marginal coefficients generalise a similar uniform idea to more categories.
These models answer different counterfactual questions. Asking “which is correct?” without naming the design is incomplete. If category totals were constrained, fixed-marginal reasoning may be relevant. If reviewers were free and the analyst wants a uniform benchmark, PABAK may be informative.
A strong report explains why a reference was selected, then shows sensitivity to another plausible model rather than choosing whichever coefficient supports a preferred story.
Did you know that binary PABAK is a linear transformation of observed agreement? Every increase of one percentage point in agreement increases PABAK by two percentage points.
At 50% agreement PABAK is zero. At 100% it is one. Below 50% it is negative. This simple mapping makes the coefficient easy to audit but also means it adds no new sample information beyond Po.
Its value lies in expressing agreement relative to a balanced two-category reference and in making comparison with other chance-corrected measures easier. The table and indices supply the additional descriptive information.
CHAPTER 13 OF 24 · Check assumptions
13. Direction matters even when the coefficient is symmetric
Swapping reviewer names exchanges b and c but leaves their absolute difference, observed agreement, kappa and PABAK unchanged. Operationally, however, eight disagreements in one direction and four in the other may matter.
For example, one marking process might systematically assign the higher category. The symmetric coefficient cannot describe that direction. Show b and c, the signed marginal difference and any relevant paired test with its assumptions.
Do not call a non-significant directional test proof of no bias. It may reflect limited power. Descriptive counts remain important.
The 100 synthetic items make arithmetic tidy, but a real estimate varies across samples. Confidence intervals should treat each paired item as the unit. If several items come from the same student, document or patient, clustering may require a cluster-aware bootstrap or model.
An interval for PABAK can be obtained from uncertainty in observed agreement, then transformed. An interval for kappa must account for its margin-dependent structure.
Precision is not validity. A narrow interval around a coefficient from an unrepresentative or poorly defined study gives a precise answer to the wrong question.
Do not split 100 paired decisions into 200 independent labels. Agreement exists within pairs. Breaking the pairing destroys the information needed for a, b, c and d.
This principle transfers to before-and-after studies, matched designs and repeated measurements. Mathematics is important because structure determines the denominator and the probability model.
Keep one row per item with both ratings, a stable item identifier and clear missing-value codes. The contingency table can then be regenerated and audited.
An indeterminate classification may be a meaningful third outcome or a missing value. Decide which before analysis. Dropping all indeterminate cases can inflate agreement if hard cases disappear.
If reviewers can abstain, report abstention rates and patterns. If a binary PABAK is still calculated on complete cases, name that subset and explain why exclusion is defensible.
Never recode “unknown” as negative solely to preserve a 2 × 2 table. That changes the question and may create artificial prevalence.
A useful report contains:
- the labelled 2 × 2 table and total;
- observed agreement with category-specific agreement where relevant;
- both reviewer margins;
- ordinary Cohen kappa and its interval;
- PABAK, prevalence index and bias index;
- sampling and missing-data rules; and
- the reason a balanced reference is useful.
This set prevents a single number from carrying more meaning than it can support. It also lets another analyst reproduce every result from four counts.
Begin with four whole-number cells and calculate all totals by hand. Colour the diagonal, then calculate Po. Use row and column proportions to calculate Pe and ordinary kappa.
Next compute PABAK, PI and BI. Verify the identity with exact decimals or fractions. Finally, rearrange counts while holding Po fixed. Observe which summaries change.
This activity joins percentages, algebra, probability and interpretation. It shows why maths in everyday decisions is often about choosing a fair comparison, not merely evaluating a formula.
- PABAK does not estimate true prevalence.
- It does not prove reviewers are unbiased.
- A larger PABAK is not automatically a better result.
- High agreement does not establish accuracy.
- The prevalence index is not always population prevalence.
- A balanced reference is an assumption, not a discovered fact.
Another mistake is to report only PABAK because ordinary kappa appears inconvenient. Sensitivity is informative when both are visible. Selective reporting hides the very dependence the decomposition is designed to explain.
Using a = 82, b = 8, c = 4 and d = 6:
- confirm N = 100 and Po = 0.88;
- reproduce both reviewer margins;
- calculate Pe = 0.788;
- show kappa = 23/53;
- calculate PABAK = 0.76;
- find PI = 0.76 and BI = 0.04; and
- verify the Byrt identity.
Then use 44, 6, 6 and 44. Explain why Po and PABAK stay fixed while ordinary kappa rises to 0.76. State which change in the margins caused the difference.
When a student sees two statistics disagree, encourage curiosity rather than panic. Ask what each denominator and reference represents. Often the apparent contradiction disappears once the comparison model is named.
Parents can support numeracy by discussing everyday classification: spam or not spam, recyclable or not recyclable, ready or needs revision. Who decides, what counts as agreement and what happens when one category is common?
For school projects, preserve raw counts and write one sentence distinguishing evidence from interpretation. That habit supports future work in statistics, computing, science and AI.
Is PABAK only for binary tables?
The familiar formula and Byrt prevalence-bias decomposition here are binary. Free-marginal ideas have multicategory extensions, but do not apply the binary indices unchanged.
Why can kappa be low with high agreement?
The margin-based expected agreement can also be high. Kappa measures improvement over that reference, not observed agreement alone.
Should PABAK replace Cohen kappa?
Usually it is better used beside the table and ordinary kappa as a sensitivity analysis. The design determines which reference is meaningful.
Can the value be negative?
Yes. PABAK is negative when observed agreement is below 50% in a binary table.
The primary paper is Byrt, Bishop and Carlin, Bias, prevalence and kappa, DOI 10.1016/0895-4356(93)90018-V. The epiR kappa documentation documents the binary PABAK, prevalence and bias outputs.
Continue with Cohen’s kappa, Gwet’s AC1 and the Mathematics Learning Hub. Read the destination mechanisms as alternatives with different assumptions, not as a leaderboard.
Before using PABAK, confirm that the table is genuinely binary, paired and labelled; show all four cells; preserve reviewer direction; report Po and ordinary kappa; explain the balanced reference; distinguish table prevalence from population truth; handle missingness and clustering; and avoid claims about accuracy or fairness that the agreement data cannot establish.
The importance of mathematics is that four simple counts can support several legitimate summaries. Clear definitions let readers see exactly where those summaries agree and where the assumptions diverge.
Extended practice laboratory
Practice case 1: Rare positive sample
Setup: Use 95 negative agreements, two positive agreements and three disagreements. Task: Calculate Po and ask whether PI describes population prevalence.
Check: The table index describes diagonal imbalance; sampling determines whether it represents a population. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 2: Balanced disagreements
Setup: Set b and c equal while keeping their sum fixed. Task: Predict the bias index.
Check: BI becomes zero, but agreement and prevalence imbalance may remain. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 3: Directional shift
Setup: Move one case from c to b. Task: Track reviewer margins and signed bias.
Check: Po is unchanged while the direction of marginal difference moves by two cases. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 4: Diagonal transfer
Setup: Move five cases from a to d. Task: Track Po, PABAK and PI.
Check: Po and PABAK stay fixed; diagonal imbalance and PI fall. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 5: Off-diagonal transfer
Setup: Move two cases from a to b. Task: Recalculate every summary.
Check: Agreement falls, one reviewer margin changes and both coefficients must be recomputed. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 6: Equal margins
Setup: Construct any table with b equal to c. Task: State what is and is not proven.
Check: Equal margins remove the descriptive bias index but do not prove correct classification. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 7: Below chance benchmark
Setup: Choose Po equal to 0.40. Task: Calculate PABAK.
Check: Two times 0.40 minus one gives −0.20 under the balanced binary reference. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 8: Perfect agreement
Setup: Set b and c to zero. Task: Compare ordinary kappa and PABAK.
Check: PABAK is one; ordinary kappa is one when both categories permit a defined denominator. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 9: Single observed category
Setup: Put every case in a. Task: Check denominator behaviour.
Check: PABAK is one, while ordinary kappa can be undefined because its expected agreement is one. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 10: Case-control sampling
Setup: Select equal known positives and negatives by design. Task: Interpret PI carefully.
Check: The sample balance is imposed and should not be called natural population prevalence. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 11: No gold standard
Setup: Treat both reviewers symmetrically. Task: Choose appropriate language.
Check: Say agreement and marginal difference; avoid sensitivity or specificity claims. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 12: Reference available
Setup: Let reviewer B be a justified standard. Task: Name additional summaries.
Check: Sensitivity, specificity and predictive values may answer accuracy questions more directly. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 13: Clustered scripts
Setup: Take five answers from each student. Task: Choose a resampling unit.
Check: Resample students, not isolated answers, when within-student dependence matters. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 14: Indeterminate labels
Setup: Allow an unknown response. Task: Decide whether binary PABAK applies.
Check: Unknown must be justified as missing or a third category before analysis. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 15: Software audit
Setup: Enter the four cells in a package. Task: Verify orientation and output.
Check: Match software Po, margins and indices to hand arithmetic before trusting the coefficient. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 16: Policy threshold
Setup: A positive label triggers support. Task: Ask what agreement cannot decide.
Check: The table does not prove the threshold is fair or the support effective. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 17: Two samples
Setup: Hold Po at 0.88 with different margins. Task: Explain the comparison.
Check: PABAK stays 0.76 while ordinary kappa can change because Pe changes. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.
Practice case 18: Reporting choice
Setup: A large PABAK looks attractive. Task: Write the ethical report.
Check: Show both coefficients, the table and the reason for the balanced sensitivity analysis. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose.