The Gold Standard of Archaeology | Laboratory Proficiency Trends, Round-to-Round Comparability and Corrective-Action Closure

1. Define the proficiency question across rounds

Back to contents

A trend must answer a stated question. Is the laboratory improving for one analyte and matrix, maintaining performance across routine work or controlling a specific preparation stage? Combining every proficiency result into one percentage can hide that the tasks are not comparable.

Define the measurand, material class, concentration or age range, workflow stage and performance statistic. In archaeology, a graphite target, raw collagen, sediment digest and ceramic paste do not challenge the same process. A round that bypasses pretreatment cannot demonstrate improvement in pretreatment.

State whether the goal is surveillance, accreditation support, method validation or corrective-action verification. The trend frame determines which results may be compared and which belong in separate series.

First-principles check: Build a longitudinal series only from results that answer the same performance question. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Classify twelve proficiency results into comparable task families. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Add a radiocarbon round using raw bone after three rounds using prepared graphite. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep task definition, matrix, measurand, range, workflow boundary and intended decision. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

2. Create a round-comparability record

Back to contents

Every round needs a compact comparability record: material, homogeneity and stability evidence, assigned-value route, uncertainty, target standard deviation, scoring rule, participant instructions and reporting convention. Changes in any of these can move a score without a change in laboratory competence.

Use a comparability matrix rather than assuming the same provider means the same task. Mark rounds as directly comparable, comparable after a documented transformation or descriptive only. If scoring thresholds change, retain the original category and, where justified, recalculate a harmonised metric from preserved values.

Do not back-calculate data that were never retained. A gap is safer than a reconstructed precision that depends on unknown uncertainty or units.

First-principles check: Separate change in laboratory performance from change in the proficiency exercise. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Compare four rounds whose matrices and scoring denominators differ. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: The provider changes from z-scores to zeta scores. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain protocols, assigned values, uncertainties, denominators, thresholds, transformations and comparability class. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

3. Keep raw difference, uncertainty and score together

Back to contents

A proficiency category compresses several quantities. Preserve the reported result, assigned value, difference in original units, relative difference, participant uncertainty, assigned-value uncertainty and the denominator used for scoring. This allows later reviewers to distinguish a large bias from a small denominator.

Track sign as well as magnitude. Repeated positive differences can indicate persistent bias even when each result remains satisfactory. Conversely, alternating signs near zero may reflect ordinary variation rather than unstable correction.

Where limits of detection, censored values or asymmetric uncertainties matter, a simple z-score trend may be inappropriate. Use a method suited to the reporting model and keep the uncompressed evidence visible.

First-principles check: Trend the measurement evidence, not only the traffic-light category. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Plot signed bias, absolute bias and performance score for six rounds. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Double the proficiency target standard deviation while raw bias stays constant. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep all inputs, units, uncertainties, calculation versions and category rules. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

4. Distinguish trend from a short run of chance

Back to contents

Three better-looking rounds are encouraging, but they do not by themselves prove a stable improvement. Examine within-round uncertainty, task comparability and the number of opportunities for extreme results. Prespecify simple signals such as sustained directional bias, repeated questionable scores or a run after an intervention.

Control charts or regression can support review when their assumptions fit, but avoid treating a handful of heterogeneous points as a precise time series. Annotate method, staff, instrument, calibration, software and preparation changes on the plot. Patterns become interpretable when operational events are visible.

Use the trend to trigger investigation, not to diagnose a cause automatically. A downward score can reflect easier material; a worse score can reflect a more demanding but more relevant matrix.

First-principles check: Require repeated, comparable evidence before calling a change persistent. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Evaluate an apparent improvement across five rounds with two method changes. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Remove the easiest round and reassess the trend. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain plots, annotations, signal rules, assumptions, alternative views and reviewer conclusions. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

5. Investigate poor performance from the measurement chain

Back to contents

An unsatisfactory result begins an investigation, not a declaration of incompetence. Verify transcription, units, conversion, result version and provider calculation before moving into sampling, preparation, blank, calibration, instrument, software, uncertainty and reporting causes.

Use internal quality-control data and retained materials where available. Ask whether the issue could affect customer results before and after the round. A clerical error may still reveal a reporting-control weakness; a correct recalculation does not make the system issue irrelevant.

Avoid choosing the first plausible cause. State competing hypotheses and the evidence that discriminates among them. Corrective action should address a demonstrated or defensibly bounded cause, not merely repeat testing until a satisfactory number appears.

First-principles check: Trace the result through every stage that could create the observed sign and size. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Build a cause tree for a high trace-element result in sediment. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: The same raw instrument data produce a correct result after a unit correction. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep raw data, calculations, quality controls, interviews, hypotheses, impact review and cause decision. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

6. Worked example: apparent improvement after an easier round

Back to contents

A laboratory reports z-scores of 2.7, 2.1, 0.4 and 0.3 across four annual rounds and claims successful improvement. The first two materials were low-concentration biological residues requiring difficult digestion; the next two were high-concentration homogeneous soils. Raw relative bias fell, but the assessment denominator also widened.

The comparability review separates matrix families. Within soil, the results are satisfactory but do not demonstrate that the digestion problem was fixed. The laboratory’s corrective action had changed acid preparation and staff training, yet no comparable biological material had been tested after the change.

Closure is therefore provisional. Internal controls on a retained biological reference support improvement, and the next external biological round is designated as effectiveness evidence. The positive soil scores remain useful, but the claim is narrowed to what they tested.

First-principles check: Do not attribute a score change to correction before task difficulty and denominator are aligned. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Rebuild the four-round table using raw bias, matrix and target spread. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Make the fourth round biological but at a much higher concentration. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Preserve round designs, corrective action, internal check, provisional status and planned external verification. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

7. Design corrective actions that reach the cause

Back to contents

Correction fixes the immediate result; corrective action changes the process that produced the problem. Recalculating a unit corrects a report. Updating software validation, independent review and training may prevent recurrence. Instrument recalibration helps only if calibration was the cause.

Assign an owner, due date, affected scope, implementation evidence and effectiveness test. Consider whether earlier archaeological results require review, notification or qualification. A proficiency result can reveal a systemic issue with consequences beyond the exercise.

Prefer controls that are observable and sustainable. A reminder to be careful is weaker than a locked unit field, automated plausibility check or second-person review tied to a documented criterion.

First-principles check: Connect each action to a causal mechanism and a future detection opportunity. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Convert five vague actions into specific process changes and tests. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Use a staff change as the suspected cause but find the same bias across two analysts. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep cause, action, owner, scope, implementation proof, retrospective impact and test plan. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

8. Verify effectiveness before closing the action

Back to contents

Completion is not effectiveness. A revised procedure may exist without being followed, and training attendance may not change measurement performance. Define the closure evidence when the action is opened: repeated internal control, blind remeasurement, retained proficiency item or a later comparable external round.

Set an observation window long enough to expose recurrence. If the relevant task is rare, closure may remain provisional until sufficient evidence arrives. Record what would reopen the action, including a renewed directional bias or failure in an adjacent matrix.

Independent review should test both implementation and outcome. The reviewer asks whether the cause was plausible, the action addresses it and the effectiveness evidence measures the same failure mode.

First-principles check: Close only when the corrected process performs under a comparable challenge. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write closure criteria for a digestion bias that appears only in low-level bone samples. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: No external round is available for eighteen months. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Retain criteria, evidence dates, results, reviewer, residual risk, provisional status and reopen trigger. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

9. Use trends without ranking laboratories unfairly

Back to contents

Round-to-round data are often attractive for dashboards and procurement, but scores depend on task mix, participation frequency and reporting. A laboratory choosing difficult relevant schemes may look worse than one selecting easy or narrow exercises. One pooled rank can penalise transparency.

Interpret trends within validated scope and comparable families. Show missing rounds and changes in participation. Do not replace accreditation, method validation or case-specific quality evidence with a league table.

When users need a summary, report what was tested, whether performance was satisfactory, recurring signals, open corrective actions and scope limits. The goal is informed selection and improvement, not institutional humiliation.

First-principles check: Compare performance only within a declared task and evidence frame. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Redesign a laboratory league table as a scope-aware evidence summary. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: One laboratory reports every failure while another withholds a round. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep participation history, scheme selection, comparability, open actions, caveats and publication rules. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

10. Audit the longitudinal proficiency system

Back to contents

The audit joins scheme selection, sample receipt, custody, routine method use, raw result, provider evaluation, internal review, trend update, investigation, action and closure. Each stage needs versioned records and clear responsibility.

Reproduce selected scores, trace category changes and verify that trend plots include all eligible rounds. Check whether method or scoring changes were annotated and whether open actions survived staff turnover. Sample closed cases to confirm that effectiveness evidence existed before closure.

Review the programme itself. Repeated participation can become ceremonial if rounds do not cover difficult matrices or if learning never changes practice. Adjust scheme choice while preserving continuity for important trend series.

First-principles check: Make the proficiency programme auditable from distributed material to closed action. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Audit one five-year analyte-matrix series and its linked actions. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: The provider revises an assigned value after publication. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Archive scheme files, submissions, recalculations, trend versions, actions, closures and programme review. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

11. Frequently asked questions

Back to contents

Can z-scores from different schemes be plotted together? Only with a clear comparability argument; identical scale does not guarantee identical difficulty. Does one satisfactory round close an action? It may, if it tests the same failure mode and the closure rule said so, but repeated evidence can be necessary. Should old results be rescored? Only transparently with preserved inputs and reason.

What if performance worsens after a method becomes more sensitive? Separate changed task from changed competence. Can a laboratory skip a difficult round? It can, but the missing evidence and scope consequence should remain visible. Does proficiency testing prove every archaeological result? No; it samples defined processes under particular materials and times.

First-principles check: Answer with the comparability, scope and closure rule. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Prepare a decision guide for interpreting a new questionable score. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Use a qualitative identification round instead of quantitative measurement. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Maintain reviewed questions, examples, source guidance, owners and update date. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.

12. The gold standard turns scores into controlled learning

Back to contents

A credible proficiency trend begins with comparable tasks and retains the raw measurement evidence beneath each score. It annotates method and scoring changes, distinguishes persistent signal from chance and investigates causes across the complete measurement chain.

Corrective actions are specific, owned and connected to affected archaeological results. They close only after a comparable challenge demonstrates effectiveness; otherwise the status remains open or provisional.

The final measure is not an ever-greener dashboard. It is a laboratory that can show what changed, why it changed, whether the change worked and where the evidence still does not travel. That record supports both scientific trust and practical improvement.

First-principles check: Require longitudinal comparability and verified effectiveness before claiming improvement. Write the answer before looking at the preferred outcome. Then identify the authority, observation or calculation that could change it. This keeps the method tied to evidence rather than hindsight.

Worked practice: Write the final five-year proficiency narrative for one method and matrix. Keep the intermediate steps visible. A reader should be able to locate the input, apply the same rule and understand why a different result would follow if one fact changed.

Changed-context transfer: Begin a new series after a major instrument replacement. Preserve the underlying principle but replace the setting-specific details. State which parts travel, which do not, and which new authority or measurement would be needed.

Evidence record: Keep the final series, interpretations, causes, actions, closure evidence, scope and future plan. Retain uncertainty and disagreement instead of compressing them into a clean label. The record should let another person reconstruct the decision without relying on the memory or status of the original decision-maker.

Decision-quality drill: Run the practice once under the most favourable plausible assumptions and again under the least favourable plausible assumptions. Compare which conclusion changes, which merely becomes less precise and which survives. If the result depends on an unstated convention, make that convention visible before acting. This is a transfer test: the method should guide a new case, not only explain the example after the answer is known.

Practical verification: Give the case file to a reviewer who did not build the analysis. Ask that person to reproduce the critical step, identify the stopping rule and name the strongest live alternative explanation. Resolve differences by returning to the governing evidence, not by averaging opinions. Record corrections, version the decision and keep a short list of conditions that would require reopening it.