中文版 · Blockchain learning guide · Previous article
Blockchain learning · Article BC091
A project can report a real improvement without proving that blockchain caused it. The measured outcome may have changed alongside the case mix, forms, staffing or training. To evaluate the claim, separate the observed numbers from the proposed explanation and ask what the available comparison actually rules out.
This lesson audits a fictional shared-record project using a complete set of aggregate figures. We calculate its reported improvement, adjust one visible case-mix difference, inspect a non-blockchain comparator and write a conclusion that preserves what is useful without inventing causation. Every quantity is synthetic; no real organisation’s performance is being judged.
The central result is deliberately qualified: the recorded reconciliation mean improved, but this evidence does not isolate a blockchain-specific effect. That is an informative conclusion, not a failure to do the arithmetic. Sources were checked on 12 October 2026.
Choose your reading route
In this guide
- 1. Freeze the claim before inspecting the chart
- 2. Understand the three cohorts
- 3. Record the changes that accompanied the technology
- 4. Reproduce the headline number
- 5. Examine the visible case-mix change
- 6. Inspect the non-blockchain comparator
- 7. Ask the counterfactual question explicitly
- 8. Test the proposed mechanism
- 9. Separate four levels of conclusion
- 10. Exercise: approve, reject or rewrite
- 11. Write the revised project statement
- Frequently asked questions
- Sources and further reading
Read sections 1–3 for the claim and dataset. Sections 4–6 contain the calculations and comparison. Sections 7–9 explain why authenticated records, plausible mechanisms and a favourable average still leave attribution questions. The exercise asks you to approve or rewrite specific claims, followed by a model assessment note.
Back to contents · Next section
1. Freeze the claim before inspecting the chart
The fictional consortium’s presentation says: “Blockchain cut shipment delivery time by 50%.” It also says the project demonstrates that a blockchain system is twice as efficient as a conventional shared database.
Those are two large claims. The first names an outcome, a magnitude and a cause. The second adds a technology comparison and a broad efficiency concept. Before accepting either, write down exactly what the project measured.
Its actual measure is elapsed minutes from acceptance of a complete reconciliation request to agreement of the participating offices on the corresponding shared record. It does not measure the time from dispatch to physical delivery. It does not include the effort spent preparing an incomplete request before acceptance. It does not measure computing energy, full operating cost, staff-hours or every aspect of efficiency.
The supplied start and end definitions are identical for all three cohorts below. This allows a limited comparison of that endpoint. It does not permit replacing its name with shipment delivery or general productivity. A clean chart cannot repair a changed definition.
Back to contents · Next section
2. Understand the three cohorts
Each cohort contains 30 completed reconciliation cases. All admitted cases in these fictional cohorts completed and appear in the totals; there are no omitted unresolved cases or missing endpoint times within the supplied dataset. We give counts and total durations, not individual case times.
Cohort A is the earlier process. It used separate spreadsheets and email reconciliation. It contains 20 standard cases with a total of 120 minutes, and 10 exception cases with a total of 180 minutes. Thus standard cases averaged 6 minutes and exception cases averaged 18 minutes.
Cohort B is the later blockchain-supported process. It contains 27 standard cases totalling 108 minutes, and 3 exception cases totalling 42 minutes. The group means are 4 and 14 minutes respectively.
Cohort C is a separate contemporary comparator using a conventional shared database. It also contains 27 standard cases totalling 108 minutes and 3 exception cases totalling 42 minutes. Its group means therefore match B’s.
The category definitions are supplied as consistent across cohorts. However, the cases were not randomly assigned. Different teams handled B and C, and unmeasured differences in case difficulty within each category are possible. A and B also occurred in different periods. The two-category label is not a guarantee that all relevant circumstances match.
Back to contents · Next section
3. Record the changes that accompanied the technology
B introduced a blockchain-supported record system, but it also introduced a common submission form, mandatory field checks and new staff training. C used the same new form and field checks, and its team received the same training programme. The earlier A process had none of those changes.
The dossier also says B’s record history was accepted under the project’s supplied validation rules. That is evidence about recorded events in that system. It does not identify how many minutes were saved by the form, by training, by the ledger architecture or by the different workload.
For the arithmetic, assume the aggregate counts and times are accurate and were collected under the stated endpoint definition. We are not using a hidden data error to undermine the presentation. Even when the numbers are correct, the explanation may exceed them.
An evaluator should keep a change list beside the performance table. If several components changed together, an observed difference attaches initially to that bundle and context. Attributing a particular share to one component requires further reasoning and evidence.
Back to contents · Next section
4. Reproduce the headline number
For A, total duration is 120 + 180 = 300 minutes across 30 cases. Its arithmetic mean is 300 / 30 = 10 minutes per completed reconciliation case.
For B, total duration is 108 + 42 = 150 minutes across 30 cases. Its arithmetic mean is 150 / 30 = 5 minutes per case.
The observed relative reduction, using A’s mean as denominator, is (10 − 5) / 10 = 0.5, or 50%. The absolute difference is 5 minutes per case. Those calculations support a descriptive statement about the supplied cohorts and endpoint.
They do not show that every case became 50% faster. The standard-group means changed from 6 to 4 minutes, while the exception-group means changed from 18 to 14. Nor can we calculate the median or a tail percentile from these grouped totals alone. Individual distributions have not been supplied.
The phrase “twice as efficient” is particularly imprecise. A mean duration ratio of two does not establish a twofold gain in every resource or outcome. The record comparison should keep the actual measured variable and denominator in its wording.
Back to contents · Next section
5. Examine the visible case-mix change
A contains 10 exception cases out of 30, or one-third. B contains 3 out of 30, or one-tenth. Since the exception-group means are higher in both cohorts, this different composition affects the aggregate comparison.
We can ask a limited descriptive question: what weighted mean results if B’s observed group means are applied to A’s supplied category counts? This is standardisation to the old 20-standard, 10-exception mix, not a claim that those individual old cases would actually have taken the new times.
The calculation is (20 × 4 + 10 × 14) / 30 = 220 / 30 = 22/3 minutes, approximately 7.333 minutes.
Compared with A’s mean of 10, the relative reduction is (10 − 22/3) / 10 = 4/15, approximately 26.667%. This is smaller than 50% because we have held this one observed composition feature constant in the arithmetic.
Do not rename 26.667% as “the true blockchain effect.” We have adjusted only the two supplied categories. We have not controlled training, forms, team selection, calendar conditions or unmeasured difficulty within categories. The calculation exposes a sensitivity of the headline; it does not identify a causal effect.
A different reference mix would produce a different standardised descriptive mean. That is why the report must name the chosen weights rather than present the adjusted number as a context-free truth.
Back to contents · Next section
6. Inspect the non-blockchain comparator
C has the same category counts and totals as B. Its mean is also 150 / 30 = 5 minutes. Therefore the observed B-minus-C difference in the aggregate mean is zero minutes for these supplied cohorts.
This is useful evidence against the claim that the presentation has demonstrated a unique 50% blockchain advantage. A non-blockchain implementation accompanied by the new process also achieved the same recorded mean in this dossier.
But zero observed difference is not proof that the architectures have identical causal effects in all circumstances. The teams differ, allocation was not random, and two group labels may conceal relevant differences. We also have no individual variation measures or predeclared equivalence margin. It would be wrong to turn an overconfident pro-blockchain claim into an equally overconfident universal anti-blockchain claim.
The comparator narrows what can honestly be said. The packet does not establish an additional measured mean advantage of B over C in these data. It also does not establish causal equivalence, equal security properties, identical governance costs or equal long-run reliability.
This distinction matters because a reader can reject an unsupported sales claim without claiming to have settled the entire architecture decision.
Back to contents · Next section
7. Ask the counterfactual question explicitly
The attribution question is: what would have happened to a suitably comparable workflow if the relevant blockchain component had not been introduced, while the other relevant conditions were handled appropriately in the evaluation?
The UK government’s Magenta Book, updated 15 May 2026, distinguishes monitoring outcomes from assessing attribution. Its discussion of experimental and quasi-experimental comparisons emphasises a suitable counterfactual and method-specific assumptions. It also describes other evaluation approaches; this lesson does not claim that randomisation is the only possible source of causal understanding. Magenta Book evaluation guidance
NIST’s description of completely randomised designs explains assigning factor levels to experimental units randomly. That is one design concept, not a retrospective label we can attach to B and C after seeing their results. Our cohorts were explicitly not assigned that way. NIST completely randomised designs
For this dossier, no analysis can simply assume away the missing allocation information. A careful review can identify possible explanations and the evidence that would distinguish them. It cannot create a controlled comparison by giving an observational spreadsheet a more impressive title.
Back to contents · Next section
8. Test the proposed mechanism
The presentation proposes that shared record ordering reduced time spent resolving contradictory versions. That is a plausible mechanism to investigate. It should be connected to evidence of the work actually avoided, rather than inferred solely from the presence of a chain.
For example, the dossier could contain traceable categories of reconciliation effort before and after the change: time spent finding the latest authorised record, correcting missing fields and contacting another office. Here those detailed effort observations are not supplied. We therefore cannot divide the five-minute aggregate reduction among those mechanisms.
The common form could reduce missing-field exchanges in both B and C. Training could improve handling speed. A changed category mix could reduce the aggregate mean even without identical changes in every type of case. These are alternative explanations compatible with parts of the dataset, not proven accounts of what happened.
NISTIR 8202’s oracle discussion distinguishes recording information from establishing the truth of external events. The same boundary cautions us against treating accepted ledger entries as proof of a physical delivery outcome that was never measured. NISTIR 8202, section 7.3
A stronger mechanism story needs evidence on the specific causal links. Merely adding signatures to the existing aggregate table can improve assurance about who recorded it without resolving those links.
Back to contents · Next section
9. Separate four levels of conclusion
The first level is observation: A’s supplied mean is 10 minutes and B’s is 5. This is established under the dataset assumptions.
The second is descriptive adjustment: applying B’s group means to A’s mix gives 22/3 minutes. This is also established, with its chosen weights clearly named.
The third is comparison: C’s observed mean equals B’s, while the allocation and team limitations remain. This is established as a descriptive equality, not universal equivalence.
The fourth is attribution: blockchain alone caused a specified reduction in reconciliation or delivery. The packet does not establish that. Moving between levels requires more than changing a chart label.
These conclusions can coexist. Saying the causal claim is inconclusive does not erase the observed improvement. Saying the observed improvement is real does not validate the proposed cause. A useful review preserves both sentences rather than demanding one simple verdict for every claim.
Back to contents · Next section
10. Exercise: approve, reject or rewrite
Evaluate these six statements using only the dossier:
- B’s observed arithmetic mean for the defined reconciliation endpoint is 50% lower than A’s.
- Physical shipment delivery time fell by 50% because blockchain was introduced.
- Standardising B’s group means to A’s mix gives approximately 7.333 minutes.
- The standardised reduction of approximately 26.667% is the isolated blockchain effect.
- B and C have the same observed mean in this packet, so every future deployment will perform identically.
- The packet supports a descriptive improvement but leaves the blockchain-specific causal contribution unresolved.
Answers and reasoning
- Approve with the endpoint, cohort and arithmetic-mean wording retained. The calculation is 300/30 versus 150/30.
- Reject. The measured endpoint is record reconciliation, and the causal attribution is not established.
- Approve as a weighted descriptive calculation using the stated reference mix, not a counterfactual outcome estimate for the old individual cases.
- Reject. Category standardisation does not remove the other changes and allocation limitations.
- Rewrite. The first clause is supported; the future universal conclusion is not. Nor has causal equivalence been demonstrated.
- Approve. It preserves both the measured result and the evidential limit.
Extension: if someone calls the five-minute B mean a median, can you confirm it? No. Counts and group totals do not provide the individual ordering needed to identify the median. A median of five may or may not be compatible with particular underlying values; it has not been established by the supplied summary.
Back to contents · Next section
11. Write the revised project statement
A defensible statement is: “In the supplied completed-case cohorts, mean elapsed reconciliation time decreased from 10 to 5 minutes. The case mix changed, and the project also introduced new forms, checks and training. Using the earlier category mix with the later group means produces a descriptive mean of approximately 7.333 minutes. A separate non-blockchain comparator recorded the same five-minute mean, but non-random allocation and team differences limit causal interpretation. The packet does not isolate a blockchain-specific contribution or measure physical delivery time.”
That statement still reports progress. It also gives the reader enough information to understand why a stronger slogan is unsupported. It avoids claiming that the database alternative is globally equivalent or that the blockchain architecture has no other possible value.
If the team wants to justify a particular architecture, it should identify which decision-relevant property remains to be tested: independent verification, control distribution, recovery, reconciliation effort or another defined outcome. Those properties need their own evidence. The current timing table cannot answer every question just because it appears in a blockchain presentation.
Back to contents · Next section
Frequently asked questions
Does before-and-after evidence have no value?
It can describe change, reveal operational problems and guide further investigation. Its value does not require pretending it isolates every cause.
Does matching two case categories solve all confounding?
No. It addresses the specified categories in the calculation. Other differences may remain, including differences within a category.
Does a zero B–C mean difference prove blockchain has no benefit?
No. It reports these data for this endpoint. It does not establish universal performance, causal equivalence or the absence of other properties.
Would storing the evaluation data onchain prove the claim?
It could support particular record-integrity claims under specified assumptions. It would not replace a valid outcome definition, comparison design or causal argument.
Back to contents · Next section
Sources and further reading
Primary methodological and technical references checked on 12 October 2026:
- Magenta Book evaluation guidance: updated 15 May 2026; monitoring, attribution and comparison assumptions.
- NIST completely randomised designs: the meaning of random assignment in that design family.
- NISTIR 8202, Blockchain Technology Overview: October 2018, section 7.3 on external input and the oracle boundary.
For general foundations, read How Assurance Cases Work and Logistics Benchmarking: Comparing Performance Without Comparing Different Jobs. The present lesson adds a specific blockchain claim audit with reproducible arithmetic and an explicitly inconclusive attribution result.