Review the evidence before approving the work
A practical workplace guide to accepting, holding, rejecting and repairing SI-generated outputs. Choose a route, follow complete fictional review packets, then test yourself on fresh cases.
Choose a chapter
Set the review boundary
- 1. Decide what the review is supposed to permit
- 2. Establish the contract before reading the polished answer
- 3. Match verification depth to the consequences of error
- 4. Give the reviewer competence, independence and permission to stop
Perform the checks
- 5. Turn fluent prose into checkable claims
- 6. Recalculate quantities and test the logic around them
- 7. Understand what sampling can and cannot establish
- 8. Use a repeatable sequence from intake to release
Learn from complete worked packets
- 9. Worked packet A: review a stationery readiness summary
- 10. Worked packet B: review meeting decisions without inventing agreement
- 11. Worked packet C: review a rewritten internal borrowing guide
Repair, record and practise
- 12. Write rework instructions that produce a checkable repair
- 13. Keep an audit trail that another person can actually use
- 14. Measure the review process without rewarding rubber-stamping
- 15. Independent practice: review before reading the answers
- 16. Questions reviewers ask in practice
- 17. Put the procedure to work on one bounded task
A workplace review is a decision about whether a particular piece of work is fit for a particular next use. The reviewer needs to know what the work is meant to achieve, which evidence should support it, what would happen if it were wrong, and who has authority to release it. A fluent paragraph is only the object being reviewed. It is not the evidence that the review has succeeded.
This guide gives a practical procedure for reviewing summaries, internal notices, routine calculations, knowledge-base answers and other bounded workplace outputs. It includes complete fictional source packets, deliberately flawed generated work, reviewer findings, corrected outputs and exercises with full answers. The examples use invented organisations, objects and records. They are teaching materials, not reports of an actual product test or measured workplace improvement.
In this eduKate series, Super Intelligence, or SI, is a practical editorial umbrella for using AI models, tools, information and human judgment together. It does not mean that today’s workplace tools have been established as technical artificial superintelligence. Technical ASI is used here for a hypothetical broadly superhuman intelligence. The procedure below concerns current, reviewable workplace artefacts, not a claimed solution to supervising a hypothetical system beyond all human expertise.
The central habit is simple: read the task, inspect the evidence, test the consequential claims, and make a bounded decision. Do not begin by asking whether the result sounds intelligent. Begin by asking what someone would do differently after reading it, and whether the available record justifies that action.
1. Decide what the review is supposed to permit
Reviewing a document and approving an action are different jobs. A draft can be accurate but still not be authorised for release. A calculation can be correct but use information that should not go to the intended audience. A meeting summary can faithfully describe a proposal while the proposal remains unapproved. Good review keeps these states separate instead of compressing them into a reassuring word such as approved.
Start with a sentence that names the artefact and its permitted use. For example: this review determines whether revision two of the stationery summary can be placed in the internal operations folder for the office coordinator to use when planning the next cupboard check. That sentence makes clear that the output is informational, the audience is internal, and the next action is a check. It does not authorise a purchase, a supplier message or a change to company policy.
The same output may require a different review if its destination changes. An internal brainstorm can tolerate clearly labelled possibilities. A notice directing colleagues to a room at a fixed time needs the room and time checked. A public statement about a service needs the supporting source, scope and release permission established. The words may remain almost identical while the consequences of being wrong change substantially.
Use three questions to establish the review boundary. What exactly can the receiver rely on? What exactly may the receiver do next? What remains outside this review? If the answers are vague, the reviewer has no stable target. A request to check everything is not a substitute for deciding which claims are required for the next use and which matters need a separate specialist process.
Consider a generated paragraph saying that the stationery cupboard is ready for the workshop. The phrase ready may conceal several tests: the requested items exist, their quantities are sufficient, the items are usable, and someone has permission to collect them. A reviewer who checks only the arithmetic has not checked the whole meaning of ready. A better output names the actual state: the recorded count meets the requested quantity for three of four items; the fourth requires a further physical check.
Treat review results as scoped conclusions. Accepted for internal planning is more useful than passed. Accepted after the listed correction is different from accepted subject to someone eventually resolving an important issue. When a missing fact could change the receiver’s action, leave the work on hold. A condition hidden in a footnote does not turn an unresolved dependency into completed review.
This is also why review should not be a vote on whether AI is good or bad. The question is narrower and more productive: does this version satisfy this contract using this evidence? A reviewer can accept one output from a system and reject another without inconsistency. The evidence and intended use, rather than enthusiasm or scepticism about the technology, should determine the result.
A useful review decision has a noun and a boundary
Write the noun first: summary, calculation, notice, checklist, draft or recommendation. Then name the boundary: internal reference, discussion only, ready for the named sender, or not ready for use. This habit prevents a local check from expanding silently into broad assurance about the model, the whole workflow, or every future output.
Where the work concerns health, legal rights, financial commitments, safety-critical operations or decisions affecting someone’s opportunities, this low-risk guide is insufficient as an approval procedure. Route the task through the organisation’s authorised qualified-review process. Do not use a generic checklist to claim professional competence or to automate a consequential decision about a person.
Back to contents · Continue to chapter 2
2. Establish the contract before reading the polished answer
A review contract is a compact description of the job, the evidence, the constraints and the acceptance criteria. It need not be a long form. Its purpose is to stop the generated answer from setting its own examination. If the answer decides which facts matter, which source version counts and which omissions are acceptable, the review may simply verify the story the answer chose to tell.
Read the request and source inventory first. Write down the required deliverable in your own words. Identify the intended reader, the scope of the task, the permitted sources, the date or cutoff, the required fields and the prohibited actions. Then inspect the generated output against that independent description. This ordering creates a practical opportunity to notice missing obligations rather than only visible mistakes.
For a routine internal event notice, the contract might require the event name, date, start and end times, room, booking requirement and contact route. It might prohibit announcing a room that has not been confirmed. It might require unresolved details to be labelled pending. None of these criteria depends on how attractive the draft looks. A beautiful notice without the date still fails a material requirement.
Separate hard requirements from preferences. A hard requirement affects correctness, authorisation or fitness for the intended use. A preference concerns a choice that can vary without changing the decision. For example, the correct room is a hard requirement; whether the opening says please join us or you are invited may be a preference. Keeping these categories distinct prevents cosmetic disagreements from consuming the attention needed for evidence checks.
Define what counts as acceptable evidence for each required claim. A room location may come from the current booking confirmation. A quantity may come from a named inventory export. A statement that an action has been completed needs a completion record, not a promise to do it. The contract should identify the authoritative record by role, version or date, rather than giving every document equal status.
Also define the treatment of missing information. A blank cell can mean not collected, unknown, not applicable or accidentally omitted. It does not automatically mean zero. A missing meeting response does not automatically mean agreement. If the task requires a value but the source does not provide it, the correct result may be an explicit unresolved field. Review should reward that honesty rather than pressuring the system to fill every space.
The contract should include the expected unit of work. Are you reviewing every claim in one notice, every row in a small table, a selected sample from a batch, or a reusable template? A template can be well designed while a particular populated version is wrong. A populated version can be correct even though the template contains a weakness that will appear under different inputs. These require different checks and different conclusions.
An effective acceptance criterion can be tested by someone other than the writer. Accurate and professional is too broad by itself. Better criteria say that every named action must have an owner explicitly present in the supplied notes; every total must reconcile to the included rows; and unresolved dates must remain unresolved. These statements specify what a reviewer must observe and what would cause rejection or rework.
Keep the contract stable without making it rigid
Sometimes review reveals a genuinely missing requirement. Record the addition and the reason, then apply it consistently to the whole artefact. Do not quietly invent a new criterion solely to justify accepting or rejecting the answer you already prefer. Conversely, do not retain an unsuitable criterion merely because it was written first. A visible revision to the contract is better than an invisible change in the standard.
Before proceeding, check that the contract is achievable with the supplied evidence. If the task demands confirmation of delivery but only an order request is available, the source packet cannot support the desired conclusion. Return that mismatch early. Asking for more elegant prose will not create a delivery receipt. The reviewer should identify the missing evidence and the person or process capable of supplying it.
Back to contents · Continue to chapter 3
3. Match verification depth to the consequences of error
Risk-appropriate review starts with the next use, not with a universal percentage of outputs to inspect. Ask how a wrong result could affect people, money, time, confidentiality, service quality or later decisions. Then consider whether the error would be easy to notice, whether it would spread, and whether it could be reversed. These questions help select checks; they are not a numerical formula that certifies safety.
For a small internal list of proposed discussion topics, a missing idea may be inconvenient but easily corrected. For an internal notice containing the wrong location, colleagues may waste time travelling to the wrong place. For a reusable procedure, one wrong instruction may be repeated many times. Even within low-risk office work, the cost of a mistake depends on how the output will be used and reused.
Use a simple local triage rather than pretending to calculate a universal risk score. The lightest route is for disposable exploratory material clearly labelled as ideas. A fuller route is for factual internal work that will guide someone else’s action. A separate authorised route is for consequential, sensitive or specialist work. The third route is a boundary on this guide, not an invitation to make high-impact decisions with a longer checklist.
For exploratory material, verify that the output is clearly presented as exploratory and does not smuggle in factual assertions, commitments or private information. For actionable low-risk work, check all decision-critical facts, calculations and conditions. For reusable material, inspect the boundary cases and the instructions that will affect future instances. A sample of ordinary cases should not replace a test of the exception the procedure explicitly promises to handle.
Review intensity can change during inspection. Suppose a cupboard summary initially appears to be a harmless count. The draft then says that the team has placed a paid order. That claim introduces a commitment and a completion assertion that the original task did not authorise. Do not keep reviewing it as a simple inventory summary. Stop that claim, establish what actually happened, and restore the approved scope.
A useful distinction is between a wrong detail and a wrong decision path. A spelling error in a descriptive sentence can be locally corrected. A rule that treats pending approval as approval changes the route by which work reaches release. The latter deserves a broader check because the same misunderstanding may appear in multiple paragraphs or outputs. Classification should follow the consequence, not how small the edit looks.
Do not infer low risk from the absence of a dramatic topic. An ordinary spreadsheet can contain private contact details. A friendly internal message can announce a change that nobody approved. A routine summary can falsely attribute a commitment to a colleague. Scope, data and audience determine the review boundary together. If any of them changes, reconsider the required checks before allowing the output to proceed.
NIST’s AI Risk Management Framework 1.0 presents risk management as contextual and describes trustworthiness through several characteristics, including validity and reliability, accountability and transparency. It is a voluntary framework rather than a certificate for a particular document. This guide uses that contextual principle to motivate bounded review decisions; the practical office procedure and fictional examples below are an editorial teaching design. See NIST AI RMF 1.0.
Choose checks that address a named failure
Instead of adding more review steps indiscriminately, connect each check to a plausible failure. A source-version check addresses stale information. A recomputation addresses an incorrect transformation. A destination check addresses inappropriate release. A completeness check addresses required information disappearing during summarisation. If a proposed check has no clear purpose, reconsider whether it adds evidence or merely makes the process look thorough.
The aim is not maximum paperwork. It is enough appropriate evidence for a defined decision, with explicit limits. A short record showing the correct source, the calculation and the resolved discrepancy can be stronger than a long form filled with unchecked boxes. The work becomes trustworthy through what was established, not through the number of review fields completed.
Back to contents · Continue to chapter 4
4. Give the reviewer competence, independence and permission to stop
A reviewer needs the ability to evaluate the work, access to the necessary evidence and authority to withhold acceptance. A name in an approval field does not provide any of these. Before assigning review, ask whether the person understands the task, can inspect the sources, can reproduce the important checks and can distinguish a genuine uncertainty from a stylistic preference.
Competence is specific to the review question. Someone who writes clear prose may be able to improve a notice but may not understand the spreadsheet logic behind its totals. Someone who can calculate a percentage may not know whether the denominator is the right population. A person familiar with the meeting may detect an invented commitment that an otherwise careful outsider would miss. Choose reviewers for the claims they need to judge.
For small low-risk tasks, one person may possess all the required knowledge. For mixed work, divide the review explicitly. One reviewer checks the source interpretation, another verifies calculations, and a release owner checks destination and authority. Record these roles without implying that each reviewer examined everything. Combining narrow checks can be useful, but only when the boundaries between them are visible and no critical requirement falls into a gap.
Independence means creating a genuine opportunity for the evidence to overturn the draft. A second person can still be anchored by the first person’s explanation. The same person can sometimes perform a useful independent check by recalculating from the original records before looking at the generated total. Independence is therefore about method as well as staffing. Neither a different name nor a different AI model automatically creates an independent evidence path.
For a numerical task, ask the checker to compute the result from the source rows rather than verify the model’s sequence line by line. For a summary, ask the checker to identify the decisions and unresolved questions from the notes before reading the generated summary. For a notice, have the checker extract the confirmed date, place and audience from the booking record. These methods make disagreement possible before the answer has framed the problem.
The reviewer also needs permission to say that the evidence is insufficient. If every review is expected to end in approval, the role becomes ceremonial. Make hold, rework and reject legitimate outcomes with clear next steps. A hold should identify what evidence or decision is missing. Rework should specify the repair and the retest. Rejection should explain why the artefact should not be used for the intended purpose.
Time and workload matter operationally even when they are not measured in a formal study. If the assigned reviewer cannot inspect the required evidence before the release time, the stated procedure has not been completed. Do not relabel a skim as a full review. Narrow the permitted use, move the release, use an appropriately qualified alternative, or keep the work on hold. A deadline does not alter what has actually been checked.
Do not use AI agreement as the final authority
An AI tool may help list claims, compare versions, identify inconsistent units or suggest questions. Treat its review notes as further candidate work. Open the cited source and perform the decisive check yourself or through an appropriate independent method. If the reviewer asks the same system whether it is correct and receives yes, the conversation has not added the missing external evidence.
NIST’s Generative AI Profile identifies confabulation and human-AI configuration risks, including over-reliance and automation bias. It also notes that generated logic or citations can be false. These are reasons to examine the support behind an answer instead of treating its confident presentation as proof. They do not establish that every output is wrong or quantify the error rate of any tool used by a particular workplace. See NIST’s Generative AI Profile, sections 2.2 and 2.7.
Back to contents · Continue to chapter 5
5. Turn fluent prose into checkable claims
A polished answer arrives as a continuous piece of writing. Review becomes easier when that flow is broken into claims that have different evidence requirements. Separate direct facts, derived quantities, interpretations, proposals, instructions and completion statements. A sentence may contain more than one kind. Mark the parts rather than giving the whole sentence a single vague correct or incorrect label.
For example, the sentence Tuesday’s session has twenty places, so everyone can attend contains a capacity claim and an inference about attendance. A booking record might establish twenty places. It cannot establish that everyone can attend unless the relevant group size and eligibility conditions are also known. A citation attached to the sentence may support the first part while leaving the second unsupported.
Use a claim ledger for work with several consequential statements. Give each claim a short identifier, quote or paraphrase the proposition, point to the source location, describe the check, and record the outcome. A small notice may need only a few lines. A longer report may need grouped claims. The ledger should help the next reviewer retrace the reasoning without making them reconstruct the entire chat history.
Evidence should be assessed for identity, relevance and sufficiency. Identity asks whether the record is what it claims to be. Relevance asks whether it concerns this task, date, item or audience. Sufficiency asks whether it supports the full proposition. A genuine document about a different workshop is not relevant evidence for this workshop. A current booking request is not sufficient evidence of confirmation. A copied quotation without a recoverable source is not a complete evidence trail.
Check the source itself, not merely the generated citation label. Open the relevant record, locate the passage or row, and read enough surrounding context to understand its conditions. A statement can become misleading if a qualifier is omitted. Available after confirmation is not equivalent to available. Up to twelve is not equivalent to twelve guaranteed. The reviewer should preserve these boundaries in the accepted wording.
Use status language with care. Proposed, requested, scheduled, approved, started and completed describe different states. When a summary changes one state into another, it can make an unauthorised action appear settled. Require a source for the transition. If the notes say that an owner will check a detail, the summary should not report that the detail has already been checked.
Absence deserves its own claim check. A generated answer may omit a condition that would change the next action while keeping every included sentence individually true. Compare the output against the contract and the source obligations, not just against itself. Ask what the reader needs to know before acting and whether the answer has preserved that information. Truthful fragments can still produce an unfit whole.
Distinguish a recommendation from the evidence supporting it. A proposal to move an internal briefing to a smaller room may be reasonable, but it is still a proposal unless the authorised owner has decided. The reviewer can accept the analysis for discussion without converting it into an instruction. Good labels help readers understand which parts are established, which are inferred and which require a decision.
Resolve source conflict explicitly
When two records disagree, do not average them or let the system choose the more convenient one. Identify the conflict and apply the agreed source-authority rule. A later timestamp alone is not always decisive: a recent informal message may refer to an older draft, while an earlier approved record may remain the operative source. The responsible source owner may need to resolve the discrepancy.
If the conflict cannot be resolved within the review, state both the disputed field and its effect. For example, the room is unconfirmed because the booking record says Cedar while the draft agenda says Birch; the notice remains on hold pending the booking owner’s confirmation. This is more useful than saying that some details may vary. It tells the next person exactly what must happen before release.
Back to contents · Continue to chapter 6
6. Recalculate quantities and test the logic around them
Arithmetic review has at least three layers: the right inputs, the right operation and the right interpretation. A mathematically correct result can still be unfit if it includes cancelled records, combines unlike units or describes the wrong population. Begin by defining the quantity in words. Then identify the records that belong in it. Only then calculate and check the wording used to report it.
Suppose a team wants the number of usable display cards. The source lists items on the shelf, items reserved for another event and damaged items. A calculation of total shelf stock is not a calculation of usable unreserved stock. The reviewer must establish how the categories relate before subtracting anything. If damaged items are already excluded from the shelf count, subtracting them again would create a new error.
Keep units attached to inputs and outputs. Packs, individual items, hours, minutes and events are not interchangeable. A statement that twelve packs equal twelve items may be a unit error even though no arithmetic symbol appears. If a packet contains eight items, write the conversion before combining totals. Where package sizes differ, compute each line separately. Do not assume a shared unit because the labels look similar.
For percentages, identify both numerator and denominator. State what counts as a success, what counts as an eligible case and how excluded cases are handled. A completion rate among cases due this week is different from a completion rate among every case in the file. Neither is automatically the correct measure; the task determines which one is relevant. The reviewer should make that definition visible before accepting the percentage.
Separate percentage change from percentage-point change. If a fictional correct-field rate moves from seventy percent to eighty percent, the increase is ten percentage points. The relative increase is ten divided by seventy, approximately fourteen point three percent. These answer different questions. A reader should not need to infer which meaning the writer intended from a loosely labelled improvement claim.
Use unrounded values for intermediate calculations when possible, then apply a stated rounding rule to the final presentation. If a small total is split across categories, rounded percentages may not add to exactly one hundred. Explain that when it matters rather than adjusting a category arbitrarily to make the display look tidy. For simple office packets, whole counts and explicitly rounded percentages usually make the review easier to reproduce.
Check conservation relationships where the task supports them. Opening stock plus received stock minus issued stock should reconcile to closing stock if those categories are defined consistently and no other movements occurred. If the record includes adjustments, returns or damaged items, include them according to the source definition. Reconciliation is useful because it tests the relationship between parts, but it cannot prove that the original physical count was accurate.
Logic checks should also test boundaries. What happens at zero? What happens when the value equals the threshold? What happens when a required field is missing? What happens when a record is pending rather than complete? A rule that works for the ordinary example may fail at these edges. State the boundary convention, such as at least or greater than, and verify that the generated work follows it.
Correct arithmetic does not establish causation
A before-and-after difference can be calculated without proving why it occurred. If a team processes a different mix of tasks, changes staffing or records time differently, the observed difference may have several explanations. Report a descriptive result as descriptive unless the evidence supports a stronger claim. In this guide’s fictional packets, the numbers are deliberately small and illustrative; none is evidence that SI improves a real workplace by a particular amount.
The reviewer should be able to explain the result without relying on the generated narrative. If they cannot identify the included records, reproduce the calculation and state its limit, acceptance is premature. A spreadsheet formula, calculator or simple script can assist recomputation, but its inputs and logic still need inspection. A tool result is evidence of what that tool calculated, not automatic proof that the task was correctly defined.
Back to contents · Continue to chapter 7
7. Understand what sampling can and cannot establish
Sampling is a choice about which items receive inspection. It does not mean that the uninspected items have passed. A useful sampling record states the full population, the selection method, the checks performed, the findings and the limits of the conclusion. Without these details, reviewed five examples can be mistaken for verified the batch.
First decide whether sampling is appropriate at all. A short notice with six decision-critical facts should usually have all six checked. A small calculation with four source rows can be recomputed completely. Sampling becomes relevant when there is a larger repetitive population and the review question can tolerate an explicitly limited conclusion. Do not sample away a requirement that every critical item must be verified before use.
Distinguish random selection from deliberate coverage. Random selection gives items a defined chance of inclusion and can support a statistical analysis under stated assumptions. Deliberate coverage selects cases because they exercise different conditions, such as missing fields, unusual units or a pending status. Both can be useful, but they answer different questions. A hand-picked set of difficult cases is not a random estimate of the overall error rate.
A convenience sample is something else again. Reading the first five outputs because they are easy to find tells you about those five outputs. It may miss a later source version, an exception category or a different template. If the first five look correct, do not extend that finding to the rest of the file without explaining why the selection is informative for the actual review question.
Here is a small mathematical illustration, not a recommended workplace sample size. Imagine twenty cards, exactly one of which contains an error, and select five distinct cards uniformly at random. The erroneous card has a five-in-twenty chance of being selected, or twenty-five percent. The chance of missing it is fifteen-in-twenty, or seventy-five percent. A clean five-card sample is therefore compatible with an error remaining elsewhere.
If the same twenty-card population contains exactly two erroneous cards, the chance that five randomly selected distinct cards miss both is the number of five-card selections from eighteen correct cards divided by the number of five-card selections from all twenty cards. This equals fifteen times fourteen divided by twenty times nineteen, or approximately fifty-five point three percent. The chance of finding at least one is approximately forty-four point seven percent. These figures depend on the stated population, error count and random-selection assumptions.
The reviewer normally does not know the true error count in advance. That is precisely why a clean sample cannot be read as proof that no errors exist. Statistical inference requires an appropriate design and assumptions; a casual spot-check cannot acquire that meaning merely because the result is expressed as a percentage. For consequential uses, obtain the organisation’s appropriate assurance expertise rather than treating these toy calculations as a release standard.
Expand review when the failure reveals a pattern
If a checked output uses the wrong source version, inspect other outputs produced from that same version. If a unit conversion is wrong, inspect the other converted quantities. If a template omits an exception, test every relevant exception path. The scope of the expansion should follow the suspected mechanism, not simply add a few more arbitrary examples.
Keep the original sample result visible after expanding the review. Record which items were originally selected, which defect triggered expansion, and what the wider check found. This avoids rewriting history so that a failed first check disappears inside a later clean result. The final accepted artefact may be correct while the initial review record still shows a useful process weakness.
Back to contents · Continue to chapter 8
8. Use a repeatable sequence from intake to release
A practical review sequence has six stages: establish the contract, inspect the source packet, identify consequential claims, perform the required checks, decide the disposition, and verify the accepted version. The sequence can be brief for a small task. Its value is that each stage produces something the next stage can use, rather than a general impression that the work has been looked at.
At intake, capture the artefact identity and version. If the writer continues editing during review, use a stable copy or an agreed pause so that findings refer to a definite object. Record the requested audience and intended use. Check whether the reviewer has the source access and competence needed. If either is missing, resolve that before declaring substantive review underway.
Next inspect the source packet for completeness and authority. Confirm that the specified records are present, readable and relevant. Identify superseded versions and unresolved conflicts. Note any source limitation that would constrain the final answer. Do not allow the generated output to conceal that the source packet itself is incomplete. A review can establish that a summary faithfully reflects an incomplete record, but it cannot turn that record into complete knowledge.
Then identify the claims that affect the next action. Start with dates, quantities, owners, statuses, conditions and assertions of completion. Add required omissions: information that must appear even if the draft left it out. For a short task, mark these directly beside the text. For a larger task, use the claim ledger. The point is to create a finite set of things to establish.
Perform the checks using the method suited to each claim. Compare source statements, recompute derived values, test conditions and inspect the destination rules. Separate evidence findings from editorial improvements. If the reviewer changes the wording, check whether the change also changes the meaning. A tidy rewrite can accidentally remove an important qualifier just as the original generation can.
Make the disposition explicit. Accept means the identified version is fit for the stated use within the recorded limits. Rework means specified repair is required before acceptance. Hold means a necessary fact, source or decision is missing. Reject means the artefact should not be used for the proposed purpose and needs replacement or a different route. These definitions are recommended local working labels, not a universal industry classification.
Finally, inspect the corrected and accepted version itself. Do not assume that a requested correction was applied correctly because the writer says it was. Compare the changed fields, rerun any affected calculations, and inspect nearby text for consequences. Where the output is moved into another system, check the saved copy and its destination. The review record should point to the version people will actually use.
Prepare checks before the pressure of release
A team can prepare recurring criteria and examples before the next batch arrives. This makes the review less dependent on improvisation at the deadline. AISI described preregistration and quality assurance of evaluations before model onboarding, including advance work on methods and reporting. That source concerns frontier-model evaluation, not office-document approval, but it offers a useful analogy for preparing review criteria before inspecting a result. See AISI’s early lessons from evaluating frontier AI systems.
Preparation should not become a rigid script that ignores the actual task. Use the recurring checklist as a starting point, then add task-specific claims and conditions. A template can remind the reviewer to check dates; only the current source can establish which date is correct. The repeatable part is the method of inquiry, not the answer.
Back to contents · Continue to chapter 9
9. Worked packet A: review a stationery readiness summary
This packet concerns a fictional internal workshop at Alder Desk Studio. The task is deliberately low risk: prepare a planning note for the office coordinator. No ordering, purchasing, supplier contact or allocation change is authorised. The reviewer must determine whether the available stationery meets the requested quantities, item by item, using the current cupboard count.
The source packet has three records. A1 is the request note: thirty notebooks, sixteen marker pens, thirty-six folders and twenty-four adhesive label strips are required. Each quantity refers to individual usable, unreserved items. The coordinator wants a readiness summary and any shortages identified. Surplus folders cannot substitute for missing notebooks. Items reserved for another event cannot be reassigned through this task.
A2 is the current cupboard record, labelled version two. It states that the physical count includes damaged items and items reserved for other activities; the damaged and reserved categories do not overlap. Notebooks: forty-three physically present, three damaged, twelve reserved. Marker pens: thirty-two physically present, four damaged, eight reserved. Folders: fifty-eight physically present, none damaged, eighteen reserved. Label strips: five packs of six strips physically present, two individual strips damaged, four individual strips reserved.
A3 is an older cupboard record, labelled version one and explicitly superseded by A2. It lists forty-six notebooks physically present, three damaged and twelve reserved. The other item counts are not part of A3. The purpose of including A3 is to test whether the review respects the declared current source rather than silently selecting the more favourable count.
The generated work to review
The fictional generated draft reads: Stationery is ready for the workshop. We have thirty-one notebooks, twenty-four marker pens, forty folders and thirty label strips available, for a total of one hundred and twenty-five items against one hundred and six requested. All requirements are covered, and the spare items provide a useful buffer. The coordinator can proceed without another cupboard check.
Do not begin by editing the tone. Extract the claims. The draft asserts four available quantities, one available total, one requested total, complete item coverage and a permission-like conclusion about proceeding. Several numbers may appear plausible, and the totals can be internally consistent, while the conclusion remains wrong. This is why the reviewer must return to the source definitions.
Independent reviewer calculation
For notebooks, available usable unreserved stock is forty-three minus three minus twelve, which equals twenty-eight. Against a request for thirty, there is a shortage of two. The generated figure of thirty-one matches the superseded count of forty-six minus three minus twelve. The error is therefore not merely an arithmetic slip; it reveals use of the wrong source version.
For marker pens, thirty-two minus four minus eight equals twenty. Against sixteen requested, there is a surplus of four. The generated figure of twenty-four appears to subtract the reserved pens while leaving the four damaged pens in the available quantity. The correct result must exclude both categories because A2 explicitly says they are included in the physical count and do not overlap.
For folders, fifty-eight minus zero minus eighteen equals forty. Against thirty-six requested, the surplus is four. This generated quantity is correct. Record that result rather than treating every field as suspect merely because other fields failed. Review should distinguish established correct work from identified defects, while still deciding whether the whole artefact is fit for use.
For label strips, convert the five packs of six into thirty individual strips. Then subtract two damaged strips and four reserved strips, leaving twenty-four available. Against twenty-four requested, the requirement is met exactly, with no surplus. The generated figure of thirty is the physical quantity before the required exclusions. The unit conversion is correct, but the meaning of available is wrong.
The available total is twenty-eight plus twenty plus forty plus twenty-four, which equals one hundred and twelve. The request total is thirty plus sixteen plus thirty-six plus twenty-four, which equals one hundred and six. The aggregate difference is six. That positive difference does not establish readiness because the notebook shortage remains. The draft’s total of one hundred and twenty-five correctly adds its own four wrong-or-right figures, but those figures are not the authorised available quantities.
Findings and disposition
The reviewer records a source-version defect for notebooks, an exclusion defect for marker pens, an exclusion defect for label strips, and a conclusion defect for item coverage. The generated requested total is correct, and the folder quantity is correct. The readiness conclusion and proceed instruction require rework because one required item is short. No purchase decision is made by this review.
The reviewer also removes the claim that no further cupboard check is needed. The request asked for a planning summary, and the current record establishes a shortage, not an authorised remedy. The coordinator may choose a physical recount or another approved next step, but the generated note should not invent that decision. Correct review respects the task boundary even when an obvious practical solution seems available.
Corrected output
The corrected fictional note reads: Based on current cupboard record A2, version two, the workshop requirement is met for marker pens, folders and label strips. Notebooks are short by two. Available usable unreserved stock is twenty-eight notebooks, twenty marker pens, forty folders and twenty-four label strips. The corresponding requests are thirty, sixteen, thirty-six and twenty-four. Marker pens have a surplus of four, folders a surplus of four, and label strips no surplus. The coordinator needs to resolve the notebook shortage before treating the stationery set as ready. This note does not authorise a purchase or reassignment of reserved stock.
The reviewer recomputes all four quantities in the corrected note, confirms the shortage statement, checks that A2 is named, and accepts the note for internal planning. Acceptance does not mean the physical cupboard has been independently recounted. It means the corrected note faithfully applies the stated rules to the supplied current record. That source limitation belongs in the review record.
What transfers to other workplace tasks
This packet demonstrates three distinct problems that a fluent summary can hide. A stale source can produce a mathematically correct wrong answer. An ambiguous category can produce the wrong available quantity. An aggregate surplus can conceal a shortage in a non-substitutable item. The useful repair is not a generic instruction to be more careful; it is to enforce source precedence, define availability and compare requirements at the item level.
When transferring the method, preserve the relationship between the categories. If a real stock system already excludes damaged items from its available field, do not subtract damage again. The worked calculation is correct because A2 explicitly defines its categories. Review should apply the actual source semantics, not copy a familiar formula into a different data structure.
Back to contents · Continue to chapter 10
10. Worked packet B: review meeting decisions without inventing agreement
This packet concerns a fictional team arranging an internal show-and-tell session. The task is to turn the supplied notes into a short decision-and-action summary for the three participants. The output is for internal reference. It must preserve proposals, conditions and unresolved matters, and it must not send invitations or create commitments that the notes do not establish.
Record B1 contains the meeting notes. Mira says, We could use Cedar Room on 14 October from 14:00 to 15:00 if facilities confirms it. Dev says, I can check the room request by 8 October; that does not mean the room is booked yet. Lin says, I will draft a six-slide outline by 9 October, but the final presentation will depend on which demonstrations are approved. Mira says, We have twelve interested colleagues so far. The room request is for up to sixteen attendees. No invitation has been sent.
Record B2 is the decision note written at the end of that meeting. It states that the team agreed to prepare a draft show-and-tell plan. The proposed event date, time and room remain conditional on facilities confirmation. Dev owns the room-status check. Lin owns the draft outline. Mira will review the outline after it is available. No deadline was assigned to Mira. No one was assigned to send invitations.
Record B3 is a later status message from Dev, dated 7 October in this fictional packet. It states: Facilities has received the request but has not confirmed Cedar Room. I still expect to check the status by 8 October. B3 adds current status but does not change the proposed event details or the task owners. The record is supplied in full so that the reader can independently distinguish a request from a booking.
The generated work to review
The fictional generated draft reads: The team confirmed the show-and-tell for 14 October, 14:00 to 15:00, in Cedar Room for sixteen colleagues. Dev has booked the room. Lin will deliver the final six-slide presentation by 9 October. Mira will approve the slides by 10 October and send invitations immediately afterwards. All attendees have agreed to join, so the team can move directly to delivery.
This draft is coherent. Every sentence fits a plausible event-planning story. That coherence is precisely why the review needs a claim-by-claim comparison. The sources contain a proposed event, a request awaiting confirmation, a draft outline, twelve expressions of interest, and unassigned invitation work. The draft has converted those separate states into a completed plan.
Reviewer findings
The event’s date, time and room are proposed and conditional, not confirmed. B1 supplies the proposal, B2 explicitly preserves the condition, and B3 confirms that the room is still pending. The first sentence therefore overstates status. The correct repair is to label the event details proposed and state the unresolved facilities dependency, not simply add the word tentative in an unrelated closing note.
Dev has not booked the room in the supplied record. B3 says that facilities received the request without confirming it. Dev’s assigned action is to check status by 8 October. The generated completion claim needs removal. A promise to check and an acknowledged request are neither evidence of completed booking nor evidence that the event can proceed in that room.
Lin promised a draft six-slide outline by 9 October. The final presentation depends on demonstrations being approved. Changing draft outline to final presentation changes both the deliverable and the commitment. The reviewer restores the exact task level and preserves the dependency. This matters even though the slide count and date in the generated text are individually supported.
Mira’s review action is supported, but the 10 October deadline is invented. The invitation action is unassigned. A sensible-looking deadline or owner does not become a source fact because it would make the plan more complete. The reviewer can separately suggest that the team assign these details, but the decision summary must not portray that suggestion as an agreement already made.
The attendee statement confuses interest, requested capacity and confirmed attendance. Twelve colleagues have expressed interest. The room request is for up to sixteen attendees. The sources do not say that sixteen people agreed to join, or that invitations were issued. Capacity describes a limit in a request, not a count of accepted invitations. No attendance rate can be calculated from this packet because the required invitation and response population does not exist in the supplied record.
Corrected output
The corrected fictional summary reads: The team agreed to prepare a draft internal show-and-tell plan. The proposed event is 14 October, 14:00 to 15:00, in Cedar Room, subject to facilities confirmation. Dev will check the room status by 8 October; his 7 October update says the request has been received but not confirmed. Lin will produce a draft six-slide outline by 9 October. The final presentation depends on approval of the demonstrations. Mira will review the outline once available; no review deadline was assigned. Twelve colleagues have expressed interest, and the room request is for up to sixteen attendees. Invitations have not been sent, and an invitation owner remains to be assigned.
The reviewer accepts this corrected summary for internal reference after checking every status and owner against B1 through B3. The unresolved invitation owner is a useful gap, not an output defect, because the source genuinely leaves it open. The summary makes the gap visible without inventing a decision. A separate planning conversation can resolve it.
The rework instruction that would have helped
A targeted rework request would say: restore the distinction between proposed and confirmed event details; use B3 for current room status; retain draft outline as Lin’s deliverable; remove Mira’s unsupported deadline and invitation assignment; distinguish twelve interested colleagues from the requested capacity of sixteen; and list invitation ownership as unresolved. After revision, provide a claim-to-source check for each changed statement.
That instruction is more useful than make the meeting summary accurate. It identifies the failure types and the fields affected. The reviewer should still inspect the revised summary, because a correct rework instruction is not evidence that the repair was correctly applied. Acceptance belongs to the returned version, not to the quality of the feedback.
Back to contents · Continue to chapter 11
11. Worked packet C: review a rewritten internal borrowing guide
This fictional packet concerns reusable display boards in an office resource cupboard. The task is to rewrite a short internal borrowing guide in plain language while preserving every operative rule. The output is intended for a draft knowledge-base entry reviewed by the resource coordinator. It does not change the borrowing policy, issue a booking confirmation or authorise any external communication.
Record C1 is the current guide, version three. Rule one: a request must specify at least one board; requests for one to three boards use the standard request route. Rule two: requests for four or more boards require the resource coordinator’s approval. Rule three: a standard loan lasts no more than two working days; a longer loan requires the coordinator’s approval before collection. Rule four: submission of a request does not confirm availability; collection is allowed only after confirmation. Rule five: borrowers should report damage to the coordinator when returning the boards. The guide does not state a fee or penalty.
Record C2 is the requested writing brief. Use friendly plain language, retain all five rules, and include a clear distinction between requesting and receiving confirmation. Do not add a charge, promise availability or invent a contact address. Use the role name resource coordinator because the packet supplies no named individual or contact details. No separate legacy policy is supplied or authorised for use.
The generated work to review
The fictional generated draft reads: Need a display board? Submit your request and collect your boards from the cupboard. You can borrow up to four boards for three working days without approval. For bigger requests, ask the resource coordinator. Please return everything on time. A small replacement fee applies if an item is damaged.
The draft is shorter and friendlier than the source. Those are real stylistic properties, but they do not establish a successful rewrite. It changes the quantity threshold, extends the standard duration, removes confirmation as a condition of collection, omits the damage-reporting instruction and invents a fee. A reviewer focused only on readability could approve precisely the characteristics that make the wrong rules easier to follow.
Reviewer findings and boundary checks
The quantity rule needs testing at the boundary. A request for three boards uses the standard route; a request for four boards requires coordinator approval. The generated phrase up to four boards without approval gets the boundary wrong. Testing only a request for two boards would not reveal the defect. This is why a review of reusable instructions should include the transition point, not just a typical case.
The duration rule is also wrong. Two working days is the standard maximum. Three working days requires coordinator approval before collection because it exceeds that maximum. The reviewer does not need to calculate a particular calendar return date in this packet: no collection date, working-day calendar or holiday rule is supplied. A corrected guide should preserve the rule without inventing a date convention.
The collection instruction fails because submitting a request and receiving confirmation are different states. The draft tells readers to collect after submission, bypassing the required confirmation. The repair must restore the condition in the instruction itself. A general line saying subject to availability would still be less precise than the source, which explicitly requires confirmation before collection.
The replacement fee is unsupported. Its amount is not merely missing; the policy basis itself is absent. The reviewer removes the fee rather than inventing a number or substituting a softer phrase such as a fee may apply. The source supplies a damage-reporting requirement. The correct rewrite should tell borrowers to report damage to the coordinator when returning the boards.
The minimum request quantity also matters for a reusable guide. The source says at least one board. A request for zero boards does not satisfy that requirement. This need not become an awkward headline in the public-facing wording, but a request form or rule implementation using this guide should not treat zero as a valid standard request. The reviewer records the boundary so that the simplified prose does not erase it.
Corrected output
The corrected fictional guide reads: To borrow display boards, submit a request for at least one board and wait for confirmation before collecting them. Requests for one to three boards use the standard request route. If you need four or more boards, ask the resource coordinator for approval. A standard loan lasts up to two working days. If you need a longer loan, obtain the coordinator’s approval before collection. When returning the boards, report any damage to the resource coordinator.
The reviewer compares this version with all five source rules. The quantity threshold, duration threshold, confirmation condition and damage-reporting instruction are present. No fee, contact address or availability promise has been added. The text is accepted as a draft knowledge-base rewrite, with final placement subject to the named coordinator’s normal release process.
Test cases for the corrected guide
Case C-a requests one board for one working day and has received confirmation. It fits the standard quantity and duration rules, so collection is allowed under the supplied guide. Case C-b requests three boards for two working days and has received confirmation. It also fits the standard route. These two cases establish ordinary and upper-bound standard use within this fictional rule set.
Case C-c requests four boards for two working days. Coordinator approval is required because of quantity, and confirmation is still required before collection. Case C-d requests two boards for three working days. Coordinator approval is required because of duration, and confirmation remains necessary. Approval of an exception and confirmation of availability should not be silently treated as the same record unless the real procedure explicitly combines them.
Case C-e requests two boards for one working day but has only submitted the request. Collection is not yet allowed. Case C-f requests zero boards. It fails the minimum-quantity requirement and should be clarified rather than processed as a valid loan. Case C-g returns a damaged board. The supported instruction is to report the damage to the coordinator at return; the packet does not establish a replacement fee.
These cases test different conditions independently. If the team changes a quantity threshold later, it should revise and retest the affected cases while preserving the unaffected rules. A rewrite review is therefore more than a reading exercise: it checks whether readers following the simplified version would take the same permitted actions as readers following the authoritative source.
Back to contents · Continue to chapter 12
12. Write rework instructions that produce a checkable repair
A useful finding has four parts: the location, the problem, the evidence and the required repair. The location lets the writer find it. The problem identifies the mismatch. The evidence explains why the reviewer reached that conclusion. The repair describes what needs to change without dictating unnecessary stylistic choices. Add the retest when the correction could affect other parts of the work.
For example: in the opening sentence, confirmed event overstates the status; B2 and B3 leave the room pending; replace the statement with a proposed event subject to facilities confirmation and check every later sentence for the same assumption. This is a repairable finding. Make it less confident is not enough because the writer may weaken the tone while leaving the unsupported booking claim intact.
Group findings by cause when that helps prevent repeated errors. If several quantities use the same obsolete source, identify the source-precedence problem and require all affected figures to be recomputed. If several action items have invented owners, require a full owner-to-source comparison. Do not force the writer to fix each sentence independently when a shared misunderstanding explains the pattern.
Keep the original findings after correction, but update their state. Open means the issue remains. Fixed and rechecked means the revised artefact was inspected against the required evidence. Not applicable means the requirement genuinely does not apply, with a reason. Deferred should identify the permitted scope and the unresolved dependency; it should not become a polite synonym for accepting a consequential error.
Review the change itself for regressions. A corrected notebook total may leave an old all requirements met sentence elsewhere. A repaired event status may still be contradicted by a heading saying confirmed programme. A guide may restore a duration rule in one paragraph while an example below uses the old value. Search the whole artefact for the affected concept, not just the line mentioned in the original finding.
Avoid endless stylistic loops after the substantive criteria are satisfied. Agree which preferences matter to the intended audience and which are optional. Once the evidence, required content, meaning and release conditions are sound, a reviewer should be able to accept a version without making it match their personal voice. Rework should improve fitness for use rather than demonstrate that the reviewer can rewrite the writer.
Know when to stop the generation loop
Stop and change the approach when the same material error recurs despite clear evidence and a specific repair request. The next useful step may be to simplify the task, repair the source packet, perform a deterministic calculation, or ask a competent human to produce the required section directly. Repeated regeneration is not inherently progress if it keeps returning unverified variants of the same unsupported answer.
Also stop when the missing element is authority rather than wording. A model cannot resolve an unassigned decision owner by generating a plausible name. It cannot turn an unconfirmed booking into a confirmed one by restating the request. Keep the dependent output on hold and direct the unresolved question to the appropriate owner through the organisation’s normal process.
Back to contents · Continue to chapter 13
13. Keep an audit trail that another person can actually use
A review record should let a colleague understand what was accepted, why it was accepted and where its limits lie. It does not need every conversational exchange. Preserve the task contract, the source identifiers or stable copies allowed by your organisation, the reviewed version, the material findings, the checks performed, the final disposition and the accepted version. Use existing approved storage and access practices.
Record the distinction between source and generated material. If a reviewer annotation is placed beside the source, label it as an annotation. If a generated explanation is included in the packet, label it as generated work. Otherwise a later reader may mistake a helpful interpretation for part of the authoritative record. Provenance is useful only when the roles of the artefacts remain clear.
For packet A, a concise record would say that revision two of the stationery note was accepted for internal planning using A1 and current A2; A3 was superseded; all four item calculations were recomputed; the notebook shortage was retained; and no purchase or reassignment was authorised. It would also state that the physical stock was not independently recounted. This gives the next user a practical boundary without making the note unreadable.
For packet B, the record would identify the reviewed source set B1 through B3 and the corrected summary. It would note that every owner, date and status was checked, the event remained proposed, and invitation ownership was unresolved. That record would make it harder for someone later to claim that the summary itself confirmed the room or assigned an invitation sender.
For packet C, retain the rule version and the boundary cases used. If a later guide changes the standard loan duration, the team can see which accepted version reflected the older rule and which tests need updating. An audit trail supports correction by connecting a conclusion to its inputs, rather than presenting acceptance as an unexplained permanent badge.
Record who performed each material check and when, using the team’s ordinary identity and timestamp conventions. Do not claim independent review if the same person merely reread their own output without a separate evidence method. A transparent limited check is more useful than an overstated assurance label. The record should describe actual work, not the process everyone hoped had occurred.
An accepted version can become stale when its sources change. Define what kind of change requires review to reopen: a new source version, a different destination, a changed quantity, a revised rule, or an additional external commitment. Do not assume that an old approval automatically transfers to a materially changed artefact. At the same time, do not restart an entire review for an inconsequential formatting adjustment if the team’s process permits a narrower check.
Back to contents · Continue to chapter 14
14. Measure the review process without rewarding rubber-stamping
Measures should help the team locate bottlenecks and missed errors, not pressure reviewers to approve work quickly. A high acceptance rate can reflect good generation, easy tasks, weak criteria or superficial checking. A low rate can reflect difficult inputs or overly strict preferences. Interpret counts with the task mix, review depth and source quality visible alongside them.
Consider a fictional cohort of twenty initial artefacts. Twelve are accepted on first review, five require rework and three are held for missing source information. The first-pass acceptance rate is twelve divided by twenty, or sixty percent. The rework share is five divided by twenty, or twenty-five percent. The hold share is three divided by twenty, or fifteen percent. These categories describe initial dispositions and together cover the twenty artefacts.
After one revision round, four of the five rework artefacts are accepted and one still requires rework. The original three holds remain unresolved. The cohort now has sixteen accepted artefacts, one in rework and three on hold. The accepted share is sixteen divided by twenty, or eighty percent. The unresolved share is four divided by twenty, or twenty percent. Do not replace the original first-pass rate with eighty percent; they describe different stages.
Suppose the first reviews took six minutes each for the twelve initial acceptances, twelve minutes each for the five rework cases, and eight minutes each for the three holds. That is seventy-two plus sixty plus twenty-four, or one hundred and fifty-six review minutes. The four successful revision checks took four minutes each and the still-open revision check took six minutes, adding twenty-two minutes. Total recorded review effort is one hundred and seventy-eight minutes.
Dividing that total by the sixteen accepted artefacts gives approximately eleven point one review minutes per accepted artefact across this cohort. This includes review effort spent on unresolved items. It is not the average elapsed turnaround time, and it is not the labour needed to generate or revise the artefacts. There is no comparison group here, so the numbers do not establish productivity improvement from SI.
Track defects discovered after release separately from findings caught before acceptance. If no post-release observation has been performed, the escaped-defect rate is unknown, not zero. A team needs a defined observation period and a way to detect and record relevant failures before treating the absence of reports as useful evidence. Even then, describe the observation limits rather than implying complete visibility.
A useful review retrospective asks which errors recurred, which checks caught them, which source problems caused holds, and which criteria reviewers interpreted differently. Choose a targeted repair and test it on fresh cases. If the repair only improves performance on examples already discussed in training, the evidence of transfer is limited. Preserve a few unseen examples for an independent check of the revised process.
Back to contents · Continue to chapter 15
15. Independent practice: review before reading the answers
The following exercises use new fictional inputs. Complete a short review record for each: intended use, required claims, source-based findings, corrected output and disposition. Work from the supplied source rather than copying the earlier packets’ conclusions. Some errors involve arithmetic; others involve meaning, authority or the limits of the evidence. A result can be numerically correct and still require rework.
Exercise one: preparing table signs
The source says a team needs thirty usable, unreserved table signs. The current shelf record shows forty-eight signs physically present. Six are damaged and nine are reserved for a different activity. The source explicitly says the damaged and reserved groups do not overlap and are included in the physical count. The task is an internal readiness note only. No transfer of reserved signs or new order is authorised.
The generated note says: Forty-two table signs are available, leaving a buffer of twelve after the thirty-sign requirement. We can release the reserved signs if needed, and no follow-up is necessary. Identify every unsupported or incorrect claim, calculate the correct available quantity and buffer, and write a corrected note within the task’s authority. Decide whether the corrected result meets the stated requirement.
Exercise two: preserving a conditional action
The complete meeting record says: Omar will draft the poster by 12 November. Sian will check the room request by 11 November. The poster can be circulated only after the room is confirmed. At the end of the meeting the room remains unconfirmed. No circulation owner has been assigned. The generated summary says: Sian has confirmed the room, and Omar will circulate the approved poster on 12 November.
List the changes of status, deliverable and ownership introduced by the generated summary. Then produce a corrected internal action summary that preserves the unresolved dependency. Explain whether a competent reviewer can make the circulation ready to proceed using only this record, even if the poster wording is otherwise excellent.
Exercise three: choosing the right denominator
A fictional task log contains twelve records. Eight tasks were due during the reviewed week; six of those were completed by their due time and two were completed late. Two other tasks are due next week and remain open. Two records were cancelled before their due dates. The requested measure is the on-time completion rate among tasks due during the reviewed week, excluding cancelled records.
The generated report says: Six tasks were on time, so the weekly on-time rate is six divided by twelve, or fifty percent. Four tasks remain incomplete. Review the denominator, the percentage and the incomplete-task statement. Write a corrected result with a clear population definition and explain what the packet does not tell you about why tasks were late.
Exercise four: interpreting a clean spot-check
A batch contains twenty internal resource cards. A reviewer reads cards one through five and finds no errors in the fields checked. The team has not specified a random-selection method, and the reviewer has not inspected the remaining cards. The generated review summary says: The batch is one hundred percent accurate and approved for reuse because all sampled cards passed.
Write the strongest conclusion the actual observation supports. State what is unknown, identify why the claim of complete accuracy is unjustified, and propose a next check if cards sixteen through twenty use a different rule version. Do not invent a statistical confidence interval or an error rate for the uninspected cards.
Exercise five: accepting a limited rewrite
An internal source says that draft display captions must include an item name and a location code. The source identifies the item as Blue model bridge, and its location code is not yet assigned. The task says to create draft captions for discussion and mark missing fields as pending. The generated caption says: Blue model bridge. Location: pending confirmation. A colleague rejects it because every caption should look finished.
Decide whether the missing confirmed code is a defect under this task contract. Explain what the reviewer should check before accepting the draft and what would change if the same caption were proposed for final placement. This exercise tests whether review can recognise honest incompleteness instead of rewarding invented certainty.
Full answer one: table signs
Available usable unreserved signs equal forty-eight minus six minus nine, which is thirty-three. The buffer above the requirement is thirty-three minus thirty, which is three. The generated forty-two excludes damaged signs but leaves the nine reserved signs counted as available. Its buffer of twelve is internally consistent with the wrong available quantity but does not answer the task correctly.
The statement that reserved signs can be released is outside the task’s authority. The source gives no permission to transfer them. No follow-up is necessary is also broader than the evidence establishes: the note can state that the recorded available count meets the requirement, but it should not dictate that no operational check or preparation remains. A planning note is not a completed physical setup.
A corrected output is: Based on the current shelf record, thirty-three usable, unreserved table signs are available against a requirement of thirty, leaving a recorded buffer of three. The stated quantity requirement is met. Reserved signs remain excluded, and this note does not authorise a transfer or order. The reviewer can accept that version for internal readiness planning after recomputing both figures and preserving the source limitation.
Full answer two: conditional action
Sian’s task is to check the room request by 11 November. The record does not say that the room is confirmed. Omar’s task is to draft the poster by 12 November, not to circulate an approved poster. The generated summary invents both approval and a circulation assignment. It also turns a draft deadline into a distribution date while ignoring the unresolved room condition.
A corrected summary is: Sian will check the room request by 11 November. Omar will draft the poster by 12 November. The room is currently unconfirmed, so the poster must not be circulated until confirmation is obtained. A circulation owner remains to be assigned. This version can be accepted as an internal action summary because it accurately represents the supplied record.
The reviewer cannot make circulation ready using the existing packet. Room confirmation and ownership require information or decisions outside the supplied evidence. Excellent poster wording does not resolve either dependency. Keep any release decision on hold while allowing the truthful internal summary to proceed for its narrower purpose. Different artefacts can therefore receive different dispositions from the same source packet.
Full answer three: the task-log rate
The eligible denominator is eight tasks due during the reviewed week. Six were on time, so the requested rate is six divided by eight, or seventy-five percent. The two late tasks belong in the denominator because they were due during the week, but they do not belong in the on-time numerator. The next-week tasks and the cancelled records are excluded under the stated measure definition.
The claim that four tasks remain incomplete is unsupported. The two late tasks were completed, and the two cancelled records should not be relabelled incomplete. The packet says that two next-week tasks remain open. A corrected statement is: Among eight non-cancelled tasks due during the reviewed week, six were completed on time and two were completed late, giving an on-time rate of seventy-five percent. Two additional tasks due next week remain open, and two cancelled records are excluded.
The packet contains timing outcomes but no explanation of causes. It does not establish whether lateness resulted from task difficulty, missing information, staffing, SI output quality or anything else. A reviewer should remove causal claims or request the evidence needed to investigate them. The arithmetic correction is complete; a causal diagnosis would be a separate task.
Full answer four: the spot-check
The observation supports this statement: cards one through five had no errors found in the fields that were checked. It does not establish that every field was correct, that the remaining fifteen cards were correct, or that the batch is suitable for every form of reuse. The selection was by position rather than a stated random design, so it should be described as a limited convenience check.
Because cards sixteen through twenty use a different rule version, inspect those cards against the applicable authoritative rules and test any changed conditions explicitly. Also decide whether the intended reuse requires every card’s decision-critical fields to be checked. The next step follows the known version difference and the release contract, rather than extending a clean finding from an unrelated portion of the batch.
A corrected review summary is: no errors were found in the checked fields of cards one through five; the remaining cards have not been reviewed, and batch approval has not been established. Cards sixteen through twenty require a separate version-aware check. The true error rate remains unknown from the supplied observation. This preserves the useful finding without overstating it.
Full answer five: an honestly incomplete draft
Under the stated contract, a pending location code is not automatically a defect. The task permits draft captions for discussion and requires missing fields to be labelled pending. The reviewer should check that Blue model bridge is the supported item name, that the location code is genuinely unassigned, and that the draft label and intended discussion-only use remain clear.
If those checks pass, the caption can be accepted for discussion. Inventing a plausible code would make it look more finished while violating the contract. If the caption were instead proposed for final placement, the review boundary would change. The required location would need to be established through the appropriate source, and the final-placement version would remain on hold until that information was available and checked.
Back to contents · Continue to chapter 16
16. Questions reviewers ask in practice
Should a reviewer always read the entire source packet?
The reviewer needs enough source coverage to establish every required consequential claim and detect material omissions. For the small packets in this guide, that means reading all supplied records. For a larger task, scope the review deliberately and record which sources or sections were inspected. Do not describe partial source coverage as a complete factual review, especially when an uninspected section could contain exceptions that change the answer.
Can the writer review their own SI-assisted work?
Yes, a self-review can be useful when the task and the organisation’s process permit it. Use a separate method: return to the source, compute independently and compare against a prewritten contract. Be honest about the absence of a second person. If the intended use requires independent approval or specialist competence, self-review cannot substitute for that requirement merely because the writer used a careful checklist.
Does a correct citation make a sentence acceptable?
Only if the cited source supports the whole consequential claim in context and is appropriate for the task. A real source can be outdated, concern another case or support only part of a sentence. The reviewer should inspect the passage, its conditions and its relation to the current request. A link is a route to evidence, not a self-executing proof.
What if two competent reviewers disagree?
Identify whether they disagree about the source facts, the interpretation, the acceptance criterion or a preference. Return to the relevant evidence and contract rather than taking a vague majority vote. A factual conflict may need a source owner to resolve it. A preference disagreement may not block acceptance. If the disagreement affects the permitted next action and cannot be resolved, keep that action on hold and record the reason.
Can passing this review establish that an SI system is safe?
No. The review establishes a bounded conclusion about a particular artefact and use. It is not a comprehensive assessment of a model or system. AISI’s account of early evaluation lessons describes independent evaluations as snapshots rather than certifications of safety, although its work concerns frontier-model evaluation at a different scale. See AISI’s early lessons from evaluating frontier AI systems. The practical lesson is to state what the evidence covers and avoid expanding a local result into a universal claim.
What should the final reviewer sentence say?
State the accepted version, the intended use, the basis and any material limit. For example: revision two is accepted for internal planning after all item quantities were recomputed against the current record; the notebook shortage remains unresolved, and no purchase is authorised. A clear sentence like this helps the next person act appropriately. It is more useful than checked, looks good, because it tells them what they can rely on.
Back to contents · Continue to chapter 17
17. Put the procedure to work on one bounded task
Begin with a small artefact whose sources you can inspect completely. Write the review contract before reading the generated answer. Extract the required claims, establish the correct source version, recompute any quantities and test the conditions that change what a reader may do. Decide whether the artefact is accepted, needs rework, must be held, or should be rejected for the proposed use.
Then inspect the final saved version and leave a concise record of what was actually established. If the source changes, the destination changes or an unresolved dependency becomes consequential, reopen the relevant checks. The goal is a workplace habit in which people can explain why an output is fit for its next use, rather than merely recognise that it reads fluently.
Human review becomes useful when it has a real decision to make and enough evidence to make it. The reviewer should neither rubber-stamp the answer nor endlessly rewrite it. They should preserve what is supported, repair what is wrong, expose what is missing, and stop where competence or authority runs out. That is how SI-generated work becomes an accountable part of a workflow.
Continue through the Super Intelligence project
For the broader implementation sequence, return to the Workplace Super Intelligence hub. For the technical mechanisms behind checking, read How Super Intelligence Works: Verification.
Primary sources
- NIST AI Risk Management Framework 1.0: contextual risk and trustworthiness.
- NIST Generative AI Profile: confabulation and human-AI configuration.
- AISI: Early lessons from evaluating frontier AI systems: preregistration and the limits of evaluation conclusions.
