Why English? | Reporting The Systematic Engine Through Appraisal Tools, Bias Profiles, Reviewer Judgements, Disagreements, Certainty and Conclusions

ORIENTATION

What this guide owns

Current fact check

The EQUATOR record identifies opaque rationales, unclear links between domain-level assessments and downstream decisions, and weak reporting of calibration and disagreement as current gaps. PRISMA 2020 has 27 main items; the current RoB 2 version for individually randomised parallel-group trials is dated 22 August 2019.

Canonical boundary

This article owns reporting of the critical-appraisal process inside evidence synthesis under The Systematic Engine concept. PRISMA owns general systematic-review reporting, tool developers own the content of their instruments, and GRADE or another appropriate framework owns certainty judgements when used.

Worked example

Every chapter returns to a fictional systematic review of school-based sleep-education programmes, with mixed randomised and non-randomised studies. It is fictional, low-stakes and educational; it does not report a real school, clinical decision or guaranteed outcome.

Warm up with eduKate’s guides to explaining scientific evidence and asking better questions in class.

CHAPTER 1 OF 16

1. Define the Appraisal Question

Back to contents

Why this move matters

State which result, study feature, effect or evidence body is being judged and for what review decision.

Open the judgement, not only the label

State which result, study feature, effect or evidence body is being judged and for what review decision. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: The team appraises the effect of assignment to a sleep programme on attendance, not every outcome in each paper. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

A narrow appraisal may not transfer to another result. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Define the Appraisal Question”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 2 OF 16

2. Match Study Design to an Appropriate Tool

Back to contents

Why this move matters

Name the design and justify the selected instrument instead of forcing unlike studies into one checklist.

Open the judgement, not only the label

Name the design and justify the selected instrument instead of forcing unlike studies into one checklist. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: RoB 2 is used for eligible randomised results and a suitable non-randomised tool for the observational evidence. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

A recognised tool can still be misapplied. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Match Study Design to an Appropriate Tool”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 3 OF 16

3. Report Tool Name, Version and Variant

Back to contents

Why this move matters

Give the exact version, date, variant, effect of interest and any approved adaptations or translations.

Open the judgement, not only the label

Give the exact version, date, variant, effect of interest and any approved adaptations or translations. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: The report identifies the 2019 RoB 2 version and the effect of assignment to intervention. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

A current webpage may later change, so archived materials matter. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Report Tool Name, Version and Variant”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 4 OF 16

4. Pre-Specify How Appraisal Will Be Used

Back to contents

Why this move matters

Explain before seeing results whether judgements affect eligibility, synthesis grouping, sensitivity analysis, certainty or interpretation.

Open the judgement, not only the label

Explain before seeing results whether judgements affect eligibility, synthesis grouping, sensitivity analysis, certainty or interpretation. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: High-risk results remain visible but are excluded in a prespecified sensitivity analysis. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Plans may require justified amendments. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Pre-Specify How Appraisal Will Be Used”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 5 OF 16

5. Train and Calibrate Reviewers

Back to contents

Why this move matters

Report guidance, examples, pilot records, calibration threshold and what changed after discussion.

Open the judgement, not only the label

Report guidance, examples, pilot records, calibration threshold and what changed after discussion. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: Two reviewers pilot five studies and revise their interpretation notes before full appraisal. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Agreement after training does not prove validity. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Train and Calibrate Reviewers”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 6 OF 16

6. State Reviewer Number and Independence

Back to contents

Why this move matters

Identify who judged each record, whether work was independent and who adjudicated unresolved differences.

Open the judgement, not only the label

Identify who judged each record, whether work was independent and who adjudicated unresolved differences. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: Two reviewers complete judgements separately before comparing answers. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Nominal independence can be weakened by shared drafts or hierarchy. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “State Reviewer Number and Independence”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 7 OF 16

7. Preserve Supporting Evidence and Rationale

Back to contents

Why this move matters

Link every judgement to page, table, protocol, registry or author response and explain how that evidence answers the tool.

Open the judgement, not only the label

Link every judgement to page, table, protocol, registry or author response and explain how that evidence answers the tool. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: A missing-outcome judgement cites the registry outcome and the report section where it disappears. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Absence of reporting is not always proof of absence in conduct. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Preserve Supporting Evidence and Rationale”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 8 OF 16

8. Resolve Disagreement Transparently

Back to contents

Why this move matters

Report whether disagreement concerned facts, tool interpretation, effect definition or judgement threshold and how it was resolved.

Open the judgement, not only the label

Report whether disagreement concerned facts, tool interpretation, effect definition or judgement threshold and how it was resolved. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: One reviewer reads baseline imbalance as chance; the other links it to a randomisation concern, and the adjudicator records the reasoning. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Consensus can conceal a power imbalance. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Resolve Disagreement Transparently”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 9 OF 16

9. Build Domain-Level Bias Profiles

Back to contents

Why this move matters

Present domains and overall judgement without collapsing informative patterns into one numerical quality score.

Open the judgement, not only the label

Present domains and overall judgement without collapsing informative patterns into one numerical quality score. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: Selection and missing-data concerns remain visible even when both studies receive an overall high-risk label. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Profiles can still overwhelm readers without explanation. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Build Domain-Level Bias Profiles”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 10 OF 16

10. Stratify Evidence Without Double Counting

Back to contents

Why this move matters

Explain grouping, exclusion, weighting or modelling decisions and avoid mechanically treating ordinal labels as numeric weights.

Open the judgement, not only the label

Explain grouping, exclusion, weighting or modelling decisions and avoid mechanically treating ordinal labels as numeric weights. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: Primary synthesis includes all eligible studies; sensitivity analysis removes a prespecified high-risk subset. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Sensitivity analyses may reduce precision and change the question. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Stratify Evidence Without Double Counting”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 11 OF 16

11. Connect Appraisal to Sensitivity Analyses

Back to contents

Why this move matters

State the exact rule, analysis, result and interpretation rather than saying findings were robust without evidence.

Open the judgement, not only the label

State the exact rule, analysis, result and interpretation rather than saying findings were robust without evidence. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: The pooled estimate changes materially when studies with missing-data concerns are removed, and both results are shown. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Stability across one analysis does not eliminate other biases. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Connect Appraisal to Sensitivity Analyses”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 12 OF 16

12. Calibrate Certainty at the Evidence-Body Level

Back to contents

Why this move matters

Describe how appraisal patterns inform certainty alongside inconsistency, indirectness, imprecision and other relevant domains.

Open the judgement, not only the label

Describe how appraisal patterns inform certainty alongside inconsistency, indirectness, imprecision and other relevant domains. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: Risk-of-bias concerns contribute to a transparent certainty judgement for attendance outcomes. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

A study-level label is not the same as certainty in a body of evidence. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Calibrate Certainty at the Evidence-Body Level”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 13 OF 16

13. Guard Interpretation Against the Strongest Limitation

Back to contents

Why this move matters

Match verbs and confidence to the least secure decisive link and distinguish association, estimate and causal inference.

Open the judgement, not only the label

Match verbs and confidence to the least secure decisive link and distinguish association, estimate and causal inference. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: The conclusion says programmes may improve attendance under studied conditions rather than prove universal benefit. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Cautious wording cannot repair a wrongly framed review. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Guard Interpretation Against the Strongest Limitation”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 14 OF 16

14. Report Automation or AI Assistance

Back to contents

Why this move matters

Name tool, version, inputs, outputs, validation sample, human checks, errors and whether automation influenced judgements.

Open the judgement, not only the label

Name tool, version, inputs, outputs, validation sample, human checks, errors and whether automation influenced judgements. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: An AI assistant highlights passages, but reviewers make and document every final judgement. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Human review can still anchor on automated suggestions. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Report Automation or AI Assistance”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 15 OF 16

15. Disclose Missing Information and Author Contact

Back to contents

Why this move matters

Report questions sent, dates, replies, non-response and how unavailable information affected judgements.

Open the judgement, not only the label

Report questions sent, dates, replies, non-response and how unavailable information affected judgements. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: Authors confirm allocation details but cannot recover the analysis plan. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

A reply may be incomplete or retrospective. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Disclose Missing Information and Author Contact”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

CHAPTER 16 OF 16

16. Conclude With an Appraisal Decision Trail

Back to contents

Why this move matters

Summarise tool fit, profiles, disagreements, synthesis consequences, certainty and guarded wording with links to full records.

Open the judgement, not only the label

Summarise tool fit, profiles, disagreements, synthesis consequences, certainty and guarded wording with links to full records. Name the review question, appraisal tool and version, domain, signalling evidence, reviewer judgement and rationale.

Trace one appraisal into the synthesis

Worked example: Readers can move from the final claim back to every decisive appraisal rationale. Follow the judgement through calibration, disagreement, sensitivity analysis, certainty assessment and conclusion wording.

Protect the boundary between tools

PRISMA reports the review, RoB 2 or another appropriate instrument supports a particular assessment, and certainty frameworks address a body of evidence. One label should not silently perform all three jobs.

Respect the development boundary

Transparent reporting does not guarantee correct judgement. The Systematic Engine reporting guideline is under development. This guide explains the registered four-function model and current tools without inventing a finished checklist.

Student, parent and career transfer

Make one evidence card for “Conclude With an Appraisal Decision Trail”: source location, plain-language meaning, limit and next question. A parent or teacher can ask for one everyday explanation and one specialist explanation; both must preserve the same conditions. This habit supports evidence synthesis, medicine, public health, education research, policy analysis, librarianship, peer review, statistics and scientific communication without promising a grade, admission, funding decision or job.

PRACTICE LAB

Build a Systematic Engine decision trail

Student route

1. Choose one original report inside this guide’s scope.
2. Copy its objective and underline actor, action, context, evidence and intended decision.
3. Draw the specific reasoning chain used in this article.
4. Mark each step reported, inferred, contested or unknown.
5. Rewrite the conclusion so it preserves the narrowest important limitation.

Parent or teacher route

Ask the learner to explain how English helps us make every critical-appraisal judgement traceable from the source evidence to the final review conclusion twice: once to a classmate and once to a method specialist. Praise a precise correction and a well-stated unknown; both show disciplined attention to evidence.

Early-career route

Create a one-page Systematic Engine audit with source link, date checked, scope, method logic, decisive wording, contrary evidence, limit and next action. A colleague should be able to continue without guessing.

Continue with eduKate’s guides to reading course and career information and giving useful feedback in group projects.

USEFUL NEXT READING

Connect this narrow owner to its neighbour

Read the PRISMA-TRAICE AI-assisted review guide for a related but different reader job. Return here when the question is specifically how to make every critical-appraisal judgement traceable from the source evidence to the final review conclusion. This boundary keeps one canonical owner per intent.

For English practice across research, school and career communication, visit the English at work and in public life guide.

FREQUENTLY ASKED QUESTIONS

Questions students and parents often ask

Is The Systematic Engine a final checklist?

No. EQUATOR lists the reporting guideline as under development.

Does it replace PRISMA 2020?

No. PRISMA reports the review; this project focuses on reporting the appraisal process.

Does it replace RoB 2?

No. RoB 2 is an assessment tool for specified randomised-trial results.

Should quality scores be averaged?

Domain-based profiles and rationales are usually more informative than an invented composite score.

Why report reviewer disagreement?

It shows where evidence or tool interpretation was uncertain and how the final judgement arose.

What is interpretive guarding?

Matching conclusion language to the strength and limitations of the appraised evidence.