Why English Matters in Real-World Evidence Appraisal

Why English? • Real-world evidence appraisal

“Real world” describes a setting; it does not guarantee a reliable comparison.

RELEVANT helps readers test questions, groups, measures, analysis, results, interpretation and disclosed interests.

Why does English matter when students read real-world evidence? Routine records do not explain themselves. A database may show what was prescribed, coded, billed or recorded, but prose must establish whether those fields represent the treatment, health state or outcome named in the conclusion.

RELEVANT is the REal Life EVidence AssessmeNt Tool. A joint Respiratory Effectiveness Group and European Academy of Allergy and Clinical Immunology task force developed it for observational comparative-effectiveness research in asthma. The development paper describes 21 sub-items across seven domains: 11 primary and 10 secondary. The application paper and official page add context.

It is a quality-appraisal tool, not a reporting guideline. A reader judges whether the question, design, measures, analysis and interpretation are credible for an intended use. Complete reporting enables that judgement but does not guarantee a sound comparison.

The practical search intent—how to appraise real-world evidence, observational comparative effectiveness, confounding and medical research evaluation—reduces to one skill: distinguish a large dataset from a good comparison, and an adjusted association from a decision-ready conclusion.

How RELEVANT works

The tool uses yes-or-no judgements. For its guideline-development use, all 11 primary sub-items must be fulfilled before 10 secondary sub-items characterise further strengths and weaknesses. This staged logic prevents an attractive secondary feature from distracting from a missing critical element.

Primary does not mean that secondary items lack importance. Secondary items deepen appraisal of planning, data-source adequacy, sample-size reasoning, participant flow, uncertainty, missing data and bias. Turning the labels into a crude score would discard their purpose.

The seven domains are Background, Design, Measures, Analysis, Results, Discussion/Interpretation and Conflicts of Interest. They create a journey from question to comparison, measurement, adjustment, observation, meaning and disclosure.

Pilot raters initially interpreted some wording differently. The developers removed redundancy, replaced subjective language and split double-headed items. Clearer English improved agreement. The lesson transfers directly to school rubrics and workplace checklists.

A failed item may reflect weak methods or missing reporting. A responsible reviewer states the information gap without pretending to know which cause applies. That distinction is both honest and useful.

Twenty-one appraisal lenses

The labels below follow the final tool’s primary and secondary structure. The explanations translate them into student-friendly reading moves and do not replace the official checklist.

P1. Clearly stated research question

This lens asks about population, comparison, outcome and time. It matters because a broad purpose can make a favourable result look pre-planned. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to copy the exact question before reading results. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

Run a language check. Identify the population noun, evidence-state verb, comparison phrase, time marker and uncertainty qualifier. Separate the direct record from the adjusted estimate, the interpretation and the recommendation. When one confident sentence collapses these stages, restore the missing boundaries.

Finally, run a decision check. Explain what this item changes about confidence for a student, parent, clinician or programme manager. Name the additional information that would most reduce uncertainty. Criticism becomes constructive when it points to the next useful evidence.

P2. Population defined

This lens asks about eligibility, setting, index date and observation rules. It matters because labels can conceal exclusions that narrow representation. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to draw the source-to-analysis funnel. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P3. Comparison groups defined and justified

This lens asks about group entry and decision relevance. It matters because routine choices reflect need, access and preference. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to write why each person could enter each group. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P4. Exposure or treatment defined

This lens asks about start, continuation, switching and adherence. It matters because a prescription may be mistaken for actual use. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to separate the record from the behaviour. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P5. Primary outcomes defined

This lens asks about event, threshold, instrument and window. It matters because better control can hide several measures. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to build an outcome dictionary. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P6. Potential confounders addressed

This lens asks about factors affecting exposure and outcome. It matters because a long covariate list can omit the key factor. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to draw a causal map. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P7. Groups compared at baseline

This lens asks about starting differences before follow-up. It matters because the final estimate can hide initial imbalance. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to read baseline characteristics first. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P8. Results presented completely

This lens asks about all endpoints, denominators and relevant confounders. It matters because selective emphasis can promote a secondary result. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to match each planned question to a result. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P9. Consistency with known evidence explained

This lens asks about agreement or disagreement with earlier work. It matters because agreement may be treated as proof and disagreement ignored. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to compare populations, measures and designs. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P10. Clinical relevance discussed

This lens asks about absolute effect, uncertainty, harms and feasibility. It matters because statistical significance may be practically trivial. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to translate estimates into decision consequences. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

P11. Conflicts and funding stated

This lens asks about relationships, roles and publication control. It matters because a funder may be named without explaining influence. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to separate the interest from safeguards. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S1. Background literature reviewed

This lens asks about what is known and why another comparison is needed. It matters because real-world can be used as a slogan instead of a gap. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to summarise known evidence before the new result. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S2. A priori design evidenced

This lens asks about protocol registration and analysis planning. It matters because observed data can reshape the apparent question. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to separate planned from exploratory analyses. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S3. Population justified

This lens asks about why the chosen people fit the decision. It matters because definition does not itself establish relevance. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to connect eligibility to the intended user. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S4. Data source adequate

This lens asks about whether exposures and outcomes are captured. It matters because millions of records cannot replace an absent key variable. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to list required and available fields. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S5. Setting justified

This lens asks about why the care environment suits the question. It matters because organisation can influence treatment and outcome. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to name transferable and local features. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S6. Sample size or power pre-specified

This lens asks about reasoning linked to the primary outcome. It matters because large samples make tiny differences precise. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to judge importance as well as precision. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S7. Participant flow reported

This lens asks about all exclusions and selection stages. It matters because losses can silently change the population. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to reconstruct every stage and reason. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S8. Statistical uncertainty described

This lens asks about intervals and estimation context. It matters because a point estimate invites false certainty. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to keep the interval beside the estimate. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S9. Missing data reported

This lens asks about extent and handling by variable and group. It matters because missingness can differ systematically. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to map missing exposure, confounder and outcome data. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

S10. Bias and confounding discussed

This lens asks about remaining threats and affected conclusions. It matters because a generic limitations paragraph reveals little. A professional-looking paper should not receive an automatic “yes”; the relevant evidence must be visible in its words, tables or figures.

The practical move is to state direction, likely size and the claim qualified. Then write one of three judgements: “fulfilled because…”, “not fulfilled because…” or “unclear because the paper does not report…”. Quote the supporting location. This makes the appraisal auditable rather than personal.

Worked example: two routine-care treatments

Imagine a healthcare-record study comparing treatment A with treatment B. The abstract reports fewer hospital visits among people receiving A. The sample is large, the adjusted result is statistically significant and the authors call A more effective. RELEVANT asks whether that conclusion survives structured appraisal.

First fix the question. Which patients, treatment definitions, comparator, hospital-visit outcome and follow-up? If many outcomes were examined but visits became central only after analysis, the focus may be result-driven. Registration or an analysis plan helps distinguish prior intent from exploration.

Now examine design. Perhaps A is newer and prescribed to younger patients with fewer conditions, while B is reserved for difficult disease. That is confounding by indication. The data source must contain enough severity information. A million records cannot compensate for one absent key confounder.

Then inspect measures. A prescription shows an order, not necessarily medicine use. A hospital visit reflects access and coding as well as health. Exposure windows, switching, adherence and outcome codes need definitions. Otherwise a convenient proxy becomes the harder concept in the conclusion.

Baseline comparison reveals measured imbalance. Adjustment can balance recorded covariates but cannot guarantee control of unrecorded severity, preference or behaviour. Results need absolute rates, relative effects, intervals, planned outcomes and missing-data information.

The discussion should compare prior evidence, consider bias and explain clinical relevance. Funding and relationships need roles. A balanced conclusion says A was associated with fewer recorded visits after measured adjustment while residual confounding and exposure misclassification limit causal interpretation.

Did You Know? Realism and causal certainty are different strengths