Why Science? | FDA Adaptive Device Trials, Prespecified Changes and Simulations

FDA defines an adaptive medical-device study as allowing prospectively planned modifications based on accumulating study data without undermining integrity and validity. The protocol should identify which features may change, what information triggers the change and how the final analysis accounts for it. A change invented after observing comparative outcomes is not protected by the original design.

For an adaptive design changes a study by prospective rule, flexibility is scientific only when it is planned. FDA defines an adaptive medical-device study as allowing prospectively planned modifications based on accumulating study data without undermining integrity and validity. The protocol should identify which features may change, what information triggers the change and how the final analysis accounts for it. A change invented after observing comparative outcomes is not protected by the original design. Keep the guardrail beside the result: Prospective flexibility is different from outcome-driven improvisation.

A strong science note separates observation from interpretation. Write the modifiable feature, trigger statistic, threshold, permitted action and analysis consequence in one rule. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Next chapter

The FDA guidance is final, dated July 2016, and identified by docket FDA-2015-D-1439. It covers medical-device clinical studies and can apply from feasibility through pivotal development. FDA lists PMA, 510(k), De Novo, HDE and IDE submissions among its premarket contexts while excluding the specific combination-product situations stated in the guidance.

Think of the july 2016 device guidance is final as a decision rule written before the answer is known. The FDA guidance is final, dated July 2016, and identified by docket FDA-2015-D-1439. It covers medical-device clinical studies and can apply from feasibility through pivotal development. FDA lists PMA, 510(k), De Novo, HDE and IDE submissions among its premarket contexts while excluding the specific combination-product situations stated in the guidance. The interpretation stops here: Device guidance scope should not be silently extended to excluded development programmes.

A strong science note separates observation from interpretation. Create a source receipt with title, date, final status, docket, issuing centres and stated scope. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

A useful adaptation addresses a real uncertainty or efficiency goal: stopping for success or futility, resizing a study, changing allocation or selecting among prespecified options. The protocol should explain why the change helps and how it relates to the device-development question. Adding complexity without a decision benefit creates operational and statistical risk.

The cheerful promise of adaptation begins with a decision problem is a smarter trial, but every shortcut must preserve a trustworthy comparison. A useful adaptation addresses a real uncertainty or efficiency goal: stopping for success or futility, resizing a study, changing allocation or selecting among prespecified options. The protocol should explain why the change helps and how it relates to the device-development question. Adding complexity without a decision benefit creates operational and statistical risk. Remember: An available algorithm is not by itself a reason to adapt a study.

A strong science note separates observation from interpretation. State the development problem, the adaptive action and the expected benefit in separate sentences. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

An interim analysis examines accumulating data while the trial is ongoing. The plan should specify timing, data maturity, variables, analysis set and who receives results. Repeated looks at outcomes can alter error probabilities and behaviour, so the statistical method and information pathway belong in the design, not in an informal meeting note.

For interim analyses are information events, flexibility is scientific only when it is planned. An interim analysis examines accumulating data while the trial is ongoing. The plan should specify timing, data maturity, variables, analysis set and who receives results. Repeated looks at outcomes can alter error probabilities and behaviour, so the statistical method and information pathway belong in the design, not in an informal meeting note. Keep the guardrail beside the result: Unplanned looks can affect both inference and trial conduct.

A strong science note separates observation from interpretation. Draw the timeline from database cut through analysis, recommendation, action and documentation. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

A rule may include zones where expert judgement is needed, but the permitted actions and decision process should be clear. A reviewer should be able to take the same interim dataset and reconstruct why the study continued, stopped or changed. Ambiguous rules invite inconsistent application and selective choices.

Think of adaptation rules should be deterministic enough to audit as a decision rule written before the answer is known. A rule may include zones where expert judgement is needed, but the permitted actions and decision process should be clear. A reviewer should be able to take the same interim dataset and reconstruct why the study continued, stopped or changed. Ambiguous rules invite inconsistent application and selective choices. The interpretation stops here: Discretion that is not bounded prospectively can introduce bias.

A strong science note separates observation from interpretation. Give a second reader a hypothetical interim result and test whether they select the same allowed action. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Access to comparative interim results can influence enrolment, treatment, follow-up or endpoint assessment. An independent data monitoring group or firewalled statistical team may help keep investigators and sponsor personnel shielded from sensitive information. The protocol and charter should describe roles, access, communications and emergency unblinding.

The cheerful promise of blinding protects behaviour as well as statistics is a smarter trial, but every shortcut must preserve a trustworthy comparison. Access to comparative interim results can influence enrolment, treatment, follow-up or endpoint assessment. An independent data monitoring group or firewalled statistical team may help keep investigators and sponsor personnel shielded from sensitive information. The protocol and charter should describe roles, access, communications and emergency unblinding. Remember: A statistical adjustment cannot repair every operational bias caused by information leakage.

A strong science note separates observation from interpretation. List each role and mark which aggregate, blinded or unblinded information it may receive. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

A prospectively defined futility rule may stop a study when success appears unlikely, reducing participant and resource exposure to an unpromising programme. The rule can be binding or nonbinding depending on the plan, but its operating behaviour and consequences should be evaluated. Reasons for overriding a recommendation must be documented.

For stopping for futility can conserve resources, flexibility is scientific only when it is planned. A prospectively defined futility rule may stop a study when success appears unlikely, reducing participant and resource exposure to an unpromising programme. The rule can be binding or nonbinding depending on the plan, but its operating behaviour and consequences should be evaluated. Reasons for overriding a recommendation must be documented. Keep the guardrail beside the result: Futility stopping is not proof that the device has no effect in every context.

A strong science note separates observation from interpretation. Specify the predictive or conditional quantity, threshold, action and interpretation of a futility stop. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Early success rules can shorten a trial when evidence crosses a prespecified boundary. Repeated opportunities to declare success can increase false-positive risk unless the design accounts for them. The effect estimate may also be exaggerated at an early stop, so reporting should include uncertainty and the actual stopping path.

Think of stopping for success needs error control as a decision rule written before the answer is known. Early success rules can shorten a trial when evidence crosses a prespecified boundary. Repeated opportunities to declare success can increase false-positive risk unless the design accounts for them. The effect estimate may also be exaggerated at an early stop, so reporting should include uncertainty and the actual stopping path. The interpretation stops here: Crossing a boundary does not remove estimation bias or clinical-meaning questions.

A strong science note separates observation from interpretation. Simulate early-success frequency, false-positive probability and effect-estimate bias under several truths. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

A design may reassess nuisance parameters or treatment effects and adjust sample size under prespecified rules. Blinded approaches can limit bias when feasible; unblinded approaches require stronger information controls. Maximum sample size, timing and analysis method should be set before the reassessment.

The cheerful promise of sample-size adaptation responds to information, not hope is a smarter trial, but every shortcut must preserve a trustworthy comparison. A design may reassess nuisance parameters or treatment effects and adjust sample size under prespecified rules. Blinded approaches can limit bias when feasible; unblinded approaches require stronger information controls. Maximum sample size, timing and analysis method should be set before the reassessment. Remember: Increasing enrolment after a disappointing trend without a valid rule can distort inference.

A strong science note separates observation from interpretation. Write the reassessment statistic, minimum and maximum sample sizes, and circumstances where no change occurs. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Response-adaptive randomisation may shift future allocation using accumulating outcomes, while other adaptations may drop an arm or select a dose or device configuration. These changes can interact with calendar time, site learning and population drift. Simulations should examine whether treatment groups remain comparable and whether enough information is collected for each question.

For allocation changes affect experience and comparability, flexibility is scientific only when it is planned. Response-adaptive randomisation may shift future allocation using accumulating outcomes, while other adaptations may drop an arm or select a dose or device configuration. These changes can interact with calendar time, site learning and population drift. Simulations should examine whether treatment groups remain comparable and whether enough information is collected for each question. Keep the guardrail beside the result: More participants receiving the apparently better option does not guarantee a less biased comparison.

A strong science note separates observation from interpretation. Model a time trend and test how the allocation rule changes balance and estimation. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Adapting an endpoint, subgroup or eligibility criterion can be especially sensitive because the current data may suggest where results look favourable. If such changes are contemplated, options, rationale, information restrictions and multiplicity handling should be rigorous. Device learning curves and user experience may also alter outcomes over time.

Think of endpoints and populations should not drift casually as a decision rule written before the answer is known. Adapting an endpoint, subgroup or eligibility criterion can be especially sensitive because the current data may suggest where results look favourable. If such changes are contemplated, options, rationale, information restrictions and multiplicity handling should be rigorous. Device learning curves and user experience may also alter outcomes over time. The interpretation stops here: Post hoc subgroup selection is exploratory, even when biologically appealing.

A strong science note separates observation from interpretation. Predefine candidate populations or endpoints and the evidence allowed to guide selection. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Complex adaptive designs often lack simple formulas for all relevant properties. Simulation can reproduce recruitment, outcomes, missingness, interim decisions and final analysis under many scenarios. Code should implement the protocol exactly and be verified; scenario grids should include null, target, poor and misspecified conditions.

The cheerful promise of simulation is the laboratory for the design is a smarter trial, but every shortcut must preserve a trustworthy comparison. Complex adaptive designs often lack simple formulas for all relevant properties. Simulation can reproduce recruitment, outcomes, missingness, interim decisions and final analysis under many scenarios. Code should implement the protocol exactly and be verified; scenario grids should include null, target, poor and misspecified conditions. Remember: A simulation validates the programmed design behaviour, not the truth of its assumptions.

A strong science note separates observation from interpretation. Create a scenario matrix crossing treatment effect, event rate, missingness and adaptation timing. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Important summaries can include false-positive probability, power, expected sample size, stopping probabilities, bias and precision. Conditional performance after each adaptation may also matter. Reporting a single favourable power number hides how the design behaves elsewhere, especially under prior-data conflict or slow endpoints.

For operating characteristics make trade-offs visible, flexibility is scientific only when it is planned. Important summaries can include false-positive probability, power, expected sample size, stopping probabilities, bias and precision. Conditional performance after each adaptation may also matter. Reporting a single favourable power number hides how the design behaves elsewhere, especially under prior-data conflict or slow endpoints. Keep the guardrail beside the result: Good average performance can conceal unacceptable behaviour in a critical scenario.

A strong science note separates observation from interpretation. Choose five operating characteristics and justify why each matters to participants or the regulatory decision. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

An adaptation is only as dependable as the data available at the interim cut. Rapid entry, endpoint verification, query resolution, device accountability and code validation reduce the risk of acting on incomplete or incorrect information. The plan should address late-arriving outcomes and corrections after a decision.

Think of infrastructure must deliver timely reliable data as a decision rule written before the answer is known. An adaptation is only as dependable as the data available at the interim cut. Rapid entry, endpoint verification, query resolution, device accountability and code validation reduce the risk of acting on incomplete or incorrect information. The plan should address late-arriving outcomes and corrections after a decision. The interpretation stops here: Fast data are not useful when their definitions or quality are unstable.

A strong science note separates observation from interpretation. Build an interim-readiness checklist with data latency, cleaning, lock and verification criteria. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Adaptive features can affect many parts of a device submission. Early engagement can focus questions about estimands, simulations, information control, type I error, clinical meaning and reporting. A concise package should contain the protocol concept, rules, scenario results and specific questions rather than an unexplained software output.

The cheerful promise of early fda discussion can clarify a complex plan is a smarter trial, but every shortcut must preserve a trustworthy comparison. Adaptive features can affect many parts of a device submission. Early engagement can focus questions about estimands, simulations, information control, type I error, clinical meaning and reporting. A concise package should contain the protocol concept, rules, scenario results and specific questions rather than an unexplained software output. Remember: Feedback on a plan does not predetermine the final regulatory decision.

A strong science note separates observation from interpretation. Prepare three questions whose answers would change the design and attach the evidence needed to discuss them. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

Reporting should show planned rules, interim dates, data available, recommendations, actions, deviations and final analysis. It should distinguish simulations done before the trial from sensitivity analyses added later. Readers need to know not only the design family but the exact path the study followed.

For the final report should reconstruct the realised path, flexibility is scientific only when it is planned. Reporting should show planned rules, interim dates, data available, recommendations, actions, deviations and final analysis. It should distinguish simulations done before the trial from sensitivity analyses added later. Readers need to know not only the design family but the exact path the study followed. Keep the guardrail beside the result: Naming a trial adaptive does not disclose which adaptations actually occurred.

A strong science note separates observation from interpretation. Create a chronological decision table linking every interim output to the authorised action and audit record. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter · Next chapter

The receipt records protocol and charter versions, adaptation rules, interim schedules, access roles, simulation code and seeds, scenarios, operating characteristics, data cuts, decisions, deviations, final estimand and analysis. This makes the flexibility inspectable and supports later learning without rewriting history.

Think of a complete adaptive-trial receipt preserves validity as a decision rule written before the answer is known. The receipt records protocol and charter versions, adaptation rules, interim schedules, access roles, simulation code and seeds, scenarios, operating characteristics, data cuts, decisions, deviations, final estimand and analysis. This makes the flexibility inspectable and supports later learning without rewriting history. The interpretation stops here: Documentation supports auditability; it cannot legitimise an unplanned data-driven change.

A strong science note separates observation from interpretation. Give the receipt to an independent statistician and ask them to reproduce both a simulated scenario and the realised decision path. Record the source, version or date, population or device context, method, result and one uncertainty. If the official record does not show a field, write “not shown” instead of filling the gap from memory.

Contents · Previous chapter