When experiments are expensive, good AI does not need to pretend it already knows the answer. Bayesian optimisation learns from completed measurements and chooses which real test could be most useful next. Here is how the reasoning works, how to check the arithmetic and how to avoid the traps that make an apparent winner unreliable.
Did you know? Choosing a next experiment and predicting its result are different tasks. One is a decision under budget and uncertainty; the other is a forecast. This guide keeps them separate and gives worked, clearly hypothetical cases.
On this page: The short answer: how to choose the next expensive test · The problem Bayesian optimisation actually solves · Four moving parts: design space, observations, surrogate and acquisition · What a Gaussian-process surrogate says—and what it does not · How acquisition functions balance exploration and exploitation · Why a valid budget includes time, cost and the cost of being wrong · Worked example: choosing three candidate tests with arithmetic you can check · Noise, replicates and the difference between a winner and a lucky measurement · How to build a defensible initial experiment set · What happens after one real experiment is completed · Choosing several tests at once without wasting the batch · Safe and constrained optimisation is a different job · How to tell whether the optimisation strategy is helping · When it goes wrong: symptoms, causes and repairs · Transfer problem: the goal switches from finding a peak to meeting a reliability floor · Limitations: when this method should not be your first tool · Questions readers often ask · Reader pathways, related mechanisms and primary sources
- Define a real objective and feasible settings before fitting a model.
- Use only completed experiments as observations; keep predictions separate.
- Compare acquisition scores under the actual cost and safety rules.
- Confirm apparent winners with repeated, independent measurements.
The short answer: how to choose the next expensive test
Did you know that some machine-learning problems are not about predicting a label at all? They are about deciding which experiment is worth running next, when every experiment costs money, time, material or human attention. Bayesian optimisation is a strategy for searching for good settings of an expensive, unknown objective. It builds a probabilistic approximation of results already measured, uses that approximation to weigh promising candidates against uncertain ones, runs a carefully chosen real test, and updates its beliefs. The model does not replace the experiment. It helps decide which experiment earns a scarce slot.
A practical starting rule is to write down the quantity you genuinely want to maximise or minimise, its units, the permitted settings, the cost of a trial, and what counts as an invalid or unsafe result. Then obtain a few reasonably diverse measurements, fit a surrogate model, inspect its predictive uncertainty, choose a candidate by an acquisition rule, test that candidate under controlled conditions, and repeat until the budget or decision threshold is reached. Preserve all attempted settings and outcomes, including failures and timeouts. The decisions are only as defensible as this experimental record.
This article is for scientists tuning a laboratory process, engineers comparing designs, educators planning costly intervention pilots and machine-learning teams allocating hyperparameter trials. The method can be useful in all four settings, but the meaning of a good experiment differs. You cannot assume a classroom intervention behaves like a chemical process just because both have numbers. Nor does choosing a promising next point guarantee a global optimum; it is an informed way to spend tests, not a magical discovery engine.
The problem Bayesian optimisation actually solves
Imagine that measuring one configuration takes six hours. There are ten adjustable choices, and some are continuous. A grid of just five values for each would contain 5 to the power of 10 combinations, or 9,765,625 experiments. At six hours each, that is beyond a sensible development plan. Blindly choosing a small subset saves time but may miss effective areas. Asking an ordinary regression model for its highest predicted score may be equally poor, because the prediction can be most confident where evidence is abundant and wrong where it is sparse.
Bayesian optimisation attacks a different question: given what has been measured, how much do we expect to learn or gain from testing a particular untested setting? The algorithm represents uncertainty about an objective function. A candidate is attractive because it might outperform the current best, because it could resolve important uncertainty, or because it helps under several objectives and constraints. These competing reasons are visible in an acquisition function, which scores where to spend the next measurement.
The phrase ‘Bayesian’ points to updating beliefs after evidence, not to any claim that one specific probabilistic model is mandatory in every implementation. The classic arrangement uses a Gaussian process as the surrogate. Other probabilistic or uncertainty-aware models can be appropriate for large, mixed, discontinuous or structured spaces. The phrase ‘optimisation’ also needs care: in many research uses the true optimum is unknown and cannot be proved from a handful of evaluations. We usually seek a practically good, reproducible design within a budget.
There is a useful distinction between optimising a mathematical expression that can be evaluated millions of times and optimising a black box that requires real equipment or an expensive simulation. For the former, conventional numerical methods may be faster and more precise. For the latter, test selection can dominate project cost. Bayesian optimisation should be compared with sensible baselines such as random search, experienced experimental design and domain-specific optimisation, not with a straw-man search that nobody would actually run.
Four moving parts: design space, observations, surrogate and acquisition
The design space states what a valid candidate means. A variable might be a temperature between two bounds, a mixing time, a discrete treatment, a model learning rate or the width of a component. Units belong in the record. A temperature of 180 degrees Celsius is not interchangeable with 180 kelvin, and a learning rate of 0.001 does not have the same geometry as a setting of 0.1 if the variable is meaningfully explored on a logarithmic scale. Feasibility is a property of the candidate, not an afterthought after an attractive prediction.
An observation joins one candidate with a measured outcome and the circumstances of the measurement. Include timestamp, instrument or software version, replicate, operator-independent metadata where appropriate, and explicit failure state. A missing measurement is not zero performance. An invalid run cannot safely be treated as a low-quality but valid measurement unless the objective explicitly defines that penalty and the design is reviewed. If a setting was never actually executed, it is a proposal, not an observation.
The surrogate is the inexpensive model that predicts the outcome and uncertainty at candidate points. It is fitted on completed experiments, not imagined results. Prediction uncertainty can be larger where experiments are sparse, but this is not automatic evidence of calibrated uncertainty. A confidently misspecified model can make terrible suggestions. Diagnostics, held-out checks where feasible, repeat measurements and scepticism about extrapolation are part of operating the surrogate.
The acquisition function turns surrogate information into the next decision. It might prefer high predicted reward, high possibility of beating the incumbent, large expected improvement or informative uncertainty. It can also account for candidate batches, noisy outcomes, constraints and differing evaluation costs. The actual objective is expensive; the acquisition function is deliberately cheap enough to optimise many times between real tests. Keeping these two levels separate prevents a common conceptual mistake: the model’s preferred point is not yet a measured achievement.
What a Gaussian-process surrogate says—and what it does not
A Gaussian-process model describes a joint distribution over possible function values at sets of input points. In approachable terms, it gives a predicted mean and a predictive spread for each candidate and links nearby candidates through a chosen covariance rule, often called a kernel. After a new measurement, the joint distribution is conditioned on that observation, changing both the mean and uncertainty. The covariance rule expresses assumptions about smoothness and relatedness before the data can settle them.
Suppose one dimensional setting is the dwell time of a process. A kernel that treats nearby dwell times as correlated is plausible if small changes produce small outcome differences. The same assumption can fail if a physical threshold triggers an abrupt regime change. Length-scale choices matter: a model with a very long characteristic length scale may smooth away a narrow improvement; one with an implausibly short length scale may treat almost every point as unrelated. These are modelling judgments, not hidden truths produced by mathematics.
Noise changes the story. A measured yield of 81 per cent may reflect both the candidate setting and instrument variation. Repeated tests under nominally identical settings can reveal how much of the apparent difference is reproducible. A Gaussian process with observation noise should not treat every small measured increase as a certain gain. The predicted spread has components driven by model structure, sampling and assumed noise; none protects against all unmeasured sources of bias such as operator changes or an altered raw material.
Gaussian processes are particularly appealing when evaluation count is modest and input dimension is manageable. Their computation becomes harder for large datasets, and their assumptions can be unsuitable for mixed categorical choices or highly discontinuous functions. Practitioners should treat the surrogate as an instrument requiring checks: plot predicted versus measured values, inspect residuals by region, note calibration problems and revisit the representation of variables before trusting elegant-looking acquisition plots.
How acquisition functions balance exploration and exploitation
Exploitation means testing where the model predicts strong performance. Exploration means testing an uncertain area that could reveal something better. The trade-off is not a slogan; it is the reason Bayesian optimisation differs from repeatedly picking the highest predicted mean. A candidate with a modest mean but a wide credible range might have meaningful upside. Another with a high mean but tiny uncertainty may be a sound choice when the remaining budget is small and a reliable improvement is more valuable than discovery.
Probability of improvement asks how likely a candidate is to beat a chosen benchmark. It is easy to explain but ignores the size of the gain once improvement occurs. A candidate with a 90 per cent chance of improving by a tiny amount can outrank one with a 40 per cent chance of a transformative improvement. Expected improvement instead averages the positive amount by which a candidate would beat the benchmark under its predictive distribution. A lower-confidence, high-upside point can therefore be worth testing.
Upper-confidence-bound-style rules combine a mean and an uncertainty term with an exploration parameter. Increasing the uncertainty weight typically encourages exploring less-known regions. Such rules are heuristic scores rather than forecasts of realised value. Expected improvement likewise depends on the model’s distribution and the reference incumbent. If the model is overconfident or the recorded incumbent is a noisy fluke, the score can mislead. The right response is to improve the experiment record and assumptions, not to worship a different acquisition acronym.
For joint batches, choosing the individually highest-scoring points one at a time without accounting for their correlation can waste slots on near-duplicates. Monte Carlo batch acquisition methods consider possible joint outcomes, including overlap in what different candidates would teach. The official BoTorch acquisition-function guide explains analytic and Monte Carlo formulations and the difference between choosing a single point and a batch. An acquisition score guides a decision; it is not a guarantee of improvement.
Why a valid budget includes time, cost and the cost of being wrong
An experimental budget is more than the number of completed trials. A setup that costs ten minutes is not equivalent to one requiring an overnight machine booking, specialist review or production downtime. If candidates have differing costs, the acquisition rule may consider expected improvement per unit cost, a constrained cost-aware objective or a sequential schedule that values early information. A simple ratio can be a useful teaching device, but it is not universally optimal because some information has value beyond the immediate gain.
Consider a company with one calibration rig and two researchers. A candidate might consume four hours of rig time and one hour of analysis; another might take one hour but require a scarce expert all day. The scarce resource may not be the same for both candidates. Record capacities separately when necessary. Do not collapse labour, safety risk and calendar deadlines into one arbitrary number merely because a software library accepts a scalar. Decide which resources are hard constraints and which may be traded.
Experiment design also includes the cost of being wrong. An unsafe operating temperature cannot be justified by high predicted performance; it should be excluded or approached only through a dedicated reviewed safety process. An educational intervention with uncertain effect on vulnerable learners may require ethics approval and safeguards that a hyperparameter experiment does not. In a manufacturing system, a failed run can destroy material or equipment. These facts change the acquisition problem before the mathematics begins.
Finally, distinguish a stopping budget from a success threshold. A team can run out of tests without finding an adequate setting. Report that outcome openly. Conversely, it can find a sufficiently good, stable setting before using every allocated trial. A stop condition might combine a validated performance threshold, uncertainty about the incumbent, an economic value of another trial, safety requirements and remaining time. A search should not continue merely because the spreadsheet has unused rows.
Worked example: choosing three candidate tests with arithmetic you can check
Here is a deliberately simplified illustration, not a claim about an actual laboratory or a fitted Gaussian process. Our current best measured score is 70 units, and we want a higher score. We have three proposed candidates and a rough two-outcome predictive picture. Candidate A has an 80 per cent chance of a 2-unit improvement and a 20 per cent chance of no improvement. Candidate B has a 35 per cent chance of a 6-unit improvement and a 65 per cent chance of no improvement. Candidate C has a 15 per cent chance of a 10-unit improvement and an 85 per cent chance of no improvement. All other predicted outcomes are deliberately omitted to make the expectation transparent.
Expected improvement means summing probability multiplied by positive gain. For A it is 0.80 times 2 plus 0.20 times 0, which equals 1.60 units. For B it is 0.35 times 6 plus 0.65 times 0, equalling 2.10 units. For C it is 0.15 times 10 plus 0.85 times 0, equalling 1.50 units. If every test cost the same and these forecasts were defensible, B has the largest expected immediate improvement. Notice that A has the highest probability of improvement, while C offers the biggest possible gain. None is automatically best without saying what the goal is.
Now introduce prices: A costs $60, B costs $120 and C costs $40. A’s simple expected-improvement-per-dollar ratio is 1.60 divided by 60, approximately 0.0267 units per dollar. B’s ratio is 2.10 divided by 120, or 0.0175. C’s is 1.50 divided by 40, or 0.0375. A cost-ratio heuristic would rank C first. That is a different decision because we changed the objective from improvement per test to improvement per dollar. The arithmetic gives a clear explanation of why.
Suppose the equipment can safely run only A and B, because C violates a maximum temperature. Then C is not a candidate, however attractive its ratio. Suppose instead all are allowed, but a remaining budget of $100 permits A or C, not B. With a single trial, compare A’s 1.60 expected units with C’s 1.50; A wins on total expected immediate gain under these simplified assumptions. With a requirement to avoid more than a 50 per cent chance of no improvement, only A qualifies. Every change is explainable because the feasibility, budget and decision rule are explicit.
Real Bayesian optimisation would derive distributions from a surrogate, often with continuous outcomes; expected improvement would integrate positive gains across those distributions. Our small example isolates the decision mechanism. It does not justify treating hand-made probabilities as calibrated predictions or claiming the value per dollar ratio is an optimal policy for multiple sequential experiments. Future information and interaction between trials can change the ranking, particularly when one test would reduce uncertainty about several other candidates.
Noise, replicates and the difference between a winner and a lucky measurement
Suppose configuration X was measured once at 84 and configuration Y once at 82. The temptation is to declare X the winner. If ordinary repeat-to-repeat variation is approximately plus or minus five units, that comparison does not establish a reliable ordering. The first task may be to repeat the two contenders, use paired experimental conditions or model measurement noise. A high observed maximum can be a selection artefact: search many candidates and some will look exceptional by chance even when their underlying performance is ordinary.
A simple fictitious check makes this concrete. Three repeated scores for X are 84, 78 and 81, giving a mean of (84 + 78 + 81) / 3 = 81. Three scores for Y are 82, 83 and 82, giving a mean of 247 / 3, or about 82.33. The apparent winner reverses when we replace one lucky measurement with repeated evidence. This does not prove Y is universally better; three observations are a small sample and testing order, drift and covariates still matter. It does show why the ‘incumbent’ used by an acquisition function needs a defensible definition.
If measurements are noisy, the current best observed value is not necessarily the current best inferred latent mean. Some noisy acquisition methods explicitly represent uncertainty about the objective and the incumbent. Others use a practical estimate such as posterior means at observed points. These details matter when someone copies a formula from a tutorial without reading the assumptions. The official BoTorch documentation distinguishes analytic, batch and noise-aware acquisition families; select the implementation that matches the observation process.
One more source of noise is not random at all: a trial performed after machine maintenance may be systematically different. Randomising trial order, including controls, recording operator-independent process conditions and checking drift can be more valuable than an elaborate model. If all ‘good’ results occurred in the newest batch of raw materials, update the experimental design before running five more supposedly optimal candidates. Bayesian updating is not a repair for biased measurement.
How to build a defensible initial experiment set
An acquisition function needs observations to start from. For a modest search space, an initial design can distribute test points across the permitted region rather than crowding them around the first guess. Random or space-filling designs can reveal ranges and gross trends before the surrogate starts making fine recommendations. Use actual scientific constraints to rule out unsafe or impossible settings first. An evenly spaced numerical grid is not automatically space-filling when one variable should be represented on a logarithmic scale.
When some choices are categorical, avoid pretending the labels form a meaningful continuous number line. Encoding ‘steel’, ‘ceramic’ and ‘polymer’ as 1, 2 and 3 can create an invented distance between materials. Either choose a model that handles categories appropriately or create representations grounded in known properties. When inputs include strong physical relationships, consider transforming variables in ways that respect units, feasible ranges and conservation laws. Data cleaning must preserve the connection between the exact conditions used and the measured response.
The initial set should also expose known failure modes. Include at least one safe reference condition, perhaps replicated, so the team can tell whether the apparatus is behaving normally that day. Define a policy for cancelled, out-of-spec or instrument-failed tests. If the surrogate sees these as blank rows and the optimiser quietly drops them, it can keep suggesting the same impossible region. Some failures reveal constraints; others reveal only that the measurement failed. Those two meanings must not be conflated.
People often ask how many initial points are enough. There is no universal number: it depends on dimension, smoothness, noise, available prior knowledge and model complexity. With a tiny budget, avoid consuming most experiments just to make a beautifully uniform starting design. With strong heterogeneity, a tiny cluster can mislead any surrogate. Plan an initial allocation, a sequential allocation and a reserve for validation at the end. A selection algorithm should earn trust by outperforming a reasonable baseline under the same total budget.
What happens after one real experiment is completed
The iteration is easy to describe and easy to implement badly. First verify that the candidate was executed exactly as approved. Confirm the instrument settings, units, timestamps and outcome definition. Next append the result to an immutable experiment record with a distinct trial identifier. Then update the surrogate using the completed evidence, calculate candidate acquisition scores under current constraints, choose the next suggestion and record why it was selected. Only after approval should a physical or operational test begin.
Do not append the model’s forecast as if it were the measured outcome. Keep columns for predicted mean, uncertainty, acquisition score, observed result and validity status separate. The forecast is a trace of the reasoning available before the test; the observed result is new evidence. This separation makes it possible to analyse whether the model was calibrated, whether selection became too greedy and whether particular kinds of proposed trials failed disproportionately.
If a trial times out, the first job is to establish whether it ran and whether its outcome can be recovered. Retrying can duplicate an expensive or dangerous action. In a cloud simulation, a timeout may hide a completed billable computation; in a lab it may hide a physical process that still needs to be stopped safely. Assign one trial identifier to the intended operation, check the equipment or job state, and recover its result when possible. This is operational reliability, not a footnote to Bayesian statistics.
Sequential experimentation is also an organisational workflow. Record the criterion used for each candidate, the human who approved any safety-relevant run, and changes to the feasible space. Someone auditing the project later should be able to reconstruct why trial 17 was chosen using only information that existed after trial 16. If a model was retrained using future results in a retrospective evaluation, the apparent efficiency is misleading. For that related issue, see the guide to data leakage and honest validation.
Choosing several tests at once without wasting the batch
Parallel experimentation changes the mathematics. Suppose three test rigs become free simultaneously. Selecting the highest one-point acquisition score three times can choose almost identical settings because the surrogate has not yet seen the outcomes of the first two. A batch acquisition function considers the joint value of trying several candidates, which may favour a more diverse set. Correlation matters because two nearby settings often teach much of the same thing.
For instance, if one candidate is predicted to yield 85 and another is a near-duplicate with almost the same prediction, the team might prefer to spend one slot on a different regime that could reveal a new peak. The trade-off is not always diversity for its own sake. If safety limits and deadlines make a robust narrow search preferable, near-neighbour confirmation may be justified. Choose the batch using a real objective that reflects the purpose of the run.
BoTorch represents candidate batches as collections of points and supports Monte Carlo acquisition functions over joint posterior outcomes. Joint sampling captures dependence between candidate outputs rather than pretending each point is independent. That is one reason adding together the top three individual expected-improvement scores is not generally the right batch objective. Consult the current BoTorch acquisition reference for supported formulations and implementation caveats instead of assuming a one-line formula remains unchanged across library versions.
A practical batch schedule needs capacity rules too. If two candidates require the same unique fixture, they are not truly parallel even if two rigs exist. If results arrive at different times, an asynchronous policy should recognise running or pending experiments, rather than treating them as untested free slots. Preserve which candidates were chosen together and which observations were available at batch selection time. Otherwise a retrospective analysis may accidentally give the optimiser information it did not possess.
Safe and constrained optimisation is a different job
A constraint may be known before testing, such as an upper temperature limit, a legal material restriction or a classroom timetable. Such candidates should be removed from the domain. Other constraints are measured only after a test, such as whether a component survives stress or a formulation passes a toxicity screen. These uncertain constraints require special care: ordinary expected improvement on performance alone will not account for the chance of a harmful or unusable outcome.
A simplified constrained acquisition might discount the prospect of improvement by the probability that a candidate is feasible, but only when the modelling assumptions justify that calculation. Dependence between performance and safety can break naive multiplication. A high-temperature setting might simultaneously boost yield and increase equipment damage risk. Modelling these outputs independently could understate the hazard. Safety-critical work needs physical bounds, expert review and procedures that do not rely on a surrogate to tell the truth at the boundary.
Multiple objectives add another layer. A candidate may improve speed but worsen energy consumption or product quality. One aggregate weighted score conceals who set the weights and what trade-offs are allowed. Sometimes a Pareto frontier is more honest: it shows alternatives for which improving one objective tends to worsen another. In other cases a hard minimum on quality plus a single cost objective reflects the decision better. The mathematical method cannot settle normative choices about acceptable risk and stakeholder priorities.
Another kind of constraint is the need for reproducibility. If a recipe is valid only on a rare instrument configuration, it may be inferior to a slightly lower-scoring option that works across operators or facilities. Robust optimisation may explicitly value performance under perturbations, not just at one nominal point. That shift changes which experiments you should run. Name the context and uncertainty you care about before interpreting the model’s notion of ‘best’.
How to tell whether the optimisation strategy is helping
There is an awkward evaluation problem: you cannot know the outcomes of every experiment you chose not to run. Comparing the observed best score of two strategies from two different, uncontrolled projects does not identify which strategy was better. Where possible, use benchmark functions or previously collected full experiment tables, then simulate sequential selection without allowing later information into earlier decisions. Where real-world costs permit, run comparable randomised or paired evaluation campaigns.
Useful evaluation quantities include best validated score after a fixed number of trials, improvement over an initial baseline, total expenditure to reach a target, probability of violating constraints, repeatability of the selected setting and calibration of predictive intervals. Report actual completed tests, not just suggestions generated. If one method discarded half its failures and the other counted them, the comparison is not fair. Separate the search phase from a final confirmation phase when repeated testing is needed.
An illustrative project might produce candidate scores of 76, 79 and 85 at three search steps, then see confirmation repeats of 81, 80 and 82 for the last candidate. The observed search maximum was 85, but the confirmation mean is (81 + 80 + 82) / 3 = 81. The baseline must also be confirmed under comparable conditions. A team should not compare an optimistic maximum from many trials with a conservative average from one baseline and announce a precise percentage gain.
For models trained on expensive evaluations, the temptation to reuse every data point is understandable. But tuning the surrogate, choosing acquisition parameters and picking the final winner on the same small set of observations can produce overconfidence. Record tuning choices and a final validation plan before declaring success. The distinction parallels early stopping and honest model selection: progress inside the search is not the same as independently supported performance afterward.
When it goes wrong: symptoms, causes and repairs
Symptom one: the system repeatedly recommends almost the same unsuccessful setting. Possible causes include an acquisition function that overexploits, a kernel with unrealistic smoothness, duplicate observations marked as independent or an unexplored parameter boundary. Check the actual distances in the transformed feature space, inspect uncertainty maps, and compare recommended points with the observed results. Consider revising the surrogate and reserving a deliberate exploratory run. Do not force variety if the best evidence genuinely supports a narrow region; diagnose before changing policy.
Symptom two: predicted uncertainty looks tiny, yet test results regularly fall far outside it. The model may be misspecified, measurement noise underestimated, environmental covariates omitted or observations inappropriately pooled. Review residuals by instrument, time, setting and batch. Repeat sentinel conditions. The response should be to recalibrate uncertainty, improve the data representation or switch modelling assumptions, not merely make the acquisition function less adventurous. Exploration based on false uncertainty numbers is still false precision.
Symptom three: the method reports a spectacular improvement that operators cannot reproduce. Inspect whether the ‘winner’ was a single lucky extreme, whether experiment conditions shifted, whether a simulation result was confused with physical performance or whether invalid runs were silently filtered. Repeat candidate and baseline under controlled, ideally blinded or randomised conditions when practicable. Report the confirmed distribution and decision criterion rather than the most flattering value. Where the objective is high-stakes, obtain independent verification.
Symptom four: the schedule consumes all capacity but yields no decision. Often the metric, stop rule or constraints were underspecified. Perhaps every run reduced model uncertainty but the team needed a minimum practical improvement by Friday. Re-estimate the value of information versus the cost of another trial, reserve validation capacity and establish an explicit ‘no acceptable configuration found’ outcome. A useful optimiser supports decisions; it is not a justification for endless experimentation.
Transfer problem: the goal switches from finding a peak to meeting a reliability floor
Suppose we originally maximised average energy output, and our process model preferred a setting giving a predicted mean of 103 units with substantial variance. Mid-project, a customer changes the requirement: at least 98 units must be achieved on 95 per cent of valid runs, with no operation above a defined safety limit. The old acquisition score was built to find the largest expected output. It is now solving the wrong problem. Merely changing the article title or reusing the previous ‘best’ point would make a poor engineering decision.
A first approximation, if predictive outcomes were genuinely normally distributed with mean and standard deviation representing future run-to-run variability, would require mean minus roughly 1.645 standard deviations to be at least 98 for a one-sided 95 per cent threshold. For candidate P with mean 103 and standard deviation 4, the lower quantity is 103 − 1.645 × 4 = 96.42, which fails. For candidate Q with mean 101 and standard deviation 1.5, it is 101 − 1.645 × 1.5 = 98.5325, which passes the illustrative statistical screen. Higher mean is not enough; variability and the decision threshold have changed.
This numerical screen is not a safety certification. A surrogate’s uncertainty about its mean is not automatically the same as the distribution of future unit-to-unit outcomes. Model misspecification, heavy tails and changing materials can invalidate the normal approximation. The new plan needs repeated production-relevant tests, independent constraint validation and perhaps a model of reliability rather than performance alone. In the decision record, explicitly replace ‘maximise mean output’ with ‘find a feasible design satisfying a lower-tail target under approved safety bounds’.
The transfer lesson is general. When an educational pilot changes from maximising average test gain to preventing any learner group from falling below an acceptable outcome, the objective and ethics change. When a model deployment changes from maximising accuracy to meeting a latency ceiling, the acquisition function changes. Bayesian optimisation can respond to new goals only after humans specify them honestly and gather evidence sufficient for the new question.
Limitations: when this method should not be your first tool
If each objective evaluation costs almost nothing, exhaustive or conventional optimisation might be simpler. If the design space has hundreds of largely irrelevant variables, a small Gaussian-process model can struggle to represent it without strong structure. If the response is extremely discontinuous or dominated by hidden categorical regimes, smooth-kernel assumptions may actively mislead. If the experiment process changes faster than the model learns, old data may cease to describe the current system.
Bayesian optimisation is also not automatically causal. A surrogate can model relationships between settings and outcomes, but it cannot isolate the effect of a setting if the actual process assigns settings in a confounded manner. For example, if hotter recipes were always run by a more experienced operator, comparing outcomes can mix the influences. Controlled experimental assignment, randomisation or sound causal design matters when the decision requires causal interpretation. Good trial selection cannot rescue an invalid experiment.
Optimisers can also exploit measurement loopholes. If a scoring pipeline awards high values to a technically malformed run, searching harder can produce more malformed winners. Validate the measurement system, not just the optimiser. A machine may find an extreme that is outside the domain where safety tests were performed; a recommender may learn to increase clicks while making users worse off. Define guardrails and evaluate real outcomes with stakeholders.
Finally, the method cannot manufacture more information than trials provide. With ten expensive experiments and a complicated objective, there may be several plausible explanations of the data. Report uncertainty and alternatives instead of claiming that a smooth acquisition plot demonstrates deep understanding. A useful result can be ‘these settings are promising and worth confirmation’ rather than ‘the globally optimal answer is known’.
Questions readers often ask
Is Bayesian optimisation the same as Bayes’ theorem? Not exactly. It often uses probabilistic updating, but its distinctive decision loop is selecting expensive evaluations by comparing expected gains and uncertainty. You can understand Bayes’ theorem and still need to learn surrogate modelling, experimental design and acquisition functions. Likewise, using a probabilistic model somewhere in a project does not automatically make the project Bayesian optimisation.
Does it always need a Gaussian process? No. Gaussian processes are a well-known choice because they offer a coherent predictive distribution and work well in many modest-dimensional, small-data settings. Other surrogates and acquisition formulations can be more appropriate for complex or mixed spaces. The crucial requirement is that the estimates used for experimental decisions are meaningful for the particular objective and uncertainty, and that the system is evaluated against sensible alternatives.
Will expected improvement always select the actual best candidate? No. Expected improvement is calculated under current model beliefs. If those beliefs are wrong, the preferred candidate may perform badly. It also optimises a chosen notion of gain, which may not represent safety, future costs or the value of acquiring information for a later decision. Confirm important results with real measurements.
Should I always spend all remaining experiments? No. Stop when further evaluation is unlikely to change the practical decision enough to justify its cost, or when the required target has been independently confirmed, or when the budget is exhausted. Record the reason. A disciplined stop is part of good optimisation, not an admission that a model failed.
How many repeats do I need? There is no universal fixed number. The answer depends on measurement variation, the size of the expected gain, consequences of a false winner and the decision standard. Plan repeats before seeing a spectacular result, especially when a small, noisy search is likely to overstate its top outcome.
Reader pathways, related mechanisms and primary sources
If you are a student, start by explaining the difference between the expensive objective and the cheap acquisition score, then reproduce the expected-improvement arithmetic in the worked example using a calculator. Try changing the budget and watching the decision change. That teaches more than memorising an acquisition acronym. The wider How AI Works introduction provides the surrounding concepts of model training, inference and human verification.
If you are an engineer, start with the measurement contract and the failure log. Check whether every suggestion can be executed safely, whether reruns duplicate physical actions, and whether the recorded incumbent survives controlled repeats. Only then compare surrogate choices and acquisition functions. For nearby technical ideas, study missing data and robust evaluation and probability calibration; the latter link should be checked against the live canonical article if reused elsewhere.
If you lead a research or education programme, begin with the decision problem: what counts as a worthwhile improvement, what changes must be reviewed, and who bears the downside if the optimum is wrong? Budget a final validation round and communicate uncertainty in ordinary language. See the education perspective in learning that transfers into life. Do not claim an intervention worked because the search model predicted that it might.
For technical grounding, read the primary BoTorch acquisition functions documentation, its models guide, and optimisation guide. These documents explain the machinery without turning numerical predictions into promises. Software APIs and recommended numerical implementations can change; check the documentation for the version you deploy. The enduring principle is simpler: spend expensive measurements where they are most likely to improve a clearly specified decision, and verify what actually happened.