The Gold Standard of Websites | Separating Field Performance Data from Laboratory Test Results

Ask whether the decision concerns real-user experience, regression diagnosis, pre-release testing or a particular device class. One dataset rarely answers all four.

Write the comparison as a ledger: object, quantity, unit, boundary, time window, protocol and evidence. Mark every input as observed, stipulated, fitted or assumed. This catches a basis change before it reaches the conclusion.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘define the performance question’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Next chapter

A lab run represents the configured test agent and moment. A field dataset represents eligible visits actually observed under its collection rules. Write both populations before comparing values.

Draw the mechanism before interpreting the number. Put the stages in order, name where each observation is made and identify a competing explanation. The arrows matter only when their conditions remain visible.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘name field and laboratory populations’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Page-level data and origin-level data aggregate different sets of URLs. Verify that the field row and tested lab URL refer to the intended scope.

Use two checks that fail differently. Reconstruct an input from outputs, calculate a limiting case, inspect a control or repeat the result from the definitions. Agreement is stronger when both checks do not recycle the same hidden assumption.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘match url and origin aggregation’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

CrUX-style reports can use a rolling 28-day window, while a lab run belongs to one timestamp. A release made yesterday cannot dominate twenty-seven earlier days.

Keep the comparison fair. Hold the population, material, interval, protocol and unit constant before assigning a difference to the mechanism of interest. A changed condition is a changed context, even when the column heading looks familiar.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘record the collection window’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

The 75th percentile is a value at or below which roughly 75% of included experiences fall under the dataset’s method. It is not the mean and does not describe every user.

Separate measurement from decision. A value can be correctly calculated yet too local, uncertain or method-dependent for the intended action. Report the value, its conditions and the evidence needed before a stronger decision.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘understand a percentile as a distribution point’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

The three lab runs have a median of 2.0 seconds; the field p75 is 3.1. The numbers can coexist because the lab conditions and the field distribution differ.

Preserve order and identity. Equal totals can hide different mechanisms when observations occur in another sequence or come from different people, specimens, devices or sessions. Keep identities until the reader question no longer depends on them.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘work the 2.0 second median and 3.1 second p75 example’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

A median of three runs and a p75 from many visits use different estimators and sample designs. Put raw or distribution information beside each summary.

Test a deliberately changed case. Change one assumption while holding the others fixed, calculate again and explain why the conclusion moves or stays stable. Transfer is demonstrated when the reasoning survives the new context.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘do not average unlike summaries’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

CPU capability, memory, viewport and connection affect loading and interaction. Document laboratory throttling and inspect field form-factor or network dimensions when available.

State the boundary of inference beside the answer. Name the system, representation and interval to which the result applies, then name one tempting stronger claim it cannot establish.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘keep device and network conditions visible’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

A cold lab navigation may fetch resources that repeat visitors already cache. Conversely, some real users can face cleared caches, first visits or cache misses. Name the state.

Use uncertainty as structure. Identify the input most capable of changing the conclusion, vary it over a defensible range and show whether the practical choice changes. One linked sensitivity check is more useful than a generic caveat list.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘separate cold and warm cache states’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Latency, peering, CDN location and regional content can differ across visits. A single lab location should not be called globally representative.

Finish with an evidence package: source records, definitions, exclusions, software or instrument version, calculations, validation result and decision rule. A later reader should be able to reproduce the result and see what would cause revision.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘account for geography and delivery paths’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Consent banners, logged-in state, personalisation, scrolling and interaction can change measured elements or timings. Preserve the user journey with the number.

Write the comparison as a ledger: object, quantity, unit, boundary, time window, protocol and evidence. Mark every input as observed, stipulated, fitted or assumed. This catches a basis change before it reaches the conclusion.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘preserve user behaviour and page state’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Controlled runs are useful for reproducing a suspected bottleneck, comparing code variants and inspecting traces. Repeat runs and control background noise.

Draw the mechanism before interpreting the number. Put the stages in order, name where each observation is made and identify a competing explanation. The arrows matter only when their conditions remain visible.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘use laboratory runs for diagnosis’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Field distributions show actual experience among included users and help prioritise population problems. They often need additional RUM detail to explain why a metric moved.

Use two checks that fail differently. Reconstruct an input from outputs, calculate a limiting case, inspect a control or repeat the result from the definitions. Agreement is stronger when both checks do not recycle the same hidden assumption.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘use field data for experienced distributions’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Public field datasets cover eligible users, browsers and sufficiently represented pages under published rules. Absence of data is not proof of fast or slow performance.

Keep the comparison fair. Hold the population, material, interval, protocol and unit constant before assigning a difference to the mechanism of interest. A changed condition is a changed context, even when the column heading looks familiar.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘recognise field eligibility and coverage limits’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Use laboratory and staging checks immediately after a change, then annotate the deployment date and wait for enough aligned field observations before claiming population impact.

Separate measurement from decision. A value can be correctly calculated yet too local, uncertain or method-dependent for the intended action. Report the value, its conditions and the evidence needed before a stronger decision.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘test a deployment before the field window turns over’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Segments can expose a slow device or region, but small cells create unstable percentiles and privacy limits. Report counts or sufficiency rules where available.

Preserve order and identity. Equal totals can hide different mechanisms when observations occur in another sequence or come from different people, specimens, devices or sessions. Keep identities until the reader question no longer depends on them.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘segment without creating tiny unstable groups’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Traffic mix, campaigns, content and browser releases can change across periods. Align windows and segment material shifts before assigning causality to code.

Test a deliberately changed case. Change one assumption while holding the others fixed, calculate again and explain why the conclusion moves or stays stable. Transfer is demonstrated when the reasoning survives the new context.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘compare before and after with aligned traffic’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

A faster lab result with unchanged field p75 may reflect window lag, population mix, cache states or a change that did not affect the slow group. Each explanation suggests a different check.

State the boundary of inference beside the answer. Name the system, representation and interval to which the result applies, then name one tempting stronger claim it cannot establish.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘investigate disagreement as evidence’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Replace ‘Lighthouse is right and CrUX is wrong’ with a statement of each dataset’s scope. Disagreement is often the clue that the populations or conditions differ.

Use uncertainty as structure. Identify the input most capable of changing the conclusion, vary it over a defensible range and show whether the practical choice changes. One linked sensitivity check is more useful than a generic caveat list.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘repair the claim that one tool is wrong’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Save tested URL, commit, timestamp, lab settings, repetitions, traces, field source, scope, collection dates, eligibility, percentile, segments and deployment annotations.

Finish with an evidence package: source records, definitions, exclusions, software or instrument version, calculations, validation result and decision rule. A later reader should be able to reproduce the result and see what would cause revision.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘build a reproducible performance record’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Which dataset should guide real-user prioritisation? Use suitable field evidence for the experienced distribution and lab tests for controlled diagnosis.

Can one lab run be compared with field p75? It can be placed beside it, but it should not be treated as the same estimator or population.

Why does field data change slowly after a fix? A rolling window still contains visits from before the deployment.

Write the comparison as a ledger: object, quantity, unit, boundary, time window, protocol and evidence. Mark every input as observed, stipulated, fitted or assumed. This catches a basis change before it reaches the conclusion.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘answer focused reader questions’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Next chapter

Use the Google and Chrome sources for dataset boundaries, then follow the linked Websites branches for timing stages, caching and network setup.

Draw the mechanism before interpreting the number. Put the stages in order, name where each observation is made and identify a competing explanation. The arrows matter only when their conditions remain visible.

Apply the rule to the central fictional case, preserve unrounded inputs and label the result with its protocol. Then write one plausible alternative explanation and the observation that would distinguish it.

For Websites, use seven fields: target question, object or population, unit, boundary, protocol, result and decision limit. Complete every field before writing a conclusion. For ‘continue through verified website routes’, a plausible value can otherwise hide a changed condition or undocumented processing step.

Change one condition that matters while holding the remaining basis fixed. Recalculate or re-inspect the evidence, say what stays comparable and name the stronger conclusion that still does not follow.

Before closing the chapter, write a two-line result. The first line states what the evidence establishes under the named conditions. The second states the nearest tempting conclusion that remains unsupported. Ask another reader to identify the condition that would make the first line false. This turns a caveat into a usable check.

Contents · Previous chapter · Return to Websites