Cross-path quarantine protects users while a shared delivery mechanism is repaired. Release is a separate engineering decision. A correct origin and one clean request do not prove that every resolver, cache, representation and promotion path now delivers the intended remote-sensing product.
This guide continues third-reopening containment and third-closure evidence. It explains how to define release units, promote a repair through representative canaries, compare each canary with a trusted route and retire the third-incident controls only after the failure mechanism has been exercised safely.
The method releases evidence, not confidence by assertion. Paths can return at different times, and residual quarantine remains visible until its own gate passes.
A practical orientation
Treat each combination of registry, resolver, redirect, cache, representation and client as a path. Group paths by shared mechanism, then choose canaries that expose the states most likely to hide stale or mis-keyed content.
The worked example follows four regional delivery paths and two representations through cold-cache, warm-cache, validation and update cycles.
- Define the release units — for incident and product owners.
- Design promotion canaries — for delivery engineers.
- Follow the regional example — for staged path release.
- Apply third-retirement gates — for accountable closure.
- Transfer to two CDNs — for independent infrastructures.
Chapter contents
Open a route that matches the decision in front of you, or read straight through. Each route returns to the same evidence chain.
Start with the operating question
Build the evidence chain
Design canaries around the hidden states
Work the example
Stress-test the decision
Set third-retirement gates beyond path release
1. Define path-level release units and residual quarantine
List the paths contained in the third incident: product, region, registry record, resolver, redirects, cache tier, representation and client class. Give each a release owner and evidence gate. A grouped quarantine can contain several units without requiring one all-or-nothing release.
Preserve the safe alternate route and the exact quarantined configuration. State what users receive during the test and how a path returns to quarantine if evidence fails.
- Intended product and version.
- Path components and negotiated representations.
- Trusted comparator and provenance.
- Canary traffic boundary.
- Automatic and manual rollback authority.
2. Freeze the repair and comparator before canary traffic
Record the repaired resolver, cache-key or promotion configuration and its deployment artifact. Verify origin and registry identity independently. The trusted route should retrieve the same intended product without relying on the suspect component.
Choose comparison fields that reflect identity and freshness: product identifier, acquisition or processing version, checksum where appropriate, metadata version, validators and age. Pixel equality is not always expected across representations, so define semantic equivalence before testing.
| Layer | Evidence | Failure revealed |
|---|---|---|
| Origin | Product and metadata identity | Wrong source artifact |
| Registry | Version and canonical target | Stale or divergent reference |
| Delivery | Status, redirects and path trace | Resolver or routing defect |
| Cache | Age, validator and key dimensions | Stale or collapsed representation |
| Client result | Decoded product identity and usable metadata | Negotiation or client-path defect |
3. Design canaries around the hidden states
Select canaries for cold cache, warm cache, stale entry, conditional validation, product update, redirect and relevant content negotiation. A canary that repeats only the easiest fresh-cache request cannot expose the mechanism that caused the incident.
Keep the cohort small enough for fast rollback but large enough to include the affected path classes. Promotion canary testing is an operational pattern; HTTP standards define the semantics of requests, responses, freshness and validators that the pattern must respect.
4. Use validators and cache evidence correctly
Record ETag or Last-Modified validators where supplied, Cache-Control directives, Age, Vary dimensions and the actual response identity. A 304 response proves that a validator comparison occurred; it does not by itself prove that the selected cache entry belonged to the intended representation.
When the failure involved cache-key collapse, send requests that differ on every relevant selection dimension. Preserve request headers and compare the received representation with the trusted product record.
| State | Test | Pass evidence |
|---|---|---|
| Cold | No stored entry | Correct origin identity and key creation |
| Warm | Stored fresh entry | Correct identity and expected Age behaviour |
| Stale | Entry requires validation | Validator targets correct representation |
| Update | New product version published | Old response no longer served beyond declared policy |
| Negotiated | Representation-selecting header changes | Vary and key dimensions keep variants distinct |
5. Promote one controlled change at a time
Move the repaired artifact through development, staging and production canary with the same version identifier. Avoid bundling unrelated resolver, redirect and cache changes; if the canary fails, the team should know which change to reverse.
Exercise rollback before broad promotion. Confirm that rollback restores the last verified configuration and does not reconnect a retired or stale path.
6. Worked example: four regions and two representations
A hypothetical service publishes product V9 as Cloud Optimized GeoTIFF and JSON metadata. Regions A and B use the repaired cache template; Region C was the original stale path; Region D uses a separate resolver but the same deployment bundle. The team sends one percent of eligible traffic to canaries and compares every response with the registry and trusted origin.
Cold and warm tests pass in A, B and C. During update from V9 to V10, Region D serves V9 JSON because its redirect target was excluded from the promotion inventory. Region D returns to quarantine; A through C continue through validation and negotiated-representation tests. The failure prevents global release but does not invalidate the independently verified paths.
| Path | Canary states | Result | Action |
|---|---|---|---|
| Region A | Cold, warm, stale, update, variants | All pass | Expand staged traffic |
| Region B | Cold, warm, stale, update, variants | All pass | Expand staged traffic |
| Region C | Original incident path; full suite | All pass after repair | Release with heightened watch |
| Region D | Separate resolver; shared bundle | V9 JSON after V10 update | Re-quarantine and repair inventory |
| Trusted origin | V9 then V10 verified | Correct comparator | Remain reference |
7. Expand traffic by evidence, not elapsed time alone
Increase from canary to limited, regional and normal traffic only after the required states pass. At each step, compare error, identity mismatch, validator behaviour and product-version distribution with the trusted route.
Hold long enough to encounter relevant expiry and update cycles. A quiet hour cannot stand in for a daily product update or a cache lifetime that has not elapsed.
8. Set third-retirement gates beyond path release
Path release means ordinary traffic can return under continued monitoring. Third retirement means the special incident controls can be reduced because the repair, promotion process, detection and rollback have all proved durable.
Retirement should be path-specific when evidence matures at different rates. Keep the incident map, repaired artifact, canary suite and residual restrictions accessible after the temporary bridge or alternate route is removed.
- Every released path passes its representative state set.
- Product and representation identity agree with the registry.
- Promotion inventory includes all relevant routes and variants.
- Rollback restores a verified configuration.
- Monitoring detects a safe replay of the old mechanism.
- Residual quarantines and owners are explicit.
9. Retain a low-burden watch after retirement
Continue sentinel checks for product version, cache age anomalies, variant mix and path-specific mismatch. Reduce frequency only after the relevant product-update and expiry cycles remain stable.
Reopening triggers should name the path and consequence. A single transient network failure is not automatically recurrence; a correctly delivered response with the wrong product identity is materially different.
10. Keep delivery proof inside its limits
The canary proves delivery behaviour for the tested products, paths, representations, cache states and client requests. It does not prove scientific accuracy for an untested sensor, atmospheric correction, spatial resolution or downstream analytical question.
Maintain provenance from the delivered object to scientific validation, but state the two claims separately. Delivery assurance can be complete while product fitness remains question-dependent.
11. Misconceptions and practical questions
Common mistakes include releasing all paths because one canary passes, testing only cold cache, treating a 304 response as product proof, forgetting negotiated variants, waiting for expiry without publishing an update and removing rollback before the watch period ends.
Must every quarantined path be released together?
No. Release each path or evidence group when its own mechanism and representative states pass, while other paths remain protected.
Is a one-percent canary always sufficient?
No. Percentage is secondary to coverage of the affected paths, representations, cache states and consequences. A small targeted cohort can be stronger than a large easy cohort.
12. Changed-context transfer, evidence check and sources
Transfer the method to two independent content delivery networks. Test shared registry and product identity, then exercise each provider’s cache, validation, purge and routing semantics separately. Do not infer one provider’s result from the other’s architecture.
RFC 9111 defines HTTP caching, freshness and validation. RFC 9110 defines representation semantics and validators, while RFC 9211 defines the Cache-Status field. These standards support the evidence model; canary scope and remote-sensing fitness still require system-specific validation.
Continue through the Gold Standard ecosystem
Return to the Remote Sensing subject root, the Gold Standard of Education parent, or the How X Works hub.
Immediate predecessor: third-reopening containment and third-closure evidence · Earlier guide: quarantine-retirement surveillance and third-reopening gates · Remote Sensing root
These routes are grouped by learning purpose: prerequisite model, immediate predecessor, and the wider transfer framework. They are not substitutes for the evidence and limits stated in this guide.