Did you know that a webpage can say it was updated today while its actual school-admissions advice has not changed? An AI research agent needs to know when to revisit sources, but timestamps, sitemaps, HTTP validators and content hashes describe different things. Source change monitoring is the disciplined process of discovering new or revised documents, checking whether their material content changed and updating affected answers without claiming that a fresh crawl automatically proves fresh facts.
The practical method is to identify canonical source URLs, store approved representations and validators, use HTTP conditional requests where supported, interpret 200, 304 and error responses correctly, track genuine last-modified evidence, detect removed or redirected pages and recheck the exact claims that depend on a source. This article works through a 120-URL fictional monitoring ledger and a changed-context forty-page education exercise. All counts are invented, not real eduKate crawl results or evidence that artificial superintelligence exists.
Begin with the How Super Intelligence Works series. Safe URL Fetching explains permitted network destinations; Material Change Detection explains the consequences of a verified edit. The specialist question here is how the agent efficiently discovers and revalidates source revisions without inventing change or missing current evidence.
Choose the best reading route
- Checking current school sources? Begin with validators, 304 and material changes.
- Building a monitoring agent? Study canonical URLs, sitemaps, redirects and source versions.
- Learning Mathematics? Verify the 120-URL and forty-page worked cases.
- Maintaining education content? Follow dependency checks, effective dates and approved updates.
Contents
- 1. The source record and its monitoring metadata are separate
- 2. HTTP validators reduce unnecessary transfers
- 3. ETag and Last-Modified have different properties
- 4. 304 Not Modified has no new source body
- 5. 200 OK means a representation arrived, not that meaning changed
- 6. Lastmod in sitemaps is a discovery hint
- 7. Sitemaps have limits and incomplete coverage
- 8. The monitor needs a material-change threshold
- 9. Redirects and missing pages require distinct status
- 10. Do not misuse conditional write headers
- 11. One invented one-hundred-and-twenty-URL laboratory
- 12. Report representation evidence, not guessed policy validity
- 13. A forty-page school-notice transfer
- 14. Source revisions can change while an agent is paused
- 15. Robots, access and reasonable crawl policy
- 16. Diagnose missed and falsely announced updates
- 17. A practical source-monitoring release checklist
- Frequently asked questions
- Primary source documentation
- Related engineering guides
1. The source record and its monitoring metadata are separate
A school’s current notice is the underlying information people need. Its URL, retrieval time, HTTP headers, sitemap entry and search-engine listing are metadata about finding or transferring that information. A timestamp alone does not establish that the content actually changed, and a successful fetch alone does not establish that the page supports this year’s education decision.
Give each monitored source a stable logical identity, expected issuer, canonical URL, relevant student cohort, latest verified representation and applicable source-effective date. A content hash or ETag can help recognize differences in bytes, while the human or machine review must decide which differences matter. The more consequential the advice, the stronger this separation should be.
A public source may be updated for formatting, navigation or a copyright footer without changing its eligibility rule. Alternatively, two small digits in the deadline may change a family’s next action. One modification indicator is not a complete materiality decision. The assistant must compare the relevant passages, not equate all changes with new policy.
Source monitoring itself requires permission. An agent should obey its approved destination list, access requirements and reasonable request bounds. A source containing private pupil records should not become publicly crawlable merely because its URL appeared in a model-generated task list. Network eligibility and information accuracy are separate trust boundaries.
Return to contents or continue with HTTP validators reduce unnecessary transfers.
2. HTTP validators reduce unnecessary transfers
HTTP supports validators describing representations, including ETag and Last-Modified. A client that previously fetched a source can include an appropriate conditional request to ask whether its stored representation still applies. This can save bandwidth and reduce duplicate parsing compared with downloading every document in full each time.
MDN’s HTTP Conditional Requests documentation describes `If-None-Match`, which can compare ETag validators for a conditional fetch, and `If-Modified-Since`, which uses a Last-Modified timestamp. The relevant HTTP method, validator strength and server support affect the outcome. A model should not invent an ETag or timestamp simply because a page has a visible updated date.
Store the fetched representation with its validators and request context. If the same URL serves different languages or other negotiated variants, caching and comparison must account for relevant `Vary` and request headers. A 304 response applies to the representation the conditional request actually validated, not necessarily every version of the page.
Conditional requests are an optimization and consistency signal under a server’s implemented contract. An incorrect server configuration may report stale validators, while a genuinely changed source may be served through an unexpected redirect or variant. Revisit actual source content and applicability where the user consequence warrants independent verification.
Return to contents or continue with ETag and Last-Modified have different properties.
3. ETag and Last-Modified have different properties
An ETag is an opaque validator supplied by a server for a representation. Its exact characters belong to the source, not the assistant. A weak ETag and a strong ETag have distinct semantics under HTTP, and a conditional GET’s cache validation does not necessarily require byte-for-byte identity. Do not interpret every ETag as a cryptographic hash or as proof the underlying document’s factual meaning is unchanged.
Last-Modified records a modification time under the server’s representation rules, but timestamps have precision and clock limitations. Two fast changes can occur within coarse timestamp granularity, or a server may produce metadata that does not accurately represent its source. Do not claim a Last-Modified check alone provides complete change discovery for every website.
When a client supplies both If-None-Match and If-Modified-Since, HTTP specifies precedence for If-None-Match under the applicable conditions. MDN documents this behaviour. The agent’s request adapter should follow the actual HTTP client contract rather than treating the two validators as independent votes in a majority decision.
Not all sites expose these headers usefully. A safe monitor can use a controlled fetch, normalize source content appropriately and compare a trusted local representation under defined conditions. It should label the method and its limitations. A model’s high confidence that a source ‘probably has not changed’ cannot replace missing validator evidence.
Return to contents or continue with 304 Not Modified has no new source body.
4. 304 Not Modified has no new source body
An HTTP 304 Not Modified response to a valid conditional GET or HEAD indicates the client can reuse the relevant stored representation under the server’s validator semantics. MDN notes that a 304 has no response body. A source-monitoring assistant should not treat an empty 304 body as an empty newly published school notice or as proof the page was deleted.
Reusing cached content requires keeping the correct earlier representation and the request context it belongs to. If that copy was missing, corrupted or outside the user’s current permissions, the application may need a fresh authorised fetch rather than generate a plausible replacement from memory. The ETag response alone does not recreate the source text.
A 304 also does not prove the source still governs a new school cohort. It validates a representation relative to the supplied condition, whereas factual applicability depends on the document’s actual issued and effective dates. An unchanged prior-year notice is still unchanged and still potentially inappropriate for current advice.
Track the observation precisely. The monitor can record that the source returned 304 with a verified stored version at a known time. It should not write ‘official policy reconfirmed for 2027’ unless the document itself supports that statement. Transport validation and factual authority are distinct claims.
Return to contents or continue with 200 OK means a representation arrived, not that meaning changed.
5. 200 OK means a representation arrived, not that meaning changed
A conditional request that does not match the current validator may return 200 OK with a current response body. That supports obtaining a new representation under the server’s semantics. It does not mean a consequential admission requirement changed. A page can re-render the same material under a different ETag because of templates, analytics snippets or other presentation details.
Compare the material regions relevant to existing claims. An agent tracking Mathematics subject selection might need the eligibility table, source-effective year and application window. A changed sidebar or footer may require no educational correction. A small modification inside the target table could trigger a deeper factual review or editorial approval.
Keep both the previous and newly observed source versions with appropriate provenance where policy allows. That permits a reviewer to inspect which sentences or rows changed and why the article was updated. It also helps a resumed agent avoid citing a mixture of the two versions as if they came from one coherent notice.
An apparent successful response can still be wrong for the task because the source redirects to a generic homepage or login screen. Validate final URL, issuer, response type and relevant content. A 200 is a network result, not an automatically trustworthy answer.
Return to contents or continue with Lastmod in sitemaps is a discovery hint.
6. Lastmod in sitemaps is a discovery hint
An XML sitemap can list canonical source URLs and an optional `lastmod` value. Google Search Central says it uses `lastmod` when it is consistently and verifiably accurate, for example when it matches significant updates observable on the page. An automatically rewritten lastmod value for unchanged content is not a truthful signal of a new document revision.
Google explains that significant updates can include main content, structured data or links, while a copyright date update generally should not count as a meaningful page change. The standards and search-engine practices distinguish modification metadata from a site owner’s wish to be crawled frequently. An agent should not change dates simply to claim freshness.
Sitemaps are discovery aids, not a guarantee that Google or another consumer crawled or indexed the page. A monitoring system can use a sitemap entry to choose an authorized source for checking, then independently retrieve and compare the actual page. The agent should not declare a new official source indexed just because it was listed in a sitemap.
Canonical URLs matter. Google recommends fully qualified absolute URLs in sitemaps and generally wants the URLs the site intends search engines to display. A model-generated relative path or a duplicate tracking URL should not become a new source identity without review. Preserve issuer, canonical identity and applicable source version separately.
Return to contents or continue with Sitemaps have limits and incomplete coverage.
7. Sitemaps have limits and incomplete coverage
Google’s XML sitemap documentation describes limits of 50,000 URLs or 50 MB uncompressed per sitemap file, with index files for larger collections. A monitor should handle pagination or sitemap indexes under the actual provider’s format. Reading the first part and assuming it contains the complete school source universe can miss later notices.
An XML sitemap may omit older pages, while an RSS or Atom feed often covers only recent entries. The absence of a page from a current feed does not prove it was deleted from the source’s official archive. Likewise, a new URL may be discoverable through normal links before the sitemap has been updated.
Use multiple authorized discovery paths where the task requires coverage: official sitemap, documented API, current navigation and a stable known source list. The goal is to find genuinely applicable evidence, not create an unrestricted crawler. Duplicate entries can be consolidated by canonical source identity while preserving separate real documents when they share similar titles.
Respect expected source volatility. An education policy may change infrequently, while a timetable can change close to an event. Choose a reasonable check policy and avoid repeatedly downloading unchanged pages just to make the dashboard count look active. Discoverability and source authority do not improve with needless traffic.
Return to contents or continue with The monitor needs a material-change threshold.
8. The monitor needs a material-change threshold
After a new representation is fetched, compare the actual content relevant to the user. A simple byte-level checksum reveals that something changed; a parser can extract key paragraphs or table rows; an independent claim checker can determine whether a published explanation still matches. Each check answers a narrower question and has its own false-positive and false-negative risks.
A minor punctuation correction in a general lesson may not require new human approval under a defined editorial policy. A different eligibility condition, recipient, public distribution flag or closing time can require new review. The Material Change Detection chapter provides the follow-on method.
Do not allow the model to decide retroactively which evidence counts as material merely because publication is convenient. Source impact rules should be established before the update and be testable with synthetic edits. A small changed integer can affect an examination pathway more than a long explanatory rewrite.
Maintain source-to-claim dependencies. If one official document changes a Chemistry entry rule, only articles making relevant claims may need immediate factual review. A source-monitoring system that raises alarms for every article on the site creates noise and reviewer fatigue without necessarily improving reader safety.
Return to contents or continue with Redirects and missing pages require distinct status.
9. Redirects and missing pages require distinct status
A source may move permanently, redirect temporarily, require authentication or return Not Found. These are different observations. An HTTP 301 or 308 may provide a candidate new location, while 404 may reflect a withdrawn document or a bad link. The monitor must still verify the actual final destination and official issuer before transferring trust.
Safe URL Fetching remains essential. A once-approved official host can redirect to a destination outside the permitted network set, and a hostname can resolve unexpectedly. The monitor should apply the network policy to each connection and redirect rather than assume a familiar source name makes every later link safe.
An inaccessible page does not prove the rule ceased to exist. It may be a temporary outage, a login requirement or a changed source directory. A responsible assistant can state that current evidence was unavailable and refer the user to a known official entry point without inventing the missing policy.
Track source deletion separately from fetch failure. A trusted deletion notice, removed canonical record or official replacement can justify retiring a source under the actual documentation. One failed GET is not sufficient to erase a linked document or silently remove a historical source from an educational article.
Return to contents or continue with Do not misuse conditional write headers.
10. Do not misuse conditional write headers
HTTP uses similarly named conditional headers for more than retrieval. `If-Match` can protect a write against an outdated representation and may yield 412 Precondition Failed, while `If-None-Match` on a conditional GET can support cache revalidation. These are related HTTP validator mechanisms but serve different business purposes.
An agent that sees 304 during source checking should not assume the user approved an update. Likewise, a 412 on a WordPress or document update can indicate the target changed since the original read, requiring a conflict-aware revision review. The headers do not replace current human authorization or source scope.
A safe source monitor primarily performs permitted reads and records evidence. If it also initiates edits to derived articles, the write step needs a separate approved target, expected version and independently observed result. Avoid giving a general crawler broad publishing credentials simply to make source refreshes automatic.
Keep the operations auditable with safe identities. A read validator, a material-diff decision, an editorial grant and a saved target receipt should be distinct records. A model’s summary can explain their relationship but must not merge them into an unsupported statement that every item was updated successfully.
Return to contents or continue with One invented one-hundred-and-twenty-URL laboratory.
11. One invented one-hundred-and-twenty-URL laboratory
Imagine Cedar Source Monitor checking 120 fictional URL records. Seventy-two return valid 304 responses matching a stored authorized representation. Twenty-two yield new 200 representations that require material comparison. Ten are newly discovered official sources with no earlier validator. Eight are refused because access or destination policy does not permit the fetch. Eight remain unresolved after timeouts or missing authoritative evidence.
The mutually exclusive counts total 72 + 22 + 10 + 8 + 8 = 120. Validated unchanged representations are 60%, revised fetches about 18.33%, new sources about 8.33%, correct policy refusals about 6.67% and unresolved cases about 6.67%. These artificial figures are not real crawler efficiency or eduKate search statistics.
Do not say that only twenty-two source documents changed in meaning. The twenty-two 200 responses are new representations for comparison; some might reflect unimportant changes. Ten newly found sources are not automatically relevant or authoritative for every student. A source monitor’s transport results require further factual and user-permission evaluation.
Likewise, a policy refusal is not a source failure to overcome by using another account. An assistant can legitimately lack access to a staff-only school record while still providing public information. The correct user outcome follows the authorized corpus and current official source, not a maximum number of fetched pages.
Return to contents or continue with Report representation evidence, not guessed policy validity.
12. Report representation evidence, not guessed policy validity
A useful monitoring table names unchanged validated, newly fetched, newly discovered, refused and unresolved. Each status has a next action. An unchanged representation can keep its stored source record; a new body needs material diff review; a discovered URL needs identity and relevance validation; an unresolved fetch needs bounded recovery.
Store the request URL’s canonical identity, validator, retrieval time, actual response status, source revision and appropriate representation identity. A general observability stream need not contain a full private document or an access token embedded in a URL. A qualified editor can inspect the source through an authorized route when needed.
One user question can depend on several sources, and one source can support several articles. The ledger unit here is monitored URL records, not unique families or completed education tasks. If the organisation wants task-level results, it must verify each final user-facing claim and source link after monitoring.
Unknown evidence should remain visible. A monitor that cannot establish whether a conditional request was answered by the intended server should not fabricate a 304 from the model’s memory. Clear gaps help staff repair networking or source trust rather than treat an attractive dashboard as evidence of completeness.
Return to contents or continue with A forty-page school-notice transfer.
13. A forty-page school-notice transfer
Transfer the method to a fictional school site with forty tracked pages. Twenty-four return valid unchanged representations, eight deliver new source bodies, four appear as newly discovered canonical notices, two are correctly refused because the user’s session lacks authority, and two remain unavailable. The totals reconcile: 24 + 8 + 4 + 2 + 2 = 40.
The unchanged share is 60%, new representations 20%, newly discovered sources 10%, protected refusals 5% and unresolved 5%. These chosen values make the arithmetic checkable. They are not actual monitoring outcomes or an estimate of how often official school information changes.
Now the institution changes a subject application closing date in one of the eight new representations. That change is material to families and should trigger a source-specific correction review. A second page changes only its copyright footer; it may not affect any educational claim. The same HTTP response class can therefore lead to different editorial actions.
Finally, a parent asks about a source that returns 304 but whose stored copy applies to last year’s cohort. The correct answer is not automatically the old notice, even though the representation was unchanged. Recheck effective scope and find a suitable current official source or state the limitation.
Return to contents or continue with Source revisions can change while an agent is paused.
14. Source revisions can change while an agent is paused
A research agent might retrieve an official notice, draft a long article and wait for approval. During that pause the source might publish a new deadline. A source-change monitor can mark dependent claims for revalidation before public release. The saved earlier source remains historical evidence but may no longer support a current-year claim.
Keep exact article or target IDs alongside source dependencies. If the original WordPress post is already present, an approved correction should update the canonical owner rather than create a second page with a similar title. A source event indicating new content is not proof that the article update actually completed.
An editor’s earlier approval may cover the original wording, not the revised deadline or a newly added distribution channel. Material changes and current permission require their own checks. A monitor that detects the new source must not become an unrestricted auto-publisher solely because the source is official.
After the approved update, inspect the saved target state and rendered content. A successful API response might leave the article as a draft or reflect concurrent edits. The user needs the correct live facts, not the monitor’s generated claim that it corrected everything.
Return to contents or continue with Robots, access and reasonable crawl policy.
15. Robots, access and reasonable crawl policy
A website may restrict certain fetch paths or require authentication. Monitoring should operate through authorized public routes or supported APIs, observing the site’s relevant access and crawl rules. The existence of a URL in a user message does not authorize the agent’s server to reach private networks or bypass login requirements.
A request policy should bound frequency, response size, redirects and retry attempts. An unavailable notice does not improve when the agent floods the source with the same conditional GET every second. The service can schedule checks according to actual source volatility and user consequence, with appropriate stop conditions.
Do not confuse an SEO sitemap with a general data-export licence. Google sitemap documentation concerns discovery and indexing signals; it does not grant a third-party agent access to protected records. A monitoring product must maintain its own source authorization and privacy rules even when URLs are publicly listed.
Source-change analysis can use a verified local archive for historical comparisons under the relevant access and retention policy. Keep private originals out of broadly shared prompt logs, and restrict who can inspect sensitive diff details. A source monitoring pipeline is still an information system with owners and lifecycle responsibilities.
Return to contents or continue with Diagnose missed and falsely announced updates.
16. Diagnose missed and falsely announced updates
A false negative occurs when material source content changes but the monitor fails to identify it. Possible causes include an inaccurate Last-Modified header, a broken dependency map, an untracked alternate URL, a source API gap or a parser that ignores the crucial table. Test these cases by modifying one controlled material fact at a time.
A false positive occurs when a monitor labels an irrelevant change as a new school rule. A new layout, regenerated ETag or footer update can make bytes differ without changing the claim. Use field-aware diff review and a defined materiality policy to avoid flooding editors with unnecessary corrections.
Another failure is wrong source identity after redirect. A familiar link might land on a generic homepage or an archived page. Check final destination, issuer, document title and effective context instead of treating any HTTP 200 as the target notice. A safe network connection can still deliver the wrong source for the user’s task.
Preserve regression cases for each defect with fictional URLs and notices. Verify the actual conditional request and stored result, then check the material claim and applicable source date. A model reporting ‘everything is up to date’ cannot substitute for this evidence chain.
Return to contents or continue with A practical source-monitoring release checklist.
17. A practical source-monitoring release checklist
Begin with a list of genuinely needed official documents, canonical URLs, source owners, current representations and validators. Define their student cohort and effective-date scope. Implement conditional GET or another supported source API and store the actual response status, representation identity and retrieval time.
Handle 304 by reusing a valid stored representation under the correct request context, 200 by validating and comparing new content, redirects by rechecking destinations, and failures by reporting explicit uncertainty. Discover new URLs through approved feeds or sitemaps, without treating listing as proof of authority.
Compare material claims and update only affected downstream articles under their own editorial approval rules. Use trustworthy source versions, target IDs and saved readbacks. If a page is current but its statement is ambiguous, provide a qualified reader explanation and route to the responsible issuer rather than guessing.
The durable lesson is that freshness is not a date stamped on a webpage. A reliable AI assistant distinguishes transport validation, actual source modification, factual applicability and the final user outcome. That discipline helps parents and students rely on real current evidence instead of an impressive but unsupported ‘updated today’ label.
Return to contents.
18. Validators must be matched to the correct representation
A single URL can sometimes serve content variants based on language, accepted encoding or other request headers. A stored ETag belongs to a particular selected representation under the server’s semantics, not automatically to every possible variant of that URL. The monitor should preserve the relevant request context and follow the caching rules that identify which saved representation can be reused.
Imagine an official site presenting the same notice in English and another language. A 304 from a conditional request for the English representation does not automatically verify an earlier cached translation from a different language or edition. The underlying facts may be shared, but each displayed source and translation can carry a different revision or validation path.
HTTP `Vary` tells caches which request-header dimensions affect the selected representation under the response semantics. A custom client should not ignore those dimensions and then treat its chosen old body as the current answer. If the correct stored representation is unavailable, fetch through a permitted route and state any remaining uncertainty.
Keep source identity separate from response variant. A logical school notice can have several authorized renderings, but its current applicable eligibility condition should still be traced to the responsible source. The agent must verify that its selected variant is both current enough and relevant to the user’s cohort before describing a present-day rule.
Return to contents for the source-monitoring route.
19. An unchanged page can become irrelevant to a new user task
Suppose a 2026 admissions notice returns 304 every day because its content truly has not changed. A parent then asks about the 2027 intake. The absence of change does not establish that the old notice governs the new year. The task’s effective-date question has changed even though the source representation did not.
Maintain a reference from each material claim to the source document and applicable time scope. A monitoring result can mark an old source representation unchanged while a new task is routed to a current policy source or an explicit uncertainty. The system should not automatically apply the latest retrieved page when the issuer has not stated which cohort it covers.
One source can be both historical evidence and inappropriate current advice. Deleting it from the local archive may make historical comparisons impossible, but presenting it without an effective date may mislead. A careful service stores versions with their status and uses a claim-specific source-selection rule.
Readers need the right answer, not a demonstration that an HTTP cache worked. The model can explain that a notice has not changed since the prior check, but it should qualify what that means for the user’s particular institution and year. Transport freshness is necessary for some evidence decisions and insufficient for others.
Return to contents for the source-monitoring route.
20. Revalidate old citations before public publication
An article draft may cite a source that was current when its research began. If a source monitor discovers a new representation before publication, the editor should examine whether the change affects any material claim. A small shift in a programme threshold can invalidate part of a finished article while leaving the surrounding explanation useful.
Use stable source IDs and relevant passages to identify affected content rather than regenerate everything by default. A status such as Source Updated is a prompt for factual comparison, not permission to rewrite or publish unapproved material. Any changed public text should follow the actual editorial and human authorization rules.
An external publishing tool may accept a corrected post update but then time out. A source-monitoring event cannot prove the update was saved live. Reconcile the original WordPress post ID, expected slug, content revision and status before saying the page is updated. Another create call can produce duplicate articles with conflicting current claims.
Good provenance keeps the chain visible: original school notice, observed HTTP source revision, material difference, approved article revision and verified final saved result. The model can summarize this chain for readers without promoting a plausible narrative into evidence of completed work.
Return to contents for the source-monitoring route.
21. Check source discovery beyond the first sitemap page
An official site can publish a sitemap index referencing several sitemap documents. A monitor that reads only the first one may miss entire subject categories. Google’s documentation describes sitemap index files for collections exceeding individual sitemap limits. The crawler should follow only relevant permitted entries under a bounded source discovery policy.
RSS and Atom feeds can expose recent content while omitting older still-current documents. One missing entry does not imply deletion. The monitor should use a stable canonical source inventory and reconcile changes from approved feeds or APIs rather than treat every scan as a complete source-of-truth export.
Do not trust the order of sitemap entries as a business priority. Google says sitemap URL order does not determine its crawling decision. An agent could accidentally prefer the last newly listed page even when an earlier official notice contains the governing subject rule. Source authority and query scope belong in a separate validation step.
Check discovered links for duplicates and canonical identities. Tracking parameters or old redirects can point to the same actual document, while similar titles can represent different school cohorts. Collapse only genuine duplicates under a transparent rule. Missing relevant sources can harm answer completeness even when the sitemap parser technically succeeded.
Return to contents for the source-monitoring route.
22. Treat clock and revision metadata cautiously
Last-Modified is a server-reported timestamp, not an independent record of every business edit. Applications may regenerate pages on a schedule or preserve modification times incorrectly during deployment. A monitoring service should not infer precise causal history from a timestamp if the underlying source or deployment system does not support that guarantee.
ETags may be weak or strong validators under HTTP. A weak validator can be suitable for some semantic caching comparisons without proving byte-for-byte identity. A client should follow the actual HTTP conditional request semantics and keep the stored representation needed for reuse. It should not label every ETag a cryptographic digest or treat its value as a signed official source revision.
An article can have several dates: original issue date, page modification date, policy-effective date and retrieval date. They answer different questions. A school requirement can become effective next year while the current source page was updated today. The monitor should preserve these fields independently rather than merge them into ‘updated at’.
Test clock-related failures with synthetic records whose timestamps are deliberately misleading but whose content and effective period are known. The desired result is to identify that observation metadata alone cannot settle the user’s current question. An honest held status is more useful than a confidently wrong deadline.
Return to contents for the source-monitoring route.
23. A revised 120-URL ledger demonstrates conservation
Return to the fictional Cedar Monitor. At first, seventy-two stored representations validated unchanged, twenty-two returned new content, ten were new sources, eight were blocked by policy and eight had unavailable evidence. They total 120 original source records. These are monitoring dispositions rather than claims about final student answers.
Suppose four of the eight unresolved sources are later fetched successfully and found to contain changed representations. The new-content category becomes twenty-six and unresolved becomes four. The total remains 72 + 26 + 10 + 8 + 4 = 120. The changed-content share is then 26 ÷ 120 ≈ 21.67%, but these are not necessarily twenty-six material policy updates.
Do not count four new fetch attempts as four new user requests merely because they occurred later. Keep the original source IDs and observation cutoff, then record the subsequent improved evidence as another snapshot. This prevents reporting metrics from growing merely because the system retried a timeout.
Source status is not final article state. A newly verified representation still needs relevance and materiality review before the publisher can claim that a corresponding education page is corrected. The monitor’s success is evidence collection, not a universal proof of downstream completion.
Return to contents for the source-monitoring route.
24. Monitor changes without creating excessive load
Many official sources change rarely. Repeated full-body downloads of identical pages can waste bandwidth and burden the issuer. Use conditional requests where supported, appropriate check intervals and bounded retry or backoff policies. One high-risk last-minute school event can warrant closer review than a stable evergreen Mathematics definition.
The monitoring plan should include rate limits, timeout behavior and network authorization. A source endpoint that returns 429 or refuses access is not permission to bypass its controls through a different credential. The assistant can explain that current verification is temporarily unavailable and preserve the last independently checked representation with an appropriate age label.
Do not assume the agent should check a source every second just because a publisher wants fresh SEO dates. Google says sitemap lastmod should reflect genuine significant modification, and sitemaps are not guarantees of immediate indexing. Inventing constant change creates noise rather than better evidence.
Measure useful outcomes: material revisions detected, source contradictions resolved, articles correctly updated and reader-facing claims still supported. Raw HTTP request count and percentage of 304 responses are valuable efficiency diagnostics but do not establish educational truth or indexing quality.
Return to contents for the source-monitoring route.
25. A practical acceptance test for monitored school documents
Build a synthetic source server with known ETag, Last-Modified, content revisions and redirect behavior. Provide one unchanged response eligible for 304, another genuinely changed response with a new body, and a deliberately misleading timestamp. The controller should preserve and reuse the correct cached representation and detect the actual material difference.
Add a new official notice in a sitemap index and a duplicate URL using a tracking parameter. The monitor should discover the new canonical source once and avoid treating the duplicate as separate authoritative evidence. Add a staff-only page to ensure the protected fetcher refuses access without exposing its content.
Finish with a material change that affects a published article’s deadline. The source monitor should record the relevant difference and route it through the appropriate review, then independently check the target after an approved update. A successful conditional fetch should not automatically trigger unsupervised publication.
Retain failed tests and the source metadata that explains them. After a parser, HTTP client or model update, rerun those cases against actual simulated responses. A model-generated statement that the sources were checked is not enough; the protected test should verify the request, stored representation and final relevant user outcome.
Return to contents for the source-monitoring route.
- 18. Validators must be matched to the correct representation
- 19. An unchanged page can become irrelevant to a new user task
- 20. Revalidate old citations before public publication
- 21. Check source discovery beyond the first sitemap page
- 22. Treat clock and revision metadata cautiously
- 23. A revised 120-URL ledger demonstrates conservation
- 24. Monitor changes without creating excessive load
- 25. A practical acceptance test for monitored school documents
Frequently asked questions
Does HTTP 304 mean a source was deleted or empty?
No. A valid conditional request can return 304 without a body, indicating reuse of the corresponding stored representation.
Does a new ETag prove school policy changed?
No. It indicates a representation validator changed; the material policy meaning needs separate review.
Should sitemap lastmod always use today’s date?
No. Google advises accurate significant-update times, not a synthetic current date for unchanged content.
Does a sitemap guarantee immediate Google indexing?
No. It is a discovery hint, not a promise of crawling or ranking.
Can a last-modified timestamp establish current cohort applicability?
No. The actual official notice, effective period and required conditions remain decisive.
How should an agent react when the official source is unavailable?
Give a truthful limitation or use an authorized alternative; do not invent the missing current evidence.
Primary source documentation
Official guidance checked on 24 October 2026. All Cedar monitoring events, dates, 120 URLs and classroom calculations are invented learning cases, not measured site performance or proof of a real source revision.
- MDN — HTTP Conditional Requests: ETag, Last-Modified, If-None-Match and conditional read semantics.
- MDN — HTTP 304: Bodyless not-modified response and cached representation reuse.
- MDN — HTTP Caching: Validator and cache-control relationships.
- Google Search Central — Sitemaps: Absolute canonical URLs, size limits and accurate lastmod.
- Google Search Central — Crawling Errors: Index discovery delays and crawl verification limitations.
- RFC 9110 — HTTP Semantics: Conditional requests, representation validators and result meanings.
Related engineering guides
Explore Super Intelligence curriculum; Safe URL Fetching; Material Change Detection; Cache Invalidation; Citation Anchoring. Verified source freshness means more than an HTTP result or sitemap timestamp.