eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How Superintelligence Works | Agent Robots Exclusion — robots.txt Rules, Error States and Polite Crawling

—

by

eduKate Asia

Did you know? An AI assistant that fetches a public web page may be technically capable of retrieving it yet still be expected to refrain from crawling that path. The Robots Exclusion Protocol, commonly expressed in a site’s robots.txt file, gives automatic clients a standard way to discover the site owner’s stated crawling preferences. A responsible agent checks the rules for the appropriate user-agent identity, matches the exact target URI path, handles failure states correctly and applies separate permission, rate and privacy controls before fetching.

The short answer is that robots.txt is a crawl-access signal, not a password and not a certificate of permission to copy, republish or use information. Under RFC 9309, the crawler retrieves /robots.txt for the correct scheme and authority, chooses the applicable group, resolves competing Allow and Disallow matches by specificity, and treats an unreachable robots file differently from an unavailable one. It then combines that answer with stronger restrictions from authentication, the task’s authority, contractual terms and load management. A server returning a page is not a promise that every automated use of it is appropriate.

This chapter concerns real software agents and the current IETF standard, not an assertion that artificial superintelligence exists. The URL lists and calculations are fictional teaching cases. They have not been fetched, and no external crawler has been operated in writing this explanation. We will build a small rule evaluator, find its failure points and transfer the method to a multi-host research assistant that must not accidentally cross an origin boundary.

Contents

Three separate questions: may it fetch, is it allowed, and is it responsible?

An agent has at least three decisions to make before using a website. The technical decision asks whether its network tools can contact the server and retrieve bytes. The access-policy decision asks what the service advertises for the crawler’s identity and which other access restrictions apply. The responsibility decision asks whether the proposed retrieval is within the user’s authorisation, acceptable to the service and proportionate to its burden. Collapsing these decisions into a single yes-or-no based on HTTP 200 is the first design mistake.

Robots exclusion occupies only part of the second decision. Its rules are a standardised request to automatic clients, rather than an authentication mechanism. The IETF RFC 9309 explicitly says robots rules are not a form of access authorisation. A protected account page does not become open to an agent merely because the robots file does not mention it. Equally, robots.txt may exclude paths that an unauthenticated browser could display.

An agent receiving a research assignment should first identify the user’s permitted sources and intended use. It may be authorised to read a few pages for comparison but not to create an exhaustive mirror or extract personally identifying records. Site restrictions, privacy controls, technical security, copyright and other terms can matter independently. This guide is an engineering explanation, not legal advice about a particular website or jurisdiction.

Finally, a permitted path is not a blank cheque to overload a service. An agent making a hundred simultaneous requests might satisfy a literal robots path test but still be inconsiderate or violate separate rate-limit rules. Build request budgets, concurrency ceilings, cancellation and backoff into the fetching component. Those controls answer different questions from path permission, and all remain relevant when the standard permits crawling.

The right mental model is a series of gates. The user and platform define whether a task is authorised; URL validation checks where the request actually goes; robots rules express crawler preferences for that authority; rate and privacy rules constrain execution; and the tool’s receipt confirms what was actually fetched. Every gate can veto the action. Passing one does not cancel the others.

Identify the exact scheme, authority and user-agent token

A robots file is attached to an authority, not to a vaguely similar domain name. For an HTTPS service the relevant location is normally https://example.test/robots.txt, where example.test represents the host and any relevant port in the URI authority. A different host, subdomain, port or scheme can have a different policy. A crawler cannot safely fetch a robots file from one site, cache its rules under a broad parent domain and apply them to every neighbouring host.

The path is precisely /robots.txt at the top level, using lowercase spelling. Searching for a similarly named file inside a documentation folder is not equivalent. A website may present redirects while obtaining the file; those need the protocol’s specified treatment. The agent must preserve the identity of the authority it originally intended to access even if the robots response is eventually obtained through a different host.

RFC 9309 groups rules under User-agent lines. A compliant crawler uses a product token identifying its automatic client and selects the relevant groups according to the standard’s matching rules. A descriptive contact page or genuine product identification may help a site owner troubleshoot traffic. An agent should not falsely impersonate a human browser merely to select a more permissive group.

Two agents with different identities may legitimately receive different rule outcomes. If one service advertises Allow rules for a named crawler and Disallow rules for a wildcard crawler, the implementation should follow the applicable matching group instead of choosing whichever group permits the desired fetch. The rule engine must use documented precedence, not a ranking of which outcome the model prefers.

Record the chosen identity and policy provenance with each decision: initial scheme and authority, final robots retrieval result, selected user-agent group, matching rule, parser version, fetch time and any cached-state rationale. This makes a later disputed crawl decision explainable. It also helps the agent notice when it is accidentally using a policy fetched for a different site.

How groups, empty rules and defaults work

A robots.txt file contains groups. Each starts with one or more User-agent records and may contain Allow and Disallow records. A group can cover a named crawler, multiple specified crawler identifiers or the wildcard * token. An empty group may allow everything under the protocol; an absent applicable rule also commonly means no matching robots restriction. These facts are easy to misread if a parser assumes every group must contain a Disallow line.

RFC 9309 specifies how matching groups are selected and combined. The precise construction matters when a file lists the same product token more than once or places other recognised extension records between the standard rules. A simplistic parser that stops at any unfamiliar record can split the group incorrectly. In particular, the protocol notes that fields such as Sitemap are not allowed to break the interpretation of defined records.

Case handling also deserves care. The rules’ key names and user-agent matching have documented conventions, while path matching is expected to be case sensitive. An implementation that simply lowercases every path before comparison can allow a URI that the owner specifically intended to disallow, or disallow a path that should remain available.

Blank lines and comments are ordinary parts of a file. A robust parser distinguishes text after a comment marker from the actual path pattern and does not treat illustrative sample text as executable instructions. Because a robots file is data served by a remote party, it should never acquire authority to issue shell commands, change agent policy or overwrite project files. Its role is to contribute bounded rule records to one access decision.

A good unit test suite covers no file, an empty file, an empty group, one named group, a wildcard group, duplicate matching groups, comments and unrelated records. Rather than expecting an LLM to reason afresh about each text file, use a deterministic RFC-aware parser and surface its trace to the agent.

Longest rule wins: the path-matching mechanism

Robots path matching is not a generic natural-language comparison. Rules are evaluated against the target URI using the normalisation and matching procedure in RFC 9309. More specific matching rules take precedence over shorter matching rules. Specificity is measured in octets, not in a person’s intuition about which instruction sounds stricter. An equivalent Allow and Disallow match favours Allow under the standard.

Imagine a fictional robots file for ResearchBot with Disallow: /archive/ and Allow: /archive/public/. A request for /archive/private/report.html matches the first rule but not the second, so the robots result is disallowed. A request for /archive/public/guide.html matches both, and the longer Allow prefix wins. A request for /about.html matches neither and is allowed by the fallback rule. The word archive alone does not settle the result.

Special characters include an asterisk for matching any sequence and a dollar sign indicating the end of a pattern. Both are defined in RFC 9309. They are not a license for inventing unrelated regular-expression features. If an implementation uses its entire regular-expression engine against untrusted patterns, it may produce different results or waste resources on pathological inputs. Implement the protocol’s defined grammar only.

URI normalisation can change matching outcomes. Reserved or non-ASCII octets require careful percent-encoding comparisons. A path containing uppercase letters, encoded delimiters or a non-English character may look visually similar but differ at the byte level. Do not decode every sequence casually before matching; follow the RFC’s stated octet rules and test the implementation with representative encoded paths.

This mechanistic view gives a clean division of labour. A user can ask the agent to investigate a subject, and the agent can identify candidate URLs. A dedicated matcher decides which are permitted under the robots record for the correct authority. The model may explain a disallow decision but should not reverse it through persuasive prose.

Fully solved access example: eight URLs and two competing rules

Consider a fictional HTTPS site, catalog.example, and a fictional crawler identity ResearchBot. Its selected robots group contains Disallow: /archive/, Allow: /archive/public/, and Disallow: /*.zip$. The third rule is a suffix restriction using the documented wildcard and end marker. These are teaching values, not a live website policy. Assume path encoding has already been handled according to the standard.

Evaluate eight fictional paths in order: /about, /archive/private/a, /archive/public/readme, /archive/public/data.html, /downloads/data.zip, /archive/papers/summary, /archive/public/index, and /docs/start. The /about path matches no Disallow and is allowed. The private archive path matches /archive/ and is disallowed. /archive/public/readme matches both archive rules; the longer /archive/public/ Allow wins. The public HTML path matches the more specific archive Allow and is allowed; a separate ZIP download outside that exception is restricted.

For this particular archive-prefix comparison, /archive/public/ contains sixteen ASCII characters including the leading and trailing slashes, while /archive/ contains nine. The more specific public Allow wins for /archive/public/data.html. Separately, /downloads/data.zip is disallowed by the ZIP suffix rule because it has no competing Allow match. A real implementation must apply the RFC’s exact octet-matching procedure when wildcard patterns overlap rather than infer priority from a casual word count.

The /archive/papers/summary path is disallowed, /archive/public/index is allowed, and /docs/start is allowed. Under these stated assumptions, five of the eight URLs are allowed and three are disallowed. The allowed count is 5/8 = 62.5%; the blocked count is 3/8 = 37.5%. These denominators describe only the eight explicitly listed candidates, not the proportion of all URLs on the invented site.

The operational repair for the three blocked paths is to omit those requests and, if the task remains useful, seek other authorised sources. It is not to change the agent’s identity or construct an encoded variant of the path to escape a rule. The example also shows why the matcher should return the exact winning rule rather than just a Boolean: a reviewer can audit a surprising result without guessing how it arose.

One caution about the example: real wildcard precedence and octet matching should be unit-tested with an RFC-conforming implementation, especially when several patterns overlap. The reader exercise is to replicate the eight results with a deterministic matcher and then change one rule at a time. The numerical proportion remains correct only for the declared results.

Fetching robots.txt through redirects is not copying a new host’s policy

A server might redirect /robots.txt to a content-delivery host. RFC 9309 says crawlers should follow at least five consecutive redirects, even across authorities. When the file is reached within five consecutive redirects, its rules apply in the context of the original authority. A new host delivering the bytes does not magically become the owner of all future target URIs.

This matters for AI agents using generic HTTP clients, because the HTTP library may automatically follow redirects and show only the final URL. If the policy cache is keyed by the final host rather than the requested host, later access decisions can use the wrong authority’s rules. Preserve both the initial authority and the final retrieval chain in structured metadata.

Robots redirects also need ordinary network safety. A remote Location header must not be allowed to lead an agent into loopback interfaces, link-local metadata endpoints or other protected networks. A crawler is not exempt from server-side request forgery controls merely because the URL being retrieved is named robots.txt. The safe URL fetching guide covers those network boundaries.

If the redirect chain exceeds five hops, the RFC allows the crawler to treat robots.txt as unavailable. That possibility is different from a server or network error that makes the resource unreachable. Implementations should report which state occurred rather than converting every exception into the same permissive fallback. A stricter organisational policy may choose to abstain from crawling when uncertain.

To test this, create a harmless mock server with one redirect, five redirects, a loop and a cross-authority redirect, and confirm that the policy remains associated with the initial site. Do not run those experiments against an arbitrary third-party service without permission. The aim is to verify a parser and transport state machine under controlled conditions.

A 404 is not a 503: unavailable and unreachable states

The robots standard draws an important distinction between unavailable and unreachable files. Under its HTTP examples, a 4xx response indicates an unavailable robots resource, in which case a crawler may access resources on that server under the REP default. By contrast, a server or network error such as a 5xx response makes robots unreachable; the crawler must assume complete disallow, subject to the standard’s specified longer-term provisions.

That difference stops a dangerous failure mode. If a site is having an outage, a crawler should not assume silence means permission to crawl harder. An agent that treats a timeout or 503 as equivalent to a missing 404 may direct more traffic toward a degraded service precisely when it is least able to cope. The correct handling is both protocol-conformant and operationally cautious.

Network uncertainty requires classification. A DNS failure, TLS failure, connection reset and HTTP response code are not all interchangeable. Capture the error category and choose the corresponding policy under RFC 9309 and local safety rules. Do not retry an unbounded number of times until a permissive response appears; retries can generate load and obscure an underlying incident.

If robots remains undefined for a reasonably long period, the RFC gives special options, including continued use of a cached copy or treating the file as unavailable after a period such as thirty days. This is not an excuse for an assistant to reclassify a fresh 503 as immediately permissive. Durable crawler products should implement the formal policy, while an ad hoc research agent can choose the simpler conservative behaviour of not fetching during uncertainty.

Keep error states visible to the user when they change the result. ‘I could not retrieve the site’s crawling rules, so I did not fetch that page’ is a better report than pretending the page was inaccessible for an unrelated reason or that the research was completed.

Caching robots policies without turning yesterday’s rules into today’s truth

Fetching robots.txt for every single page is wasteful, yet caching forever is unsafe. RFC 9309 permits caching, refers to standard cache controls, and says a cached copy generally should not be used for more than twenty-four hours unless robots.txt is unreachable. A program should preserve retrieval time, HTTP freshness directives, cache validators and the authority and user-agent context for which the decision is valid.

A cached positive decision is especially important to reconsider before a large crawl. If the site owner adds a new Disallow entry, an old cache could otherwise permit thousands of requests that the current robots file discourages. Keep cache lifetime bounded, refresh at sensible checkpoints and record which policy version was applied when a request was scheduled.

A cache key should not conflate different origins. Nor should it allow one user-agent identity to borrow another’s permissive group decisions without re-evaluating the rules. It is practical to cache the parsed robots text per authority and compute matching for each agent identity; it is also possible to cache individual decisions provided the key includes all relevant dimensions and expires correctly.

When a cached policy cannot be refreshed because the server is unreachable, the protocol’s behaviour must be handled deliberately. The fact that yesterday’s policy allowed a path does not justify dropping all network-failure safeguards. A conservative application may stop and ask for later processing rather than guessing that the error is transient.

For monitoring, record policy fetch age and refresh failures separately from target-page fetch errors. A sudden increase in policy errors should cause controlled backoff rather than an accidental global crawl outage or a silent permission bypass. This is a concrete example of maintaining evidence quality across time.

Why a permitted URL still needs a rate and concurrency budget

Robots exclusion answers which URI paths a crawler may access according to the specified rules; it does not, by itself, determine a safe global request rate. Some sites publish crawl-delay or other extension records, but RFC 9309 does not define a universal crawl-delay interpretation for all crawlers. Other service policies and ordinary engineering judgement can require a lower rate.

Suppose an authorised agent has fifty permitted pages to read, a maximum of one request every two seconds, and a concurrency limit of one. If responses were instantaneous, the earliest possible start times would be 0, 2, 4 and so on through 98 seconds for the fiftieth request. That is forty-nine intervals of two seconds, not fifty intervals. Actual elapsed time may be longer because each response, retry and external budget adds delay.

If a server returns a Retry-After response or explicit rate-limit signal, the client should respect the supported semantics and reconcile that signal with its own conservative rate budget. A maximum concurrency of one prevents parallel bursts but does not stop a tight serial loop from overloading an application. Rate, concurrency, response size and total session duration are separate controls.

One full fetch may be more expensive to the service than several cached small responses. An agent can reduce demand by narrowing search queries, caching safe retrieved content, using official APIs where authorised and stopping once the reader’s question is answered. The best crawling system is not the one that fetches the most pages; it retrieves the smallest sufficient evidence set within policy.

The arithmetic is not a recommended universal rate for real websites. Two seconds was an invented scheduling limit. The right numbers must come from the service’s actual conditions and the agent operator’s access agreement. There is no single public safe crawl rate for every web host.

Access signals do not erase privacy and content boundaries

A robots Allowed decision is not a finding that a page is free of personal information, copyrighted content or sensitive business data. A page may be public in the technical sense and still be inappropriate for bulk extraction or publication. The user request, applicable law, site terms and the system’s privacy controls remain independently relevant.

Never treat a robots file as a directory of attractive secret paths. The IETF security section warns that placing paths in robots.txt makes them discoverable; it does not secure them. Site owners who need to protect an internal resource must use actual authentication or other access control. Likewise, an agent must not use a robots disallow line as a hint to test its way around restrictions.

A public page can also contain adversarial text. Instructions embedded in an article or in robots.txt itself are source content, not new privileges for the agent. They must not override a user’s boundaries, request new credentials or cause tools to make unrelated network calls. The browser-facing and model-facing security layers are separate from path selection.

When extracting material for an educational explanation, paraphrase responsibly and attribute exact sources rather than copying long passages. Do not infer consent to contact people named on a page or combine profiles into a new personal dataset. Proportional retrieval and clear citations help preserve usefulness without making a permission claim that robots alone cannot support.

A good agent response can say which sources were not fetched due to crawling restrictions and route the reader to other official or direct resources where appropriate. That is an honest limitation, not a reason to fabricate what the restricted source probably said.

A minimal deterministic decision pipeline for agent developers

Begin with a trusted request contract that defines the research purpose, allowed website set, accepted user-agent identity, maximum requests, size limits, concurrency and data-handling rules. Parse a candidate URL and resolve its scheme, authority and path before network activity. Reject unsupported schemes, credentials embedded in URLs and destinations that violate your network egress policy.

Next, obtain the proper robots file through a bounded, safety-checked request. Preserve the initial authority and redirect chain. Classify retrieval as successful, unavailable, unreachable or other relevant state. Only a successfully obtained and parseable policy yields a normal rule-matching process; use the RFC-prescribed defaults or a stricter local abstention when the policy state is not successful.

Then select matching user-agent groups and compare target paths using a tested implementation of the standard’s octet and pattern rules. Produce a machine-readable decision such as allowed, disallowed, blocked-by-other-policy, or unresolved, with evidence fields. Avoid representing an unresolved fetch as a Boolean true merely because a function returned no matching rule.

After rule approval, check other authorisation constraints and the current rate budget. Fetch with bounded time, body size, redirect count and retry count. Revalidate the destination on every redirect to prevent the permitted original URI from turning into a request to an unapproved host. The redirect-safety guide develops that boundary separately.

Finally, label the evidence actually retrieved: final URI, retrieval time, content type, content length, source attribution and any parse limitations. Keep untrusted page instructions as data. If a reader request cannot be completed because required sites disallow crawling, report the limitation rather than inventing a result or quietly switching to more aggressive techniques.

Nine common robots errors and their actual repairs

Error one is fetching robots.txt from the wrong host or scheme. Repair it by keying the check to the exact initial authority and preserving redirect provenance. Error two is pretending that a crawler is a common browser to obtain a more favourable group. Repair it by declaring the actual agent product token and applying its matching group.

Error three is using the shortest or first rule instead of the most specific matching rule. Repair it with an RFC-based matcher and tests for overlapping Allow and Disallow patterns. Error four is lowercasing every URI path or blindly decoding percent escapes before matching. Repair it with correct byte-level normalisation and case-sensitive path tests.

Error five is treating a 503 as if it were a 404. Repair it by modelling unavailable and unreachable states separately. Error six is caching yesterday’s policy indefinitely. Repair it with freshness handling, time-stamped decisions and bounded refetches. Error seven is letting a robots redirect bypass the agent’s protected-network rules. Repair it by validating every destination.

Error eight is assuming robots permission also grants permission to log in, download everything or publish the content. Repair it by separate task, legal, privacy and access gates. Error nine is honouring path rules but flooding a server with hundreds of simultaneous requests. Repair it with rate, concurrency and session budgets and with early stopping when enough evidence exists.

These failures sit at different layers; fixing just the natural-language prompt will not make the engine trustworthy. The durable solution is a deterministic decision service with test cases and logged outcomes. An LLM can help a user understand a denial, but it should not be given a hidden exemption from the checks.

Changed-context exercise: a researcher crosses a documentation subdomain

Imagine a fictional agent researching science lessons from docs.example and images.docs.example. The owner approves reading up to twelve documentation pages and at most four image descriptions. The first site’s robots file contains Disallow: /draft/ and Allow: /draft/public/. The image host has its own robots file that returns a temporary server error. What can the agent responsibly do?

For docs.example, it evaluates candidate paths against the actual chosen user-agent group. A page /draft/public/guide is allowed by the longer rule if no other more specific match applies; /draft/internal/notes is disallowed. It may fetch other allowed documentation pages within the twelve-page budget after other safety checks. The decision is local to that host and its policy version.

For images.docs.example, the server error makes robots unreachable under the RFC example. The compliant default is complete disallow, so the agent does not fetch the four image descriptions while that state persists. It must not import the documentation host’s positive rules just because the subdomain labels look related. Nor should it send parallel retries until one happens to return a permissive response.

Suppose seven of the twelve proposed docs paths pass the robots check, two are disallowed, and three are irrelevant to the user question. The agent retrieves only the seven relevant permitted pages, not the full twelve budget. Its completion report has a clear numerator and denominator: seven candidate pages fetched out of twelve proposed, with two disallowed and three intentionally skipped. The separate four image candidates were not fetched due to the image host’s unreachable policy.

This transfer shows the mechanism survives a change in context, but the old answers do not. New authority means a new robots policy; a temporary error means a different state; and the original request budget remains a ceiling rather than an obligation. The result can still be useful if the seven retrieved pages answer the question. If they do not, the agent should explain what is missing instead of fabricating image findings.

Limits of robots exclusion and responsible interpretation

Robots exclusion is a cooperation protocol for crawlers, not a firewall or proof of legal entitlement. It should sit alongside authentication, permission controls, rate management and privacy rules. The standard defines how conforming crawlers interpret its syntax and error states; it does not define every service’s acceptable-use contract or every country’s legal rules.

Some hosts are configured imperfectly. A malformed file may contain partially parseable rules; the RFC says crawlers must try to parse each line and use what is parseable. A parser with too small a limit may discard required information. The standard calls for at least a 500 KiB parsing limit as a protection and interoperability floor. Treat the document as untrusted text subject to memory and size bounds.

The rules can change while a long crawl is in progress, so time-stamped cache policy matters. A site may also return different content based on access conditions, which reinforces the need for deterministic logs. No finite parser or crawl delay eliminates every risk of incorrect scope or service overload.

Finally, a source blocked under robots can still be directly visited by its owner or linked by a human in many contexts. That distinction does not authorise the agent to pretend it is a human reader to evade the restriction. The program should explain the boundary and preserve the user’s choice of lawful alternative research routes.

These examples are intentionally narrower than discussions of search-engine indexing or model-training law. The reader job is to implement a safe robots decision for current agent fetching, with precise provenance and measured network impact.

Frequently asked questions

What is robots.txt? It is a plain-text file at the service’s top-level /robots.txt path that expresses access preferences for compliant crawlers through user-agent groups and Allow or Disallow rules. It is not a login mechanism.

Does robots.txt make content private? No. The file itself is public and can reveal protected-looking path names. Access control requires real authentication or another security mechanism.

Can the agent crawl if the file is missing? Under RFC 9309, an unavailable file indicated by a 4xx response generally permits access under robots rules, but other task, service and privacy restrictions may still block it.

What if robots.txt returns a server error? An unreachable file, including the RFC’s 5xx example, requires the crawler to assume complete disallow, with limited longer-term provisions described in the standard.

Which rule wins when Allow and Disallow both match? The most specific matching rule wins; an equivalent match favours Allow under RFC 9309. Test wildcard and encoded-path cases using the standard’s octet rules.

Is every page allowed if no Disallow matches? The robots default may allow it, but authentication, privacy, legal and load safeguards remain independently relevant.

Does crawl-delay have one universal meaning? No. It is not part of the normative Allow and Disallow matching core of RFC 9309; use separately applicable service rules and conservative rate budgets.

Can an LLM interpret the rules from memory instead of using a parser? It can explain concepts, but a real crawler should use a deterministic, tested matcher and record the precise winning rule.

Should the agent fetch every allowed page? No. Retrieve only the relevant, authorised evidence necessary for the reader’s task and stop once it is sufficient.

Reader pathways, practice and primary standards

If you build agent tools, implement the RFC parser and transport error state machine first, then test with a local mock host. Audit exact authority, user-agent, selected group, specific winning rule, cache age and decision. Give the model a bounded fetch tool that refuses requests when that deterministic gate returns disallowed or unresolved.

If you teach computing, let students classify a small invented set of paths, first with one Disallow rule and then with a longer Allow exception. Ask them to explain the difference between a 404 and a 503 robots response, then transfer the exercise to a second subdomain with independent policy. The aim is reasoning that survives changed conditions.

If you manage a public website, use robots exclusion as a preference signal while keeping sensitive resources behind actual access controls. Provide clear service policies and meaningful rate limits; a crawler that behaves correctly should not need to guess which traffic patterns are acceptable.

Primary source: IETF RFC 9309, Robots Exclusion Protocol, especially Sections 2.2, 2.3, 2.4 and 3. Supporting HTTP policy and cache definitions appear in RFC 9110 and RFC 9111. Apply current site-specific rules rather than assuming these examples describe a real host.

For wider system design, use the verified How Super Intelligence Works hub and education for usable knowledge. No crawler was executed and no external page was fetched as part of the invented classroom scenario in this guide.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading