The web is not handed to a search engine as one finished catalogue. It has to be discovered, page by page, link by link, again and again.
That discovery process is web crawling. A crawler starts with known URLs, fetches resources, reads enough of each response to understand what it found, extracts links and other discoverable references, then adds eligible new URLs back into a queue for later visits. At large scale, this becomes a continuously changing map of a web that never stops changing.
This article sits beneath How Search Works and How Indexing Works. Search owns the route from human need to ranked results. Indexing owns the transformation from fetched resources into searchable state. Crawling owns the earlier question: how does the system find the resource in the first place?
1. The Web Is a Graph, Not a Folder
To a crawler, the public web can be approximated as a vast directed graph. Pages are nodes. Links are edges. Some pages are densely connected; others have only one path into them; some are completely orphaned from the crawler’s current view.
This explains the first practical truth of crawlability: a page that exists is not necessarily a page that can be discovered. If nothing known points to it, and it is absent from other discovery channels such as a sitemap, the crawler may simply have no route into it.
Internal linking therefore does more than help human navigation. It is also part of the discovery topology. The existing How Indexing Works owner explains what happens after a resource reaches the indexing pipeline; crawling explains how that resource enters the pipeline at all.
2. Crawling Begins With Seeds
A crawler needs starting points. These are often called seed URLs. They can come from previously known pages, historical crawl records, submitted sitemaps, feeds, manually supplied URLs, high-value domains or links discovered in already fetched content.
Seed choice shapes early visibility. Start from a small set of central pages and the crawler can expand outward. Start from isolated fragments and the map grows slowly.
At web scale, the system is not performing one clean breadth-first walk. It is managing many priorities, hosts, recrawl schedules, resource constraints and rules simultaneously.
3. Fetching Is an HTTP Transaction
To crawl a URL, the system makes a request much like a browser does. The response can include an HTML document, redirect, error, image, PDF or another resource type.
The response status matters. A successful response may be eligible for parsing. A redirect can send the crawler to another URL. A missing page can eventually be removed from active consideration. A server error can trigger retry logic instead of immediate abandonment.
This is one reason site reliability affects search infrastructure. If a site is consistently slow or error-prone, the crawler has to spend more effort obtaining less useful state.
4. Parsing Turns a Response Into Discoverable Structure
Fetching gives the crawler bytes. Parsing turns those bytes into structure.
For HTML, the system can identify links, canonical hints, robots directives, structured metadata, language signals, headings and other elements useful to later stages. Links are especially important because they create new graph edges.
Parsing is not the same as full semantic understanding. It is the first extraction layer that converts a document into signals the discovery and indexing systems can act on.
5. Link Extraction Feeds the Frontier
Every eligible link discovered on a fetched page is a candidate for future crawling.
The crawler normalises the URL, checks whether it has been seen before, applies rules and policies, then decides whether and when the URL should enter the crawl frontier — the continuously managed set of known but not yet fetched or due-for-refetch URLs.
This is where web crawling stops looking like “follow every link” and starts looking like a scheduling system. The crawler may know billions of URLs and have capacity to fetch only a fraction of them at any moment.
6. Duplicate Discovery Does Not Mean Duplicate Work
The same resource can be discovered through many links. URLs can differ by parameters, tracking codes, protocol, hostname variants, trailing slashes or other syntactic changes.
A strong crawler therefore keeps a memory of discovered and fetched states, normalises URLs where appropriate and later cooperates with canonicalisation systems so that multiple addresses do not automatically become multiple independent resources.
That distinction matters: URL identity and content identity are related but not identical. The dedicated canonicalisation article later in this corridor handles the preferred-resource problem directly.
7. Crawlers Revisit Because Pages Change
The first crawl is only discovery. Search systems also need freshness.
News pages change quickly. Product pages update prices. evergreen articles change slowly. URLs disappear. redirects are added. templates evolve. A crawler therefore assigns recrawl attention based on expected change, importance, historical behaviour and available capacity.
This connects directly to the existing Incremental Indexing and Freshness branch. Crawling discovers changed source state; indexing decides how that changed state becomes searchable.
8. Rendering Can Be a Second Stage
Some pages expose meaningful content directly in the initial HTML. Others depend heavily on client-side JavaScript.
Large search systems may therefore separate initial fetch from later rendering. Rendering is more computationally expensive because the system must execute enough page logic to recover content and links that were not present in the first response.
This creates an important publishing principle: critical content and navigation should not depend on fragile execution paths if they can be represented clearly in the document itself.
9. Crawl Capacity Is Finite
Even very large crawlers operate under resource limits. Bandwidth, computation, storage, host responsiveness and fairness all constrain how much can be fetched.
This means not every discovered URL is crawled immediately, and not every crawled URL is revisited equally often.
Sites with enormous numbers of low-value parameter combinations, calendar pages, duplicate filters or generated URLs can spend crawl attention on states that add little information. Good information architecture makes the important resource graph easier to traverse.
10. Crawl Politeness Protects the Publisher
A crawler that requests too aggressively can overload a smaller server. Responsible crawling therefore includes host-aware rate control, retry behaviour and respect for published access rules.
This is not only etiquette. It is systems engineering. The crawler and publisher share an interface, and the interface fails if discovery itself degrades the service being discovered.
The dedicated How Crawl Politeness Works article in this batch owns that mechanism.
11. Robots Rules Shape Access, Not Meaning
Site owners can publish crawler directives that indicate which paths automated agents should or should not fetch. These rules do not explain whether a page is valuable or trustworthy; they describe access preferences.
The Robots Exclusion Protocol is standardised in RFC 9309. The important separation is between may the crawler fetch this path? and should this resource be indexed or ranked? Those are different stages and should not be collapsed into one setting.
12. Worked Example: A New eduKateSG Article
Imagine eduKateSG publishes a new article on a topic already connected to the How X Works hub.
The article can be discovered because the hub links to it, a category page links to it, an XML sitemap lists it, or an external page references it. A crawler fetches the URL, parses the document, extracts its links and metadata, then passes the resource toward indexing.
If the article is updated later, the crawler must return before the index can reflect the new state. Discovery, recrawl and indexing are therefore a continuous loop rather than a one-time submission.
13. What Crawling Does Not Guarantee
- Crawled does not mean indexed.
- Indexed does not mean ranked highly.
- Ranked does not mean clicked.
- Clicked does not mean useful.
Each transition has its own mechanism and evidence. Crawling is only the first public-web acquisition layer.
14. A Web-Crawling Checklist
- Begin with known seed URLs.
- Fetch responses without overloading hosts.
- Parse enough structure to recover links and directives.
- Normalise and deduplicate discovered URLs.
- Apply robots and access policy.
- Place eligible URLs into a managed frontier.
- Prioritise crawling according to value, freshness and cost.
- Render where necessary to recover meaningful page state.
- Revisit important URLs when source content is likely to change.
- Pass fetched resources into the indexing pipeline.
15. Read the Mechanism Forward, Backward and Sideways
Forward: known URL → fetch → parse → extract links → schedule new URLs → revisit change. Backward: start from a missing search result and ask whether the resource was discoverable, crawlable, fetchable and indexable in the first place. Sideways: compare publisher, crawler, indexer and searcher. The publisher sees pages; the crawler sees reachable graph state; the indexer sees representations; the searcher sees only the final ranked surface.
16. The Civilisation Lesson
Civilisations have always needed discovery systems. Libraries catalogue books. ports record ships. cities map roads. archives preserve records. search crawlers perform a digital version of the same civilisational task: continuously discovering what exists, where it points and whether the map still matches the territory.
Web crawling is the acquisition layer of search: the repeated act of finding the web again because the web is never finished.
Continue through How Search Works, How Indexing Works and the master How X Works hub. Next: crawl frontiers — how an enormous set of known URLs becomes an ordered schedule of what deserves attention next.