How we crawl the web for AI search

A web crawler is, on paper, one of the simplest programs you can write. Start from a page, download it, pull out the links, follow them, repeat. You could sketch it on a napkin in a minute.

The reason crawling is actually hard is that the thing you are walking is enormous, adversarial, and never stops growing. Getting from the napkin sketch to Mercator, the crawler that feeds our AI search engine, took years, and almost none of the difficulty is in the part you would have guessed. This post is about where the difficulty actually is: deciding how much of the web to crawl, and then crawling it fast without being rude.

The path this post takes

We go from theory down to implementation, in four moves:

1. What to crawl. The web as a graph, and how we bound it with three ideas: a seed, a frontier, and a horizon.
2. Why it is hard. Crawling politely at scale is a concurrency problem. Little's Law explains why, and it sets the hardware we need.
3. How we built it. The crawler is built on Apache StormCrawler, where politeness comes from routing every host to a single fetcher and letting that fetcher meter the rate.
4. How we run it well. With billions of pages in reach, we spend a finite per-host budget between coverage and freshness.

The web is a graph, and you cannot have all of it

Think of the web as a graph: pages are nodes, links are edges, and a crawler is a walk over it, visiting a node and following the edges it has not seen yet.

The trouble is that this graph is effectively infinite and mostly junk. It also expands faster than you can read it, because every site you visit links out to sites you have never seen, and those link to more. Follow every edge and your walk never converges. You spend most of your budget on spam, mirrors, crawler traps, and auto-generated pages that no model should ever be handed as evidence.

So the first real decision is about how much of the web you allow yourself to crawl, before any question of how. Three ideas do all the work here, and they are worth naming precisely, because we spent months getting them straight ourselves.

• Seed: the finite set of hostnames we deliberately choose to crawl (a hostname like docs.linkup.so, not a whole domain). The seed changes only through a selection process we control, never on the crawler's whim.
• Frontier: everything reachable by following links while staying inside the seed hostnames. The frontier is the enclosure the crawler is allowed to roam. At the very start, the frontier is just the seed.
• Horizon: pages we discover through outbound links that fall outside the seed. We record that they exist. We do not crawl them.

The word "frontier" is a little slippery, and it tripped us up for a while. For a follow-everything crawler it is an ever-receding edge of an infinite space. For us it is a bounded enclosure, and that is the sense we always mean.

That boundary raises two questions: how do we decide what goes in the seed, and how does the seed grow over time? Neither has an obvious answer, and getting them right is most of what makes a crawl good.

Controlled crawling, and how we learned to do it

We did not arrive at that model on the first try. Our first crawler followed every link it found, and it played out exactly as the graph predicts: the boundary meant to contain the crawl was really the whole expanding web, and we spent real money accumulating garbage. So we inverted the design and made the boundary the point.

A controlled crawler never leaves its seed. It fetches everything it can reach inside a seed hostname and treats any link that points elsewhere as horizon. The trade-off is deliberate. The crawl is easy to reason about and to keep clean, and its per-host cost is trivial to budget, but new sites enter more slowly, because entering is now a decision rather than an accident of link-following.

That decision runs on a loop, built on one more structure: the link graph, every URL we have ever discovered across seed, frontier, and horizon, together with the links between them. The loop itself is short:

1. Crawl the seed. Links found inside the seed grow the frontier. Links pointing outside it grow the horizon.
2. Rank the graph. On a regular cadence we recompute PageRank over the whole link graph, horizon hosts included.
3. Promote. The best-ranked horizon hosts cross the boundary into the seed, alongside hosts our customers ask us to cover.
4. Repeat. A bigger seed grows the frontier, which reveals more horizon, which gives the next ranking pass more to work with.

Steps 2 and 3, ranking the graph and promoting from it, are where the real work happens, so they are worth slowing down on.

Ranking answers a hard question: how do you tell whether a host you have never fetched is worth crawling? You cannot look at its content, because you have never downloaded it. The only thing you can see is which pages link to it. That is exactly the input PageRank was built for, so we run PageRank over the whole link graph. Every link from a page we already crawl counts as a vote for the host it points to, and PageRank weighs each vote by the authority of the page casting it. A link from deep inside a trusted seed site is worth a lot; a link from some obscure page is worth almost nothing. A horizon host's PageRank score is just that: a number that says how strongly the part of the web we already trust points at it.

That same score does most of our spam filtering, without a separate filter. A link farm can mint millions of links among its own pages, but it cannot make a reputable seed site link to it, and those reputable links are the only ones PageRank really rewards. Junk hosts stay near the bottom of the ranking because nothing trustworthy points at them, so they simply never rise high enough to be considered.

Promotion runs on the same cadence as the ranking, and it is the only way into the seed. We take the highest-ranked horizon hosts and add them, together with any hosts our customers have explicitly asked us to cover. Nothing is promoted for being linked once; a host either earns its place on the ranking or is requested by name. The moment a host is promoted the crawler starts fetching it, its pages become frontier, and its own outbound links finally enter the graph, so the next ranking pass can judge it by what it links to as well as by what links to it.

Each turn of the loop leaves the seed almost fixed, grows the frontier as we crawl, and pushes the horizon further out:

Coverage grows in depth, inside a boundary we chose, rather than sprawling outward into the whole junk-filled graph.

None of this is as simple as it sounds, and the seed is the hard part. When we run technical interviews we hand the candidate a seed already built and ask them to crawl from it, because working out what belongs in a good seed, and how the seed relates to the frontier and the horizon, took us weeks the first time and is a problem in its own right.

The hard part is concurrency

Now the engineering problem. Once you have decided what to crawl, you have to actually fetch it, and fetch it politely.

Politeness is not optional. Every web server on the internet is a shared resource that someone else pays to run. Fire hundreds of requests per second at a single host and you compete with its real users for CPU, bandwidth, and database connections, and a smaller site can fall over under the load.

We learned this the hard way. One of our earliest crawl runs pointed at a single test site with no delay between requests. Within minutes its responses turned into a stream of 500 errors: we had put enough load on the server to take it offline. We stopped the run and reverted immediately, but the lesson was permanent.

The convention well-behaved crawlers hold to is roughly one request per second per host, plus backing off when a server slows down and identifying yourself so an operator can reach you.

To see why that convention shapes everything, it helps to borrow one piece of theory. Little's Law says that for a stable system, the number of items being processed at once equals the arrival rate times the time each item spends in the system. Applied to a crawler:

pages in flight  =  throughput  ×  latency

so:  throughput  =  pages in flight  ÷  latency

Latency is the time to fetch one page, and the server's response time is out of your hands. Some servers answer in 50 milliseconds, some take five seconds, and no amount of engineering on your side changes that. That leaves one lever for throughput: pages in flight, which is to say concurrency.

Politeness then caps concurrency in the cruelest possible place. It limits you to one in-flight request per host. The only way to go faster is to crawl more hosts at the same time, each one still held to one request per second.

The unit of scale is the host. What sets total throughput is how many hosts are fetching at any given instant, and a billion pages is only a slice of the open web. So at any moment we have thousands of hosts actively fetching, drawn from a much larger working set: most hosts in the set are idle at any instant, waiting out their one-second delay, blocked on a slow server, or simply not yet due.

From the theory to the machine

Little's Law also sizes the hardware, before you write much code. Turn it around: connections held open at once equals throughput times average in-system latency. Take a round 5,000 fetches per second as a worked example. A fetch spends a couple of seconds in the system on average, across DNS, TCP connect, TLS, transfer, and parse. So:

connections in flight  =  5,000 fetches/s  ×  ~2 s  ≈  10,000 open at once

Leave headroom for peaks and you are budgeting for tens of thousands of concurrent connections. That number is what sets the file-descriptor limits, the memory per worker, and the size of the connection pools. The page count barely enters into it.

The math sizes the machine. What it cannot tell you is where the real speedups hide, and those you only find by building a version and watching it run.

The biggest single win came from refusing to treat connections as disposable. The naive design opens a fresh TCP connection and TLS handshake for every fetch, and those handshakes can cost more than the request they carry. Because we route every URL for a host to the same fetcher, consecutive fetches of the same host can reuse a warm keep-alive connection instead, so the handshake is paid once and then amortized across every page we pull from that host. For a crawl that hits the same hosts over and over, that took a real bite out of both latency and the churn of opening and closing sockets.

The other early ceiling was write contention on the frontier index. Under load, a too-small dedup cache let the same newly discovered URL get written over and over, until the index spent more time fighting itself than crawling. Widening the cache until duplicate writes stopped dominating cleared it.

Work like that, connection reuse, cache sizing, and a long tail of smaller fixes, took a couple of months to get the crawler to its first steady thousand fetches per second. Scaling up from there was far easier, because the structural limits were gone and the system mostly just wanted more workers.

Today the crawler sustains billions of pages per day. At an average page around 150 KB, that is hundreds of terabytes of raw web fetched, parsed, and processed every day. The frontier is not static either: driven by the seed-expansion loop above, in one recent stretch the index doubled in about six weeks.

Implementing it: Apache StormCrawler

We need a system that can hold tens of thousands of concurrent fetches, scale horizontally when we add machines, and let us express our own crawl logic. A crawl is an unbounded, continuous stream of URLs that produce more URLs, so this is a stream-processing problem. We built the crawler on Apache StormCrawler, an open-source SDK for exactly this: a collection of crawl-specific components, fetchers, parsers, URL filters, and frontier logic, packaged as reusable building blocks. StormCrawler runs on top of Apache Storm, a distributed stream-processing engine, so we inherit a battle-tested crawl core and spend our own effort on the parts that are specific to us.

A StormCrawler pipeline is a Storm topology: a directed graph of spouts (sources) and bolts (processing stages), each running as many parallel tasks spread across worker processes on many machines. Ours looks like this:

The frontier lives in a sharded, queryable index. Every URL the crawler knows about is a document that records its crawl status and when it is next due to be fetched, and each document is keyed by hostname. Sharding by hostname is the important property: it keeps all of a host's URLs together, which is what makes the next step possible. (Which specific store backs this is an implementation detail, and not a very interesting one.)

Because the frontier is an index, crawling becomes a query: give me the URLs that are due for a fetch, grouped by host. One query hands back many URLs at once, and often several for the same host.

Politeness is about what happens to that batch, and it comes from two things working together: routing every URL for a host to the same place, then metering how fast that place is allowed to fetch.

The routing is Storm's stream groupings. Fields grouping sends every tuple with the same field value to the same downstream task, so when we group by hostname, every URL for a host lands on the same fetcher:

The metering is StormCrawler's fetcher. Even when a single query hands it a dozen URLs for one host, it does not fire them off together. It holds a per-host queue, releases one request at a time, waits out the politeness delay between them, and pushes backpressure up the topology so the upstream stages do not run ahead. A burst of due URLs for a host drains at roughly one request per second, however many arrived in the batch.

Without the routing this falls apart: two fetchers could each obey the one-per-second rule and still hit a host twice a second between them. Grouping by hostname is what guarantees a single fetcher, and so a single politeness clock, per host. Different hosts, and different shards, still fetch fully in parallel across the cluster.

StormCrawler gives us the rest close to free. Its components are extensible enough that all of our crawl logic lives inside the topology, and scaling the crawl is mostly a matter of adding workers.

Running the crawl well: coverage against freshness

Getting hosts in flight is the mechanism. The judgment is in what you point them at. Once billions of pages sit in the frontier, every per-host request per second is scarce capital, and it is spent against two competing goals:

• Coverage: reach pages you have not fetched yet.
• Freshness: re-fetch pages you already have, before they go stale.

We spend against that tension at three grains, each nested inside the last: the global split between new and known, then a guard that stops any single host from eating the budget, then a per-page cadence for how often a known page comes back. The rest of this section is those three.

Discovery versus refresh

A fixed split between fetching new URLs and re-fetching known ones is always wrong, because the right answer depends on how far behind discovery is. So the crawler computes it live. It measures its own emission rate, counts the discovery backlog, and works out how long that backlog would take to drain:

backlog_drain_seconds  =  discovered_backlog  ÷  emit_rate

That drain time drives a sliding split. The further behind discovery is, the more of the budget goes to reaching new pages. The closer to caught up, the more goes to refreshing what we already have. A floor keeps a little discovery running no matter what, so exploration never stops.

The share slides continuously rather than stepping through fixed buckets, so as the backlog builds and drains across a day the mix moves with it. The target is a fraction, but each fetch cycle is a discrete yes-or-no choice, so a small credit accumulator carries the remainder forward and the realized split lands exactly on target with no rounding drift.

Keeping the frontier under control

A controlled crawl still has to defend itself from its own seed. Politeness makes large hosts expensive: a host with a million pages, fetched at one request per second, would occupy a single fetch thread for over eleven days if you crawled it fully. Do that for a handful of mega-sites and the long tail starves.

So when a host grows large enough to threaten the budget, it is automatically flipped into surface mode: its crawl depth is capped to a shallow limit, and pages deeper than that are dropped from the frontier.

No single site can consume a disproportionate share of a finite budget. We keep the front page and the important near-surface pages of an enormous site, and spend the rest on breadth.

Freshness

The worst way to spend a scarce request is to re-fetch a page that has not changed. So every page gets its own refresh interval, from 1 day to 6 months. After each fetch we fingerprint the content: strip the boilerplate (navigation, scripts, attributes), then take an MD5 over the cleaned markup. A rotating ad or a live timestamp does not move that hash, so only real content changes count.

If the content changed, the interval shrinks. If it held steady, the interval grows. Each page drifts toward the cadence that matches how often it actually changes. Adaptive scheduling like this is something StormCrawler already builds in; the part we own is the fingerprint that decides what counts as a real change.

Two lanes sit on top of that baseline for pages where staleness is unacceptable. They are easy to confuse, so to be clear: one is automatic and continuous, the other is manual and one-shot.

• Freshlane (automatic): fast-moving sources get their root re-fetched as often as once a minute, and freshly discovered links are pulled through on a priority path, where batches close in seconds instead of minutes.
• Fastlane (on demand): an operator or an upstream system triggers an immediate, one-off recrawl of something that must be current right now.

These lanes are deliberate over-spends. For an AI agent acting on live information, a week-old snapshot produces a wrong answer, so for those sources we choose to spend the extra budget. We steer that budget toward the pages that earn it: the ones our customers retrieve most often, and the ones they flag as critical to their use case.

Where this leaves us

The whole design is one idea applied at every layer: decide deliberately what to crawl, then let a distributed system fetch it as fast as politeness allows.

• The web is a graph you cannot have all of, so we crawl a controlled seed, roam its frontier, record the rest as horizon, and let a PageRank loop promote the best of the horizon into the seed.
• Throughput is bounded by concurrency, since the server's response time is out of your hands, so hostname routing puts each host on a single fetcher, StormCrawler meters that fetcher to about one request per second, and scale comes from thousands of hosts fetching at once.
• Every per-host request is budgeted between coverage and freshness, from a live discovery-versus-refresh split down to per-page refresh intervals of 1 day to 6 months.

This is our version of a crawler. We think it is the right set of trade-offs for feeding an AI search engine, and it is not the only one worth making.

Two things we deliberately left out, each big enough for its own post: what happens to a page after the crawler hands it off (clean markdown, atomic indexing, contextual embeddings, and comparing a billion vectors with XOR and popcount), and how we get petabytes of crawled data out of the pipeline and into storage without losing any of it, which was its own months-long adventure.

We are hiring engineers across distributed crawling, petabyte-scale data engineering, GPU inference, and search serving. The whole stack is named after explorers and cartographers. Come name the next one.

References and further reading

• Apache StormCrawler: the open-source crawl SDK the crawler is built on, providing fetchers, parsers, URL filters, and frontier components (source on GitHub).
• Apache Storm: the distributed stream-processing engine StormCrawler runs on. See Concepts for spouts, bolts, and stream groupings.
• Little's Law: the queueing-theory result behind the concurrency argument.
• PageRank: the graph-ranking algorithm we run over our own link graph.