Build a concurrent web crawler
Every page under a root, each fetched once, built in three stages: a breadth-first crawl, then a pool of workers that claim URLs under a lock and know when they are done, then a limit per host, retries, and the asyncio version beside it.
How do the worker threads of a crawler know the crawl is finished?
Count the work in flight, not the queue. A URL counts from the moment it is queued until the worker that fetched it has queued every new link on its page, so the count is zero only when nothing is waiting and nobody is fetching. An empty queue is not enough, because a worker that is still fetching may be about to add links. With queue.Queue the count is built in: call task_done after a page is fully handled, and join returns when every queued item has been marked done. Then put one stop marker per worker on the queue and join the threads.
Why does the seen set need a lock if the GIL is there?
Because checking whether a URL is in the set and adding it are two steps, and the GIL only promises that one thread runs bytecode at a time, switching between instructions. Two threads can both find a URL missing and both add it, so the page is fetched twice. Hold one lock across the check and the add. The lock also has to cover the queueing and the count of work in flight, so they can never disagree with the set.
Should a crawler use threads or asyncio?
The work is almost all waiting on the network, and both handle that well. CPython releases the GIL while a thread waits on I/O, so threads overlap their fetches. Threads fit when the fetching library is blocking, which is most of them. asyncio fits when the client library is async and you want thousands of connections open at once, because a task costs far less than a thread. Under asyncio a check and an add with no await between them cannot be interleaved, so the seen set needs no lock at all.
How do you stop a crawler from overloading one site?
Give every host a semaphore with a small number of permits, and take the permit around the fetch. The pool size is then a limit on total concurrency and the semaphore is a limit per host, and the two are separate numbers. Make the semaphores before any worker starts, from the set of hosts you are allowed to visit, so no two threads ever race to create one. A real crawler adds a delay between requests to the same host and reads the site's robots rules as well.
What should a crawler do when a fetch fails?
Retry a bounded number of times, then record the URL as failed and carry on. The failure must never end the worker, or the pool quietly shrinks, and it must never be swallowed, or a page that failed looks like a page with no links. In a real crawler, retry only failures that can pass (a timeout, a reset connection, a server error) with a growing delay between tries, and record a missing page at once.