Producer-consumer, pipelines, worker pools

Why use queue.Queue instead of a list and a lock?

Because the queue is the lock and the condition variable, already written and already correct. A list guarded by a lock gives you the mutual exclusion but not the waiting: a consumer that finds the list empty has to spin, or you have to add a condition variable, remember to wait in a while loop rather than an if, and remember to notify on every path that adds work. The standard library documents queue.Queue as implementing all the required locking semantics, and it adds the bound, task_done and join on top. Writing it yourself is three more ways to be wrong for no throughput you can measure.

What actually is backpressure, in one sentence I can say out loud?

Backpressure is the system telling a producer to slow down, and with queue.Queue(maxsize=n) the telling is a blocking put. When the queue is full the producer stops inside put until a consumer takes something out, so the slowest stage sets the pace of every stage in front of it. Without the bound the producer never waits, the queue grows to whatever the speed difference times the run length happens to be, and memory is what pays for the difference. In the recorded run on this page the unbounded queue reaches six items while the bounded one never holds more than two.

How does a worker know there is no more work?

Put a sentinel in the queue, one per worker, after the last real item. Each worker takes exactly one sentinel and returns, so N workers need N sentinels. A boolean flag checked between items does not work: a worker blocked inside get is not checking anything, and it stays blocked because nothing more is coming. If you would rather not count workers, the other shape is an unbounded number of sentinels from a separate closing step, or Event plus a queue with a timeout, and both are more moving parts than one sentinel each.

When do I write the pool by hand and when do I use ThreadPoolExecutor?

In real code, the executor, every time: it owns the threads, the shutdown and the queue, and a Future carries the exception back to the caller instead of losing it in a thread that already died. By hand is for the round where the interviewer asks you to build a pool, and for the case where you need something the executor does not give you, such as a bounded input queue that blocks the submitting thread. Say both out loud and say which one you would ship, because the version you would ship is what is being scored.

Why are my results in a different order from my inputs?

Because results arrive in completion order and completion order is whatever the work turned out to cost. If you need input order, either use Executor.map, which hands results back in submission order, or submit a dict of future to index and put the results back in place yourself. as_completed is for when you want to react to each result as it lands, and it is the right choice for a progress bar or an early exit. With asyncio, gather already returns results in the order the awaitables were passed.