Build a concurrent cache

Why can a thread-safe LRU cache not skip the lock on get?

Because a get is not a read. A hit moves the key to the fresh end of the recency order, and a get of an expired key deletes it, so every get writes shared state. A test that holds a get inside the injected clock, after it found its entry and before it moved it, shows a put evicting the key meanwhile, and the get then raises KeyError from move_to_end. Take one lock for get and put alike, and say why when you do.

How do I stop several threads loading the same missing key at once?

Keep a record of loads in flight, keyed by the cache key, and consult it under the same lock as the data. The first thread to miss registers a flight and runs the loader with no lock held; every later thread that misses on that key finds the flight and waits on its event instead of calling the loader. When the load finishes, the loading thread stores the value and removes the flight under the lock, then sets the event.

Should the loader run while the cache lock is held?

No. The lock guards the dict, and a load can take as long as a network call. Held across the load, it makes a hit on an unrelated key wait for somebody else's miss. A test that holds a load open inside its loader shows it: a get of a key the cache already holds is not served until the load is let go. Register the flight under the lock, release it, load, then take the lock again only to store.

What happens to the waiters when a load fails?

Every thread waiting on that flight gets the same error, and nothing is cached. The loader's exception is stored on the flight, the flight is removed under the lock, and the event is set, so each waiter wakes and raises it. Because nothing was stored, the next caller after the failure starts a fresh load. Returning None instead would make a failure look like a miss, and a caller that retries on a miss would load again.

Is functools.lru_cache thread-safe, and why not use it?

Its documentation says the cache is threadsafe, meaning the underlying data structure stays coherent during concurrent updates. It also says the wrapped function may be called more than once if another thread calls before the first call has completed and been cached, and a recorded test shows two threads inside the function at once. It has a size limit and no expiry. For a pure function that is cheap to run twice, it is the right tool; for a slow load that must run once per key and expire, write a class like the one built here.