Design a Distributed Cache (Memcache), stage 6 of 9: change it
Deletes for every cluster
Web servers currently delete keys in their own cluster after writing. Deletes must now reach every cluster, reliably, at a very high rate.
System so far· 6 parts
Select a component to see what it is responsible for and which state it owns.
- 1Users → Web servers: Page request
- 2Web servers → mcrouter: get / multiget, delete
- 3mcrouter → memcached pool: Keys by consistent hash
- 4Web servers → MySQL: Query on miss; writes
- 5mcrouter → Gutter pool: On server failure
What you need to know
0 of 2 checks done
With several clusters, a popular key may be cached in each of them, and a write must reach every copy. Two properties matter for that delivery:
- Durable: a delete must not be lost because the web server that wrote crashed.
- Efficient: deletes are frequent, so they need batching across cluster boundaries.
Think first
A web server commits a write, then crashes before sending deletes to the other clusters. What do those clusters keep serving?The fix records the invalidation with the write: the keys to delete are part of the committed transaction, and daemons that tail the database's commit log send them out. The log is durable and replayable, so a crash or a delivery bug can be recovered from. It's the Transactional outbox idea applied to caches.
Check
Deletes now flow from the commit log. Why does the writing web server still delete the key in its own cluster right away?