Design a Distributed Cache (Memcache), stage 7 of 9: change it
A second region
Replicas can lag behind the master. Caches in the replica region are filled from the local replica.
System so far· 7 parts
Select a component to see what it is responsible for and which state it owns.
- 1Users → Web servers: Page request
- 2Web servers → mcrouter: get / multiget, delete
- 3mcrouter → memcached pool: Keys by consistent hash
- 4Web servers → MySQL: Query on miss; writes
- 5MySQL → Invalidation daemon: Committed deletes
- 6Invalidation daemon → mcrouter: Batched deletes
- 7mcrouter → Gutter pool: On server failure
- Request / response
- Bulk data
- Asynchronous
What you need to know
With replication, writes go to a primary and are copied to replicas some time later: usually under a second, sometimes much longer. A replica that hasn't received a write yet serves the old data. See Replication.
A cache filled from a lagging replica stores that old data.
Think first
The master region writes, then immediately sends a delete to the replica region's cache. The data arrives at the replica 2 seconds later. A read in the replica region happens in between. What gets cached?For the user who wrote, a remote marker helps: before writing, set a marker key in the local cache saying "recently written". A miss that finds the marker reads from the master region (slower, but current) instead of the local replica.
Check
Why is evicting a remote marker different from evicting a normal cached value?