Skip to content
sysgeeks

Design a Distributed Job Queue, stage 2 of 9: break it

Why the queue stopped draining

Select the lines that describe causes in the design, not just symptoms.

System so far· 4 parts
12SERVICEWeb serversQUEUERedis queuesWORKERWorkersDATABASEDatabasesand services

Select a component to see what it is responsible for and which state it owns.

  1. 1Workers → Redis queues: Lease jobs
  2. 2Workers → Databases and services: Do the work

What you need to know

0 of 3 checks done

  1. Redis has a memory limit (maxmemory). When a Redis used as a queue reaches it, commands that would add data fail with an out-of-memory error. Commands that only remove data still work.

    The catch is in the details: a reliable dequeue usually moves the job to a "processing" list (RPOPLPUSH) so it isn't lost if the worker crashes. Moving writes a new entry, and writing needs memory.

  2. Think first

    Redis is at its memory limit. Workers dequeue with RPOPLPUSH, which writes the job into a processing list. What happens to the queue?