Skip to content
sysgeeks

Design a Distributed Job Queue, stage 4 of 9: decide

One slow job type

Dropbox's task framework gives each combination of task type and priority its own queue. Slack's relay can rate-limit each job type independently.

System so far· 7 parts
123456SERVICEWeb serversSERVICEEnqueue gatewayLOG / STREAMKafkaWORKERRelayQUEUERedis queuesWORKERWorkersDATABASEDatabasesand services

Select a component to see what it is responsible for and which state it owns.

  1. 1Web servers → Enqueue gateway: Enqueue job
  2. 2Enqueue gateway → Kafka: Append to topic
  3. 3Relay → Kafka: Read topics
  4. 4Relay → Redis queues: Push at a controlled rate
  5. 5Workers → Redis queues: Lease jobs
  6. 6Workers → Databases and services: Do the work

What you need to know

0 of 2 checks done

  1. Isolation means giving each kind of work its own capacity, so that a problem in one can't consume another's. In a job system that means separate queues, separate worker pools, and separate rate limits per job type or priority.

    A priority is different: it changes the order work is taken in, but everything still shares the same workers.

  2. Check

    Notifications have high priority and indexing low. All workers are already busy with slow indexing jobs when a notification arrives. When does it run?