How to design a Video Processing Pipeline

The two numbers to keep are ~30 minutes for an upload and ~90 concurrent transcodes at peak. Together they rule out most of the prototype.

• The upload outlives every timeout and deploy cycle on the API tier, so the bytes cannot flow through it.
• The transcode outlives any reasonable request, so it must become a durable job that some other process picks up.
• Once work is a job that survives crashes, it can run twice, and the rest of the design has to make that safe.

Notice that the capacity question has no single correct answer. It turns into a choice between paying for peak capacity and making people wait. A design review that skips that conversation has decided it by accident.

The underlying move is separating the control plane from the data plane. The control plane (who may upload, which video this is, what state it is in) is small, needs authorization and belongs to your API and database. The data plane (gigabytes of bytes) should travel the shortest path to storage built for it. See Object storage.

In practice the API runs CreateMultipartUpload, inserts videos(id, owner, status = 'uploading', raw_key), and returns a presigned URL for each part. The browser uploads parts in parallel, retries any part that fails, and collects each part's ETag.

What you give up: you can no longer inspect bytes on the way in. Validation (is this actually a video? is it under 10 GB?) moves to after the upload, using size limits on the presigned request and content sniffing in the worker. Abandoned multipart uploads also leave invisible parts behind; a bucket lifecycle rule that aborts incomplete uploads after a few days cleans them up.

Treat video status as a state machine whose transitions are conditional updates:

UPDATE videos SET status = 'queued'
WHERE id = $1 AND status = 'uploading';

If two triggers race (the client's retry and the sweep, or a duplicated storage event), only the first changes a row; the second matches zero rows and is a harmless no-op. That one AND status = … clause turns "at least once" triggers into "effectively once" transitions; see Idempotency.

The sweep (the reconciler) handles the case the fast path cannot: any uploading video older than a few hours is checked against storage, then completed or marked failed: upload abandoned. Any design that relies on a client calling back needs something like it. Clients are not obliged to call back.

The deciding question is not "which tool is built for queues" but "which design leaves no window where state and work disagree?". With a jobs table, confirming the upload is one transaction:

BEGIN;
UPDATE videos SET status = 'queued' WHERE id = $1 AND status = 'uploading';
INSERT INTO jobs (video_id, status, attempt) VALUES ($1, 'queued', 0);
COMMIT;

Either both happen or neither does; see Transactions. Workers claim with SELECT … FOR UPDATE SKIP LOCKED, so concurrent workers skip rows another worker is claiming instead of blocking on them.

A message queue is the better tool when many consumers need the same event, when claim rates reach thousands per second, or when the job database is already the bottleneck. When you adopt one, the transaction above becomes "update the row and insert an outbox row", and a relay publishes outbox rows to the queue. The dual write never goes away; you just move it to a place where it can be retried. See Message queues.

A useful habit: for every write, ask what a reader sees if the process dies right after it.

• After the renditions but before the manifest: orphaned segments nobody references. That is garbage, not corruption.
• After the manifest but before the status flip: a complete but unpublished attempt. Another worker may redo it later, which wastes compute but breaks nothing.
• After the status flip: done. The CDN serves immutable objects under attempt-specific keys, so long cache lifetimes are safe; see Caching.

No state in that list is wrong, only incomplete, and incomplete states are repaired by retrying. That property is what makes the next stages survivable.

Push and pull are not a matter of modern versus old-fashioned. The deciding factors are update frequency, latency tolerance, and who knows about the change; see Server push: polling, long polling, SSE, WebSockets.

Polling here costs very little. At 5-second intervals, 200 concurrently open upload pages make 40 requests per second, each one indexed read. Push would save some of those requests and gain sub-second latency nobody needs, at the cost of a fan-out path from workers to whichever API instance holds the connection.

The update that matters most, "your video is ready" twenty minutes later when the tab may be closed, is not solved by either: it needs an email or notification, which is a separate asynchronous job triggered by the same status transition.

A lease is ownership with a deadline. The worker claims the job with leased_until = now() + 2 minutes and heartbeats every 30 seconds to push the deadline forward. Jobs can run for 20 minutes because heartbeats keep extending a short lease; the lease does not have to cover the whole job.

Recovery time is roughly the lease duration. Shorter leases recover faster, but a worker that stalls for longer than the lease (a long GC pause, a slow disk, a network blip to Postgres) loses its job while it is still working. Expiry means "probably dead", never "certainly dead". Keep that in mind for the next stage.

Note that expiry is computed with the database's now(), not each worker's clock, so clock skew between machines does not cause premature expiry.

This is the classic failure of lease-based systems: the lease expired, but the leaseholder did not know. You cannot prevent it, because you cannot tell a paused process from a dead one. You can make it harmless.

• Fencing. Each claim generates a new lease_token. Completion is UPDATE … WHERE id = 812 AND lease_token = :mine. Worker A's late update matches zero rows, and A treats that as "I lost; discard my work."
• Isolated output. Each attempt writes to renditions/812/attempt-{n}/. Two attempts can run simultaneously without touching each other's files.
• Publish by pointer. The video row stores manifest_key. Setting it inside the fenced transaction is the single step that makes one attempt's output visible. The losing attempt's files are orphans, swept later.

Together these make the job idempotent in the sense that matters: running it twice wastes compute but publishes exactly one consistent result. That is the realistic version of "exactly once", and it extends well beyond video: see Leases and fencing tokens.

The pattern generalizes: ownership is a row; every write made by an owner is conditioned on proving it still owns the row. Distributed locks, leader election and job queues all reduce to this shape, and the common bugs are the same everywhere: a check-then-act that is not atomic, a write that does not carry the token, or a failure count that only increments on paths a crash never reaches. See Concurrency control.

Retries answer the question "might this succeed if tried again?", and that depends on the error; see Retries, backoff and jitter.

• Transient (timeouts, 503s, reclaimed instances): retry with exponential backoff and jitter.
• Permanent (invalid input, unsupported codec): fail on the first attempt with a specific reason.
• Unknown (the worker died): retry, but within a budget, because a crash is exactly what a poison input looks like from outside.

The video's state machine gains an edge: processing → failed(reason), set in the same transaction that gives up on the job. A good failure reason is one the instructor can act on, such as "the file appears to be corrupted; try re-exporting it", not "exit code 139".

Deletion is a distributed operation across stores with no shared transaction: Postgres, object storage and every CDN edge. The recipe is the same as everywhere else in this design:

1. One authoritative, atomic state change. status → deleted in Postgres. From this moment the API refuses to serve the video and every conditional transition fails.
2. Asynchronous, idempotent cleanup of everything else: cancel or fence the job, delete the raw and rendition prefixes, purge CDN paths. Each step can be retried, and the reconciler re-drives anything left over.

Order the steps so that a crash at any point leaves unreferenced garbage rather than dangling references. Garbage costs money; dangling references produce broken behaviour that users see.

The scale exercise has an anticlimactic answer: the architecture holds; the bill does not. The coordination layer (Postgres, leases, a few writes per job) scales comfortably past 100x. The resource that runs out is CPU, at about 8,700 concurrent transcodes at peak, followed by egress.

So the useful changes are about capacity and cost: autoscale workers on backlog age, use spot or preemptible instances (leases already make preemption safe), pick a cheaper rendition ladder, and lean on the CDN. Replacing the jobs table would be work spent on a part that was never the bottleneck. See Backpressure and capacity and Partitioning.

Latency guarantees under load are bought with headroom. Every option either holds spare capacity (reserved pool), takes capacity from someone else (preemption), or acquires it too slowly to guarantee anything (autoscaling).

Sizing the reserve: if paid uploads arrive at 0.5 per second at peak and take 750 seconds, Little's law says about 375 are in flight. Add enough slack that a worker is almost always free, and the promise holds. That is the number to bring to the conversation with sales, priced in instance-hours.

In the data model this is one column (lane) and one extra condition in each pool's claim query. The design absorbed a new requirement without a new component, which is a good sign about its boundaries.

A good defense is not insisting that your design is right. It is showing that you know which parts of it are essential and which are incidental. Here the essential parts are the state machine, the atomic transitions and the idempotent handling of at-least-once events. The workers and leases are one implementation of "run this computation reliably", and a vendor is another.