Skip to content
sysgeeks

Design a Video Processing Pipeline, stage 9 of 14: break it

Write the claim and the completion

Turn the last two stages into code. You need two operations: one that atomically claims the next available job, and one that completes a job only if the caller still owns it. SQL, an ORM, or pseudo-code are all fine. What matters is which conditions are checked, and where.

System so far· 8 parts
123456789CLIENTInstructorbrowserSERVICEVideo APIDATABASEPostgresOBJECT STOREObject storageWORKERTranscodeworkersWORKERReconcilerEDGECDNCLIENTStudent player

Select a component to see what it is responsible for and which state it owns.

  1. 1Instructor browser → Video API: Create upload, report parts, poll status
  2. 2Instructor browser → Object storage: Upload parts via presigned URLs
  3. 3Video API → Object storage: Complete multipart upload, verify object
  4. 4Video API → Postgres: Video row and job row in one transaction
  5. 5Transcode workers → Postgres: Claim lease, heartbeat, fenced completion
  6. 6Transcode workers → Object storage: Read raw upload, write attempt output
  7. 7Reconciler → Postgres: Find abandoned uploads and orphaned output
  8. 8CDN → Object storage: Origin fetch on cache miss
  9. 9Student player → CDN: Manifest and segments
  • Request / response
  • Bulk data

What you need to know

0 of 2 checks done

  1. A claim has to be atomic: choosing a job and marking it taken must be one step, or two workers can choose the same job. In Postgres, one UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1) statement does both.

    The row lock lasts only for that statement. The lease is what holds ownership for the 20 minutes the transcode takes.

  2. Check

    Where should the attempt counter be incremented?