Object storage
A flat namespace of immutable blobs addressed by key, built for durability and size rather than queries or in-place updates.
Storage & state
Learn it
Videos, images, backups and exports are large and written once. In a relational database they bloat backups, replication and memory. On a server's local disk they're tied to a machine that will eventually be replaced.
Object storage keeps objects (a byte blob plus metadata) under a key in a bucket. The interface is deliberately small:
PUTa whole object,GETit (or a byte range),DELETEit,LISTkeys by prefix. There are no in-place edits; changing an object means writing a new one.Check
Can you append log lines to an existing object?Properties that shape designs:
- Durability: each object is stored redundantly across devices and facilities before the PUT succeeds; see Durability.
- Multipart upload: large objects upload in parts (5 MB to 5 GB each), in parallel, retried individually, then assembled by a "complete" call. Until then the object doesn't exist.
- Presigned URLs: your server signs a URL allowing one operation on one key until an expiry, so clients can upload or download directly, keeping bytes off your servers.
Two more:
- LIST is slow and costly at scale. Don't use it as an index. Keep the index of what exists in your database.
- Immutability is a feature. A key you never overwrite never changes, so it can be cached forever. Versioned keys (
renditions/812/attempt-3/…) make CDN Caching trivial.
Think first
A video's bytes are in object storage and its metadata in Postgres, with no shared transaction. In what order should you delete them, and why?
Quick reference
The same ideas, condensed for revision.
How it goes wrong
- Database and store disagree
- A row points at a key that was never written, or objects exist that no row references. Order writes so failures leave garbage, not dangling pointers.
- Abandoned multipart uploads
- Uncompleted parts are invisible but billed until a lifecycle rule aborts them.
- LIST as an index
- Listing a large bucket to find work is slow, expensive and eventually unworkable.
- Leaked presigned URLs
- A URL with a long expiry grants access to anyone who obtains it.
Instead, consider
- Database BLOB columns
- Objects are small (kilobytes) and must change transactionally with other rows.
- Block storage or a filesystem volume
- Software needs POSIX semantics: random writes, appends, file locks.
- Local disk
- The data is scratch space that can be lost with the machine.
In practice
- Amazon S3, Google Cloud Storage, Azure Blob
- The reference implementations.
- Cloudflare R2, Backblaze B2
- S3-compatible APIs with different egress pricing.
- MinIO, Ceph
- Self-hosted S3-compatible stores.
It assumes
- Objects are written whole and rarely modified; access is by key rather than by query.
- Your database holds the metadata (which objects exist, who owns them, what state they are in).
- Latency of tens of milliseconds per request is acceptable.
Explain it in your own words
Where you practise it
Further reading
Engineers describing it in systems they run.
- What we learned from a 22-Day storage bug (and how we fixed it)
Mux · Drew Rodman and Constantin Britcov · Post, Mar 2026
A candid incident report. Three small races in a segment store, exposed when a scaling change slowed object storage, and why it took weeks to connect the symptoms to the cause.
- How to transcode video 100x faster; or, a Gordian knot cut
Mux · Jon Dahl · Post, Apr 2023
Why transcoding everything before publishing is slow, and the alternative: split the upload into segments and transcode each one the first time someone watches it.
Related concepts
- Durability
What has to have happened before a system may say "saved": which failures the data must survive, and where that guarantee is actually made.
- Caching
Keeping a copy of data closer to where it is used, trading freshness and complexity for speed and reduced load on the source.
- Asynchronous processing
Accepting a request, recording the work durably, and doing it later in another process, so the work can outlive the request.