Large-Scale Products

Design File Storage & Sync (Google Drive / Dropbox)

Watch: "Design Dropbox or Google Drive w/ a Ex-Meta Staff Engineer" — Hello Interview

In simple words

Your files are cut into small numbered pieces. The cloud keeps each unique piece once, plus a list of which pieces make each file. When you edit a file, only the changed pieces are uploaded, and your other devices fetch just those pieces.

Problem Statement

Design a cloud file storage and sync service like Google Drive or Dropbox: users store files in the cloud, see the same folder on all their devices, and changes made on one device appear on the others. The interesting parts are efficient uploads of large files, keeping devices in sync, and handling conflicting edits.

Functional Requirements

Upload, download, delete files and folders.
Automatic sync across a user's devices.
Share files/folders with other users (view/edit).
File version history (restore an older version).
Out of scope: real-time collaborative editing (Google Docs style), search inside files.

Non-Functional Requirements

Durability: never lose a file (it's the product's whole promise).
Consistency of metadata: a file's version history and folder tree must be correct; content may sync with some delay.
Efficiency: don't re-upload a 1 GB file because one byte changed; work on flaky networks.
Scale: 100M users, 10 PB+ stored, files up to tens of GB.

Core Entities

File / Folder (namespace entries): id, parent_id, name, owner, latest_version.
FileVersion: file_id, version, list of block hashes, size, modified_by, created_at.
Block: content hash (e.g. SHA-256) → location in object storage. Content-addressed.
Device and its sync cursor; Share (ACL entries).

API Design

POST /v1/files/commit
  { path|file_id, parent_version, blocks: [h1, h2, h3, ...], size }
  -> 200 { file_id, version }                      (all blocks present)
  -> 409 { conflict: true, current_version }       (someone else changed it)
  -> 412 { missing_blocks: [h2] }                  (upload these first)

PUT  /v1/blocks/{hash}      (or pre-signed URL)   body: block bytes
GET  /v1/blocks/{hash}
GET  /v1/changes?cursor=<c>  -> { entries changed since c, next_cursor }   (long-poll / push)

High-Level Design (HLD)

Only changed blocks are uploaded; other devices learn about changes and download just those blocks.

desktop/mobile client
  - watcher detects local changes
  - chunker splits files into blocks, hashes them
  - local DB of synced state
      |                                   |
      | metadata (small, consistent)      | block bytes (big)
      v                                   v
  Metadata service  <-->  Metadata DB     Block service -> object storage
  (namespace, versions,   (relational,    (dedup by hash; blocks
   ACLs, change log)       sharded by      are immutable)
      |                    user/namespace)
      v
  Notification service --(long-poll / WebSocket)--> other devices:
      "namespace changed" -> they call /changes and download missing blocks

Low-Level Design (LLD)

A commit writes the entry, a new version and a change-log row in one transaction; blocks are shared by hash.

Chunking, dedup and delta sync

file -> blocks of ~4 MB (fixed size; content-defined chunking survives
        insertions better) -> hash each block

upload:
  client sends the block list to commit
  server replies "missing: [h2]"   (h1, h3 already stored, maybe from another user)
  client uploads only h2, then commits again -> new version = [h1, h2', h3]

A one-byte edit in a 1 GB file uploads ~4 MB, not 1 GB.
Identical files across users are stored once (dedup by hash).

Metadata model

entries(entry_id PK, namespace_id, parent_id, name, type, latest_version, deleted)
  UNIQUE(namespace_id, parent_id, name)
versions(file_id, version, block_hashes[], size, modified_by, created_at,
         PRIMARY KEY(file_id, version))
changes(namespace_id, seq, entry_id, op)    -- per-namespace, monotonically increasing
blocks(hash PK, storage_key, size, refcount)

Metadata needs transactions (a commit updates the entry, adds a version and appends to the change log atomically), so it lives in a relational database sharded by namespace (a user's or shared folder's tree). Blocks live in object storage.

Sync protocol and conflicts

Each device stores a cursor into its namespaces' change logs. On a notification (or periodic poll), it fetches changes since the cursor, downloads missing blocks, applies them locally, and advances the cursor.
Commits are conditional on parent_version (optimistic concurrency). If two devices edit offline and both commit, the second gets 409; the client keeps both, saving its copy as "file (conflicted copy from Laptop).docx". No silent data loss.
Deletes are soft (tombstones kept for the retention period) so they sync and can be undone; unreferenced blocks are garbage-collected by refcount after retention.

Follow-up Questions

Why separate metadata from block storage?: They have opposite needs: metadata is small, relational and must be strongly consistent; blocks are huge, immutable and need cheap durable storage. Separating them lets each scale on its own terms.
How do other devices learn about a change quickly?: They hold a long-poll or WebSocket to a notification service that only says "namespace X changed"; they then pull the actual changes. Lightweight signals, heavy data on demand.
Is dedup across users a privacy problem?: It can leak whether a file already exists (upload is "instant"). Mitigations: dedupe only within a user/namespace, or always require the upload but store once; with client-side encryption cross-user dedup mostly disappears.