A photo studio with a waiting room: you hand in your request and get a token, artists (GPUs) take requests in order (priority members first, but everyone moves), and you're called when your pictures are ready. If a request fails, you're not charged.
Problem Statement
Design a text-to-image service like Midjourney or DALL·E: users type a prompt, and the system generates images on GPUs and shows them when ready. Each job takes seconds to tens of seconds of expensive GPU time, so the design is an asynchronous job system with queues, fair scheduling, credits, and delivery of results.
Functional Requirements
Submit a prompt (with options: size, style, number of images); see progress and results.
Gallery of past generations; download and share.
Credits/plans: each generation costs credits; paid users get faster queues.
Safety filtering of prompts and outputs.
Out of scope: model training, image editing tools.
Non-Functional Requirements
Throughput per GPU: batch compatible jobs; keep GPUs busy.
Predictable wait: show queue position/ETA; paid tiers get priority without starving free users entirely.
Reliability: a failed GPU job is retried; users are never charged for failures.
Scale: e.g. 20M images/day (~230/s), each ~5–15 GPU-seconds → thousands of GPUs at peak.
POST /v1/jobs { "prompt": "a lighthouse at dawn, watercolor", "n": 4, "size": "1024x1024" }
Idempotency-Key: ...
-> 202 { job_id, status: "QUEUED", position: 37 }
GET /v1/jobs/{id} -> { status, progress, images: [url...] }
GET /v1/jobs/{id}/events (SSE) -> queued -> running 40% -> done
GET /v1/users/me/gallery?cursor=...
High-Level Design (HLD)
Jobs wait in tiered queues, a fair scheduler batches them onto GPUs, and results are delivered from storage via the CDN.
Job API: validates, runs the prompt safety check, reserves credits, writes the job, enqueues it by tier.
Scheduler: assigns jobs to GPU workers by priority and model, batching jobs that share model and resolution.
GPU workers: run the diffusion model, upload images to object storage, run the output safety classifier, report status.
Delivery: status updates over SSE/WebSocket; images served via a CDN with signed URLs.
Low-Level Design (LLD)
Credits are reserved on submit and settled or refunded at the end, all as ledger entries tied to the job id.
Job lifecycle and credits
submit: reserve credits (ledger entry -4) -> QUEUED
worker: lease job (visibility timeout) -> RUNNING -> progress events
success: upload images, safety check passes -> SUCCEEDED (reservation becomes a charge)
failure: retry up to N (new lease on another GPU) -> FAILED -> refund (ledger +4)
blocked: output safety check fails -> BLOCKED -> refund per policy
Fair scheduling
Separate queues per tier (e.g. "fast" for paid, "relaxed" for free); the scheduler serves them by weight (e.g. 4:1) so free users still progress.
Within a tier, round-robin between users so one user submitting 500 jobs doesn't block everyone.
Batching: group jobs with the same model/size into one GPU batch to raise throughput.
Workers and failure
Workers pull work (lease with timeout and heartbeats); a dead GPU's lease expires and the job is re-queued.
Job execution is idempotent on job_id: re-running writes to the same storage keys.
Autoscale GPU pools on queue depth per tier; spot/preemptible GPUs are fine for the relaxed tier because jobs can simply be retried.
Follow-up Questions
Why asynchronous instead of a request that waits for the image?
Generation takes seconds to minutes and depends on queue length; holding HTTP requests that long ties up connections and fails on flaky networks. Submit → poll/stream status → fetch result is robust and lets you show queue position.
How do you avoid charging for failed jobs?
Reserve credits on submit and settle only on success (or refund on failure), all as ledger entries tied to the job id, so retries and duplicate events can't charge twice.
How would you cache results?
Exact repeats (same prompt, params and seed) can return stored images; otherwise generation is nondeterministic by design and caching helps little. Caching the text-encoder output for popular prompts can save a little compute.