An AI feature has a marginal cost on every request, unlike most classic software where an extra request is almost free. Each call spends GPU seconds or provider tokens, and the bill scales with traffic, prompt length, output length and how careless the architecture is. Interviewers probe cost because it exposes whether you think in unit economics: what does one successful task cost, where does the money actually go, which lever gives the biggest saving for the least quality risk, and how do you know an optimisation did not silently make answers worse? For a Backend or AI engineer, cost is a first-class design constraint next to latency and reliability, and the answer is never "use a cheaper model" alone. It is measure first, pick levers by expected saving, verify quality, then keep the cost visible with attribution, budgets and alerts. All prices and rates on this page are ILLUSTRATIVE assumptions for arithmetic, not real quotes; check current provider pricing and your own GPU rates.
Every time an AI feature answers a question or generates an image, it costs real money, whether that's a fee paid to a provider like OpenAI or the electricity and hardware bill for a company's own servers. Cost optimization for AI systems is the practice of figuring out where that money actually goes and then reducing the bill without quietly making the product worse. Unlike a lot of traditional software, where handling one more visitor to a website costs almost nothing extra, every single AI request adds a small but real charge, so the costs can add up fast as a product grows.
Think of a restaurant kitchen deciding how to staff a busy night. Every dish that goes out costs ingredients and a cook's time, so the manager doesn't just try to cook faster — they figure out which dishes are expensive to make, which ones customers barely notice the quality of, and where a cheaper ingredient works just as well. Cost optimization for AI systems is that same kitchen-management thinking, applied to computer requests instead of meals.
Cost centre Driver Typical surprise
--------------------- -------------------------------------- ------------------------------------
LLM tokens (API) input tokens * in-price output tokens cost more per token
+ output tokens * out-price than input; long prompts repeat on EVERY call
GPU-hours (self-host) GPUs * hours * $/hr, mostly idle time paying for peak capacity all day (low utilisation)
Vector DB memory vectors * dims * bytes * index overhead RAM is the expensive resource, not disk
Storage raw data, embeddings, logs, checkpoints old checkpoints and verbose logs pile up
Egress / network GB leaving a region or cloud cross-region calls and model downloads
Retries / waste failed calls, agent loops, duplicates a runaway tool loop can burn a day's budget
Per-request arithmetic (ILLUSTRATIVE: large model $3.00 in / $15.00 out per 1M tokens)
2,000 input tokens * $3.00 / 1e6 = $0.0060
300 output tokens * $15.00 / 1e6 = $0.0045 -> $0.0105 per request
1M requests/day = $10,500/day = ~$315k/month (input is 57% of the cost, output 43%)
Vector memory (ILLUSTRATIVE): 10M vectors * 768 dims * 4 bytes (float32) = 30.7 GB raw
ANN index (e.g. HNSW graph links) adds overhead on top; if held in RAM on replicated nodes,
cost = raw size * overhead * replicas (quantizing vectors to int8 cuts the raw part ~4x)The first step is knowing which line item dominates, because the levers are different. For a pay-per-token API the bill is roughly input tokens times input price plus output tokens times output price, and the input side is easy to overlook: a 2,000-token prompt with a 600-token system prompt and 1,200 tokens of retrieved context is paid for again on every request, even when the user message is 200 tokens. Output tokens are priced higher and are also the slow part, because decode generates them one at a time. For self-hosted models the cost is GPU-hours, and the dominant waste is usually not computation but idle or half-used GPUs: reserved for peak, queueing little of the time. Vector databases add a less obvious line item because indexes are memory-hungry (see vector databases), and egress and storage quietly grow with logs, checkpoints and cross-region traffic. A useful habit is to write the per-request arithmetic on the whiteboard before proposing any optimisation.
Bad metric : total monthly spend (grows when you succeed; says nothing about efficiency)
Better : cost per request (tokens_in*p_in + tokens_out*p_out + retrieval + infra)
Best : cost per SUCCESSFUL task = total cost / (requests * success rate)
Tag every call: team | feature | model | prompt_version | environment | tenant
-> cost_per_feature, cost_per_tenant, cost_per_prompt_version, top-N most expensive users
Controls: per-key/per-tenant budgets hard cap (reject/degrade) vs soft cap (alert only)
alerts on: daily spend vs forecast, cost/request drift, tokens/request drift,
cache hit rate falling, retries/agent-steps per task rising
runaway guards: max output tokens, max agent steps, max retries, per-request cost ceilingYou cannot optimise what you cannot attribute, and the metric you pick steers the outcome. Total spend rises with success, so it hides efficiency; cost per request improves if you simply return worse answers faster; cost per successful task divides by the success rate, so a cheaper model that fails 15% more often and triggers retries is correctly shown as less attractive than its price suggests. Attribution needs tags on every call (team, feature, model, prompt version, tenant) so that the bill can be sliced, and in practice a small number of features or tenants usually account for most of the spend, which tells you where to look first. Budgets and alerts are the safety net: a hard cap protects against runaways at the cost of degraded service, a soft cap only notifies. Alert on drift in the drivers (tokens per request, agent steps per task, cache hit rate, retry rate), not only on the dollar total, because those move first. The observability stack and an LLM gateway are the natural places to record token counts and enforce per-tenant budgets.
Strategy (from the script below; 1M req/day, ILLUSTRATIVE prices) $/day per successful task
--------------------------------------------------------------- ------ -------------------
baseline: everything on the large model 10,500 $0.01105
routing: 60% to the small model, 40% large 4,745 $0.00506
semantic cache: 25% hit rate + lookup + fixed infra 7,955 $0.00844
prompt trimming: 2,000 -> 1,300 input tokens (top-k 8 -> 4) 8,400 $0.00894
trim + cache + routing combined 2,930 $0.00316
Routing needs a decision: easy? --yes--> small model (cheap, fast)
--no---> large model
a cascade variant tries the small model first and escalates on low confidence / failed check
(pays for BOTH on escalations, so it wins only if the small model resolves most requests)Routing to smaller models is usually the biggest lever, because model size dominates per-token price: if the assumed small model is a fraction of the large model's price and handles 60% of traffic acceptably, cost falls by more than half in this example. The catch is the router itself: a misroute sends a hard question to a weak model, so you need an evaluation set that measures success per traffic segment, and a fallback path (see multi-provider routing). Caching removes repeated work entirely: exact-match caching is safe but low hit rate for natural language, while semantic caching raises hits at the risk of returning a wrong answer to a similar-but-different question, and it costs a lookup on every request plus cache infrastructure. Prompt trimming attacks the recurring input cost: shorten the system prompt, drop unused few-shot examples, deduplicate context, and reduce RAG top-k, but retrieval quality bounds answer quality, so cutting k from 8 to 4 must be checked against recall and groundedness rather than assumed free. Some providers also discount repeated prompt prefixes; treat that as provider-specific and check current docs. Capping max_tokens and asking for concise or structured output cuts the pricier output side.
Self-hosting is a fixed cost with a variable-cost API as the alternative (ILLUSTRATIVE numbers):
API: $1.25 per 1M output tokens (assumed) -> cost grows linearly with volume
Self-host: 2 GPUs always on * $2.50/hr * 24h + ops overhead ($333/day) = $453/day floor
break-even volume ~= 453 / 1.25 = ~362M output tokens/day; below that the API is cheaper
utilisation matters: $2.50/hr GPU at 35% util -> $0.79 per 1M tokens; at 10% util -> $2.78
Spot / preemptible training (8 GPUs, 200 GPU-hours, spot at 40% of on-demand, interruption ~12 h):
on-demand $500
spot, no checkpoints $675 (expected restarts from scratch make it WORSE)
spot, checkpoint every 60 min $215 (sweet spot in this model)
spot, checkpoint every 5 min $244 (checkpoint stalls now dominate)Compute levers raise the useful work per GPU-second. Batching (continuous batching in serving engines such as vLLM) fills the GPU while decode is memory-bandwidth bound, which lowers cost per token at the price of some per-request latency; quantization shrinks weights and KV cache so a model fits on fewer or cheaper GPUs, with a quality risk that must be measured on your task; and autoscaling stops paying for idle capacity, limited by slow model-load times and cold starts. For work that is not interactive (bulk summarisation, embeddings backfills, evaluation runs) use asynchronous or batch endpoints, which many providers price at a discount in exchange for delayed completion (check current terms), or run it off-peak on cheaper capacity. Spot or preemptible instances cut hourly price sharply but can be reclaimed, so they suit fault-tolerant jobs with checkpointing, and the model above shows the checkpoint interval is a genuine optimisation: too rare loses work, too frequent stalls the job. The self-host versus API decision is a break-even on volume and utilisation, plus engineering time, which is why low-volume products are usually cheaper on an API and high, steady volume can justify owning GPUs.
Every lever moves BOTH axes. Plot cost per successful task vs quality on your own eval set:
quality ^
95% | * large (baseline)
94% | * trimmed prompt
93.8% | * routing
92.6% | * trim+cache+routing
+------------------------------------> cost per successful task (lower is better)
Rule: accept a lever only if quality stays inside the agreed floor (and guardrails hold),
then re-measure cost per successful task after shipping - not the estimate.Cost work goes wrong when savings are computed from token counts alone. Every optimisation changes quality somewhere: a cache can serve stale or wrong answers, trimming can drop the one chunk that contained the answer, quantization can degrade rare tasks, routing can misclassify hard questions. So each change goes through the same gate as a model change: an offline evaluation on a representative set, then a staged rollout and A/B test with cost per successful task, quality and latency as explicit metrics (see A/B testing and canary deployments). Set the quality floor first, then minimise cost under it; the reverse leads to gradual erosion nobody signs off on. Also count second-order costs: an agent that loops five extra times, a retry storm after a timeout, or a prompt-injected request that inflates output can dwarf any per-token saving, so steps, retries and output length need hard ceilings. Finally, mind engineering cost: a saving of a few hundred dollars a month is not worth a fragile custom system, whereas a 40% cut on a six-figure bill is.
max_tokens and trim prompts and RAG top-k, verifying retrieval quality after any cut.# Part 1 - compare strategies: baseline vs routing vs caching vs trimming, plus cache break-even
# Python 3 stdlib only: LLM cost model. EVERY number below is an ILLUSTRATIVE assumption,
# not a real price or benchmark. Check current provider pricing before using real figures.
N = 1_000_000 # requests per day
IN_TOK, OUT_TOK = 2000, 300 # avg tokens: system 600 + retrieved context 1200 + user 200 in; 300 out
LARGE = (3.00, 15.00) # $ per 1M tokens (input, output) - assumed
SMALL = (0.25, 1.25) # $ per 1M tokens (input, output) - assumed
Q_LARGE, Q_SMALL, Q_CACHE_HIT = 0.95, 0.93, 0.92 # assumed fraction of acceptable answers
def call_cost(price, tin, tout):
return (tin * price[0] + tout * price[1]) / 1e6
def report(name, daily_cost, success_rate):
ok = N * success_rate
print(f"{name:<34} ${daily_cost:>9,.0f}/day ${daily_cost/N*1000:>6.2f} per 1k req "
f"success {success_rate:.1%} ${daily_cost/ok:.5f} per successful task")
base = N * call_cost(LARGE, IN_TOK, OUT_TOK)
report("baseline (all large)", base, Q_LARGE)
# Routing: a cheap classifier sends 60% of traffic to the small model
route_frac, router_cost = 0.60, 0.00002
cost = N * router_cost + N * route_frac * call_cost(SMALL, IN_TOK, OUT_TOK) \
+ N * (1 - route_frac) * call_cost(LARGE, IN_TOK, OUT_TOK)
report("routing 60% to small model", cost, route_frac * Q_SMALL + (1 - route_frac) * Q_LARGE)
# Caching: 25% hit rate; every request pays a lookup, plus fixed daily cache infrastructure
hit, lookup, infra = 0.25, 0.00002, 60.0
cost = N * lookup + infra + N * (1 - hit) * call_cost(LARGE, IN_TOK, OUT_TOK)
report("semantic cache, 25% hit rate", cost, hit * Q_CACHE_HIT + (1 - hit) * Q_LARGE)
# Prompt trimming: RAG top-k 8 -> 4 and shorter system prompt: 2000 -> 1300 input tokens
cost = N * call_cost(LARGE, 1300, OUT_TOK)
report("prompt trimming (2000 -> 1300 in)", cost, 0.94)
# Combined: trim, then cache, then route the misses
c_small, c_large = call_cost(SMALL, 1300, OUT_TOK), call_cost(LARGE, 1300, OUT_TOK)
miss = 1 - hit
cost = N * (lookup + router_cost * miss) + infra + N * miss * (route_frac * c_small + (1 - route_frac) * c_large)
succ = hit * Q_CACHE_HIT + miss * (route_frac * 0.92 + (1 - route_frac) * 0.94)
report("trim + cache + routing", cost, succ)
# Break-even: at what cache hit rate does the cache pay for itself?
per_req = call_cost(LARGE, IN_TOK, OUT_TOK)
h_star = (N * lookup + infra) / (N * per_req)
print(f"\ncache break-even hit rate = (lookup + infra) / (requests * cost per request) = {h_star:.2%}")
print(f"cache infra alone is ${infra:.0f}/day; at 25% hits it saves ${N*hit*per_req:,.0f}/day")
low_traffic = 20_000
print(f"at only {low_traffic:,} req/day the same infra needs a hit rate of "
f"{(low_traffic*lookup + infra)/(low_traffic*per_req):.1%} just to break even")
# Real output from python3:
# baseline (all large) $ 10,500/day $ 10.50 per 1k req success 95.0% $0.01105 per successful task
# routing 60% to small model $ 4,745/day $ 4.75 per 1k req success 93.8% $0.00506 per successful task
# semantic cache, 25% hit rate $ 7,955/day $ 7.96 per 1k req success 94.2% $0.00844 per successful task
# prompt trimming (2000 -> 1300 in) $ 8,400/day $ 8.40 per 1k req success 94.0% $0.00894 per successful task
# trim + cache + routing $ 2,930/day $ 2.93 per 1k req success 92.6% $0.00316 per successful task
#
# cache break-even hit rate = (lookup + infra) / (requests * cost per request) = 0.76%
# cache infra alone is $60/day; at 25% hits it saves $2,625/day
# at only 20,000 req/day the same infra needs a hit rate of 28.8% just to break even# Part 2 - self-host vs API break-even, and spot training with checkpoint intervals
# Python 3 stdlib only. ILLUSTRATIVE assumptions throughout - not real prices or benchmarks.
import math
# ---- A) Self-host vs pay-per-token API: break-even --------------------------------------
GPU_HR = 2.50 # $ per GPU-hour, assumed
TPS = 2500 # assumed sustained output tokens/s per GPU at good batching (hypothetical model)
API_PER_M = 1.25 # $ per 1M output tokens, assumed API price for a comparable small model
MIN_GPUS = 2 # always-on floor for availability
OPS_PER_DAY = 333 # assumed engineering/ops overhead, ~$10k/month
def self_host_daily(tokens_per_day):
capacity = TPS * 86400 # tokens/day one GPU could produce at 100% util
gpus = max(MIN_GPUS, math.ceil(tokens_per_day / (capacity * 0.6))) # plan for 60% peak utilisation
return gpus * GPU_HR * 24 + OPS_PER_DAY, gpus
print("output tokens/day API $/day self-host $/day (GPUs) cheaper")
for tpd in (10e6, 50e6, 100e6, 200e6, 400e6, 800e6):
api = tpd / 1e6 * API_PER_M
sh, g = self_host_daily(tpd)
print(f"{tpd/1e6:>9,.0f}M {api:>9,.0f} {sh:>9,.0f} ({g} GPUs) {'self-host' if sh < api else 'API'}")
floor = MIN_GPUS * GPU_HR * 24 + OPS_PER_DAY
print(f"\nbreak-even vs the always-on floor: {floor/API_PER_M:,.0f}M output tokens/day")
print(f"utilisation-only view: self-host $/1M tokens at 35% util = "
f"${GPU_HR/(TPS*3600*0.35)*1e6:.2f}, at 10% util = ${GPU_HR/(TPS*3600*0.10)*1e6:.2f} (API assumed ${API_PER_M:.2f})")
# ---- B) Spot/preemptible training with and without checkpointing -------------------------
GPUS, HOURS = 8, 25.0 # job needs 8 GPUs for 25 wall-clock hours = 200 GPU-hours
ON_DEMAND, SPOT = 2.50, 1.00 # $ per GPU-hour, assumed (spot at 40% of on-demand)
MTBI = 12.0 # assumed mean hours between interruptions for the whole job
RESTART_MIN, CKPT_STALL_MIN = 10, 1 # restart overhead and per-checkpoint stall, minutes
print(f"\non-demand: {HOURS:5.1f} h -> ${GPUS*HOURS*ON_DEMAND:,.0f}")
lam = 1 / MTBI
t = (math.exp(lam * HOURS) - 1) / lam # expected time if every interruption restarts from scratch
print(f"spot, NO checkpoints: expected {t:5.1f} h -> ${GPUS*t*SPOT:,.0f}")
for interval in (120, 60, 15, 5):
lost = (interval / 2 + RESTART_MIN) / 60 # expected hours lost per interruption
w = HOURS * (1 + CKPT_STALL_MIN / interval) / (1 - lost / MTBI)
print(f"spot, checkpoint every {interval:>3} min: expected {w:5.1f} h -> ${GPUS*w*SPOT:,.0f}")
# Real output from python3:
# output tokens/day API $/day self-host $/day (GPUs) cheaper
# 10M 12 453 (2 GPUs) API
# 50M 62 453 (2 GPUs) API
# 100M 125 453 (2 GPUs) API
# 200M 250 453 (2 GPUs) API
# 400M 500 573 (4 GPUs) API
# 800M 1,000 753 (7 GPUs) self-host
#
# break-even vs the always-on floor: 362M output tokens/day
# utilisation-only view: self-host $/1M tokens at 35% util = $0.79, at 10% util = $2.78 (API assumed $1.25)
#
# on-demand: 25.0 h -> $500
# spot, NO checkpoints: expected 84.4 h -> $675
# spot, checkpoint every 120 min: expected 27.9 h -> $223
# spot, checkpoint every 60 min: expected 26.9 h -> $215
# spot, checkpoint every 15 min: expected 27.3 h -> $219
# spot, checkpoint every 5 min: expected 30.5 h -> $244Notes on reading the output. Routing wins the most on its own because it changes the price per token on most traffic, while the cache saves less at a 25% hit rate but has a tiny break-even (under 1%) at one million requests per day; the same fixed infrastructure at 20,000 requests per day needs a 28.8% hit rate to pay off, which is why caching is a poor first move for low-traffic systems. Combining levers multiplies savings but also multiplies quality risk: the combined row has the lowest success rate, which is exactly what cost per successful task is meant to expose. All prices, hit rates, throughput and interruption rates are assumptions; the value of the script is the structure of the calculation, so swap in your own measured values.
Q: Where does the money go in an LLM application, and how do you find out?
A: Mostly tokens (input plus more expensive output) if you use an API, or GPU-hours and idle capacity if you self-host, plus vector DB memory, storage and egress. You find out by tagging every call with model, feature, prompt version and tenant, then slicing cost per request; usually a few features or tenants dominate. I would compute the arithmetic first: for example 2,000 input and 300 output tokens at assumed prices of $3 and $15 per million is about $0.0105 per request, $10.5k per day at a million requests.
Q: Why use cost per successful task instead of cost per request?
A: Cost per request rewards cheaper, faster, worse answers, and it ignores retries, escalations and rework. Dividing total cost by the number of successful outcomes penalises quality loss, so a cheaper model that succeeds less often can end up more expensive per success. The trade-off is that you need a reliable success signal (human labels, task completion, judged quality), which is itself noisy and costs money to collect.
Q: How would you cut the bill for a RAG chatbot by 40% without hurting quality?
A: Start with measurement, then take the levers with the best ratio of saving to risk: route easy questions to a smaller model, trim the system prompt and reduce top-k after checking recall and groundedness, cache frequent questions, and cap output length. Validate each on an evaluation set and A/B test with cost per successful task and quality as metrics. In my illustrative model routing alone cut the bill by about 55% and trimming about 20%, but the actual numbers depend on your traffic mix and must be re-measured.
Q: When does a semantic cache not pay off?
A: Every request pays a lookup and you carry the cache infrastructure, so it needs enough traffic and repetition. With assumed $60/day infrastructure the break-even hit rate is under 1% at 1M requests/day but about 29% at 20k requests/day. It also fails when queries are unique, personalised or time-sensitive, and a false hit returns a wrong answer, so quality risk has to be measured.
Q: Self-host a model or use an API?
A: It is a break-even on volume, utilisation and engineering time. An API is variable cost and cheaper at low or spiky volume; self-hosting is a fixed GPU floor that wins when volume is high and steady enough to keep GPUs well utilised, and it also gives control over latency, data locality and quantization. In my illustrative numbers the floor is about $453/day, so the crossover is roughly 360M output tokens/day, and ops overhead and availability (at least two replicas) push it higher.
Q: How do batching and quantization reduce cost, and what do they cost you?
A: Decode is memory-bandwidth bound, so batching many requests amortises each weight read across more tokens, raising throughput per GPU at some latency cost per request. Quantization shrinks weights and KV cache, letting a model fit on fewer or cheaper GPUs and raising throughput, at a possible quality loss that varies by model and task and has to be measured. Both lower cost per token only if the GPU was underutilised or memory constrained to begin with.
Q: How do you prevent surprise bills?
A: Per-tenant and per-key budgets enforced at the gateway (hard cap that rejects or degrades, soft cap that alerts), alerts on daily spend versus forecast and on driver metrics like tokens per request, retries and agent steps, and hard ceilings on max output tokens, agent iterations and per-request cost. The trade-off is that hard caps can degrade a legitimate spike, so set them per tier and route overflow to a cheaper model or queue.
Follow-up: Your cost per request dropped 30% after routing but users complain. What do you check?
A: Check success and quality per routed segment, not the average: the router may be sending hard queries to the small model. Look at escalation and retry rates, thumbs-down and human-handoff rates, and compare with the pre-change baseline. If quality dropped, tighten the routing threshold or use a cascade with a verifier, accepting a smaller saving; recompute cost per successful task, which probably rose less than the raw number suggested.
Follow-up: How do you decide the checkpoint interval on spot instances?
A: Balance the expected work lost per interruption (about half an interval plus restart time) against the stall of writing checkpoints each interval. With an assumed interruption every 12 hours, a one-minute checkpoint stall and 10 minutes of restart, the total cost bottomed out around the 60-minute interval and got worse both ways; real numbers depend on checkpoint size, storage speed and the actual preemption rate. Async checkpointing and sharded checkpoints reduce the stall.
Follow-up: A batch summarisation job costs $8k a month on a real-time endpoint. What would you change?
A: Move it off the interactive path: use a provider's async or batch API if it offers a discount, or schedule it on spare or spot capacity with a larger batch size, since nobody waits on the latency. Also trim shared prompt prefix, use the smallest model that meets the quality floor, and add retries with idempotency so a failure does not double the cost. The trade-off is delayed results and needing a queue and status tracking.
In a design interview, estimate cost the way you estimate capacity: state assumptions and compute. For a customer-support assistant at 1M requests per day with 2,000 input and 300 output tokens, an assumed large-model price of $3 in and $15 out per million tokens gives about $10.5k per day, so a serious design puts a gateway in front of the models that does routing to a smaller model, exact and semantic caching, per-tenant budgets and token accounting, and it feeds cost per successful task into the same dashboards as latency and error rate (see capacity planning and SLOs). For self-hosted serving, size GPUs from tokens per second, plan for peak but scale down off-peak with autoscaling, and use continuous batching and quantization to raise throughput per GPU; check GPU utilisation before buying more. For the vector store, estimate memory as vectors times dimensions times bytes times index overhead times replicas (10M 768-dimension float32 vectors is about 31 GB raw before overhead), and mention quantized vectors or disk-based indexes if memory dominates. Put non-interactive work (embedding backfills, bulk summarisation, evaluations) on batch or spot capacity. Close with the guardrails: hard limits on output tokens and agent steps, alerts on cost drivers, and an experiment plan to prove each optimisation with quality unchanged.
Every cost lever trades something: smaller models and aggressive routing trade some quality for price; caching trades correctness risk and a lookup cost for skipped inference; prompt and top-k trimming trade recall for tokens; batching trades per-request latency for throughput; quantization trades some accuracy for memory; spot capacity trades reliability for price and needs checkpointing; self-hosting trades variable cost for a fixed floor plus operational burden. The optimisation is the wrong choice when the spend is small relative to engineering time, when the feature is still finding product fit and requirements are changing weekly, when the quality metric is not trustworthy enough to notice regressions, or when latency and reliability matter more than the last percentage of cost (an interactive assistant may deliberately pay for a larger model and idle headroom). Measure before and after, keep a quality floor, and prefer a few simple, well-understood levers to a stack of clever ones that no one can debug.