Kubernetes is a control system for containers: you declare "run N copies of this image with these resources, reachable at this name", and a set of controllers keeps reconciling reality towards that declaration, placing containers on machines, restarting them, replacing them during upgrades and scaling them. Interviewers probe it because it is where most model-serving and backend services actually run, and because its defaults surprise people: a Pod that "has 4 GB" may be killed at 4 GB, a "healthy" deployment may serve errors during a rollout, and a GPU Pod can sit in Pending forever. For a Backend or AI engineer the goal is not to memorise kubectl flags but to explain how requests turn into placement, how probes turn into traffic decisions, and why big-model workloads (multi-GB images, minutes-long start-up, whole-GPU allocation) stress every one of those defaults. It pairs with VPC and subnets (Pod and Service networks live inside, and must not collide with, your VPC address space) and with autoscaling for inference workloads.
A container packages an application with everything it needs to run, so it behaves the same on a laptop and in the cloud. Container orchestration is the software that takes a pile of these packages and a set of machines and decides which package runs on which machine, restarts anything that crashes, spreads copies across machines for safety, and replaces old versions with new ones without an outage. Kubernetes is the most widely used tool that does this job, and it does it by constantly comparing "what should be running" against "what is actually running" and fixing any gap it finds.
Think of a large office building's facilities team. Tenants don't pick their own room or worry about which floor has spare space; they just say "we need a room for 20 people," and the facilities team assigns a space, moves people if a room floods, and adds more rooms if the company grows. Kubernetes plays that facilities-team role for software: application teams say what they need, and Kubernetes finds the machines, keeps things running, and reacts when something goes wrong.
Desired state (YAML you apply) Actual state (what runs)
Deployment "llm-api", replicas: 3 ---> ReplicaSet (one per Pod template version)
|
+--> Pod llm-api-7d9f-a (node A, IP 10.244.1.7)
+--> Pod llm-api-7d9f-b (node B, IP 10.244.2.3)
+--> Pod llm-api-7d9f-c (node B, IP 10.244.2.9)
Service "llm-api" (ClusterIP 10.96.0.20:80) selects Pods by label app=llm-api
| endpoints list = only Pods that are READY
v
kube-proxy / dataplane rules on every node spread connections over ready endpoints
Ingress "api.example.com/v1" ---> Ingress controller (a Pod running nginx/Envoy/cloud LB)
| routes host/path (L7) to the Service
Client --> cloud load balancer --> ingress controller --> Service --> PodA Pod is the unit of scheduling: one or more containers sharing a network namespace (one IP) and volumes. Pods are disposable and their IPs change, so you never manage them directly. A Deployment owns a ReplicaSet, which owns Pods, and a controller loop constantly compares "3 wanted" with "how many exist and are ready" and creates or deletes Pods to close the gap; this is the whole "self-healing" story and it is why deleting a Pod under a Deployment just produces a new one. A Service gives a stable virtual IP and DNS name to whatever Pods currently match its label selector, and only Pods passing their readiness probe are in its endpoint list. An Ingress is only an object: nothing happens until an ingress controller is installed, and it adds host/path routing, TLS termination and often a cloud load balancer (the newer Gateway API is the more expressive successor). Compare it with the L4/L7 split in L4 vs L7 load balancing and the discovery model in service discovery.
Where it breaks: a Service balances connections, not requests. Long-lived gRPC or streaming connections from a few clients stick to a few Pods and leave the rest idle unless you use a client-side or L7 balancer. Also the control plane is eventually consistent: after a Pod is deleted there is a short window in which some nodes still route to it, which is why graceful shutdown (a preStop delay, handling SIGTERM, a sufficient terminationGracePeriodSeconds) matters for streaming LLM responses.
Two different numbers, two different enforcers:
requests limits
used by the SCHEDULER (placement) the KUBELET + Linux cgroups (runtime)
question answered "does it fit on this node?" "how much may it actually use?"
CPU exceeded (never exceeded; it is a share) THROTTLED (CFS quota) - process keeps running, slower
memory exceeded - OOM-KILLED (exit 137, reason OOMKilled)
GPU (extended res.) must be whole numbers; set in limits (requests, if given, must equal it)
Scheduler = FILTER (hard constraints) then SCORE (preferences), per Pod:
filter: enough allocatable CPU/mem/GPU after existing REQUESTS? nodeSelector/affinity match?
node taints tolerated? volume zone ok?
score : spread (least allocated) vs pack (most allocated) vs topology spread, etc.
QoS class (derived, decides who is evicted first under node memory pressure):
Guaranteed requests == limits for every container (last to be evicted)
Burstable some requests/limits set, not equal
BestEffort nothing set (first to be evicted)The scheduler never looks at real usage, only at the sum of requests already placed on each node against its allocatable capacity (node capacity minus system reservations). That means requests are a capacity contract: set them too high and you strand expensive nodes at 20% real utilisation; set them too low and the scheduler over-commits nodes, and the first traffic spike triggers throttling, evictions or OOM kills. Limits behave differently for the two resource kinds. CPU is compressible: exceeding the limit means the kernel's CFS quota pauses your threads for the rest of each scheduling period, so a latency-sensitive server with a tight CPU limit shows p99 spikes even when the node has idle cores. Memory is not compressible: the kernel has nothing to slow down, so crossing the limit ends in an OOM kill and a restart. Many teams therefore set memory limits == requests and either omit CPU limits or set them generously, but this is a judgement call, not a rule.
The Try It Yourself demo below implements the filter and score steps in about 40 lines. Its key lessons are that a "spread" policy can fragment a cluster so that a later big Pod cannot fit even though total free CPU is enough, and that GPU-node CPU and memory are consumed by Pods too, so a GPU node can be full without its GPUs being the constraint.
# Labelled illustrative manifest (YAML). Values are examples, not recommendations; no output shown.
apiVersion: apps/v1
kind: Deployment
metadata: {name: llm-api}
spec:
replicas: 3
revisionHistoryLimit: 5 # old ReplicaSets kept so rollback is possible
progressDeadlineSeconds: 900 # marks the rollout "failed" (it does NOT roll back by itself)
strategy:
type: RollingUpdate
rollingUpdate: {maxSurge: 1, maxUnavailable: 0}
selector: {matchLabels: {app: llm-api}}
template:
metadata: {labels: {app: llm-api}}
spec:
terminationGracePeriodSeconds: 60
containers:
- name: server
image: registry.example.com/llm-api:1.4.2 # pin a tag or digest, not :latest
ports: [{containerPort: 8000}]
resources:
requests: {cpu: "2", memory: 8Gi}
limits: {memory: 8Gi}
startupProbe: # slow start: weights loading can take minutes
httpGet: {path: /healthz, port: 8000}
periodSeconds: 10
failureThreshold: 60 # up to ~10 min before the container is restarted
readinessProbe: # "may I receive traffic?" (gates Service endpoints)
httpGet: {path: /ready, port: 8000}
periodSeconds: 5
failureThreshold: 2
livenessProbe: # "is the process wedged?" (failure = restart container)
httpGet: {path: /healthz, port: 8000}
periodSeconds: 10
failureThreshold: 3
lifecycle:
preStop: {exec: {command: ["sleep", "10"]}} # let endpoint removal propagate first
# Rollout and rollback commands (behaviour described, not shown as output):
# kubectl rollout status deployment/llm-api
# kubectl rollout undo deployment/llm-api # back to the previous ReplicaSet
# kubectl rollout undo deployment/llm-api --to-revision=3The three probes answer three different questions and mixing them up causes outages. Readiness failure removes the Pod from Service endpoints without restarting it; this is the right response to "still loading the model" or "temporarily overloaded". Liveness failure makes the kubelet restart the container; use it only for genuine deadlock, and keep it independent of downstream dependencies, otherwise a database hiccup makes every replica restart at once and turns a partial outage into a total one. The startup probe holds off liveness and readiness until it first succeeds, which is what lets a model server take five minutes to load weights without being killed by a liveness probe tuned for fast APIs.
A rolling update creates a new ReplicaSet and shifts replicas gradually, bounded by maxSurge (extra Pods allowed above the desired count) and maxUnavailable (Pods allowed below it). With maxUnavailable: 0 and a correct readiness probe, capacity never dips; without a real readiness probe Kubernetes will happily send traffic to a Pod that has started but not loaded its model. Rollback is just scaling the previous ReplicaSet back up (kubectl rollout undo), which only works while revision history is kept, and Kubernetes will not roll back on its own: progressDeadlineSeconds only flags the rollout as stalled. That is why canary tooling and metric-based rollouts (see A/B testing and canary deployments) sit on top. The GPU catch: a surge Pod needs a free GPU, and if none exists it stays Pending and the rollout stalls, so GPU deployments often use maxSurge: 0, maxUnavailable: 1, accepting a dip in capacity.
# Labelled illustrative manifest (YAML). Requires a GPU device plugin (e.g. NVIDIA's) on the nodes.
# Label/taint names differ by cloud and installer; check your provider's docs. No output is shown.
apiVersion: v1
kind: Pod
metadata: {name: llm-worker}
spec:
nodeSelector:
accelerator: nvidia-a100 # example label put on GPU nodes (often by GPU feature discovery)
tolerations:
- {key: "nvidia.com/gpu", operator: "Exists", effect: "NoSchedule"} # nodes are tainted to keep CPU Pods off
containers:
- name: server
image: registry.example.com/llm-worker:2.0
resources:
limits:
nvidia.com/gpu: 1 # whole GPUs only; requests default to the limit
memory: 48Gi
requests: {cpu: "6", memory: 48Gi}GPUs are an extended resource. A vendor device plugin runs on each GPU node and tells the kubelet "this node has 4 nvidia.com/gpu"; the scheduler then treats that like any other countable resource, so the placement logic is the same bin-packing as CPU, just with integers that cannot be over-committed or split by default. (Sharing a GPU between Pods needs an extra mechanism such as MIG partitions or time-slicing; that changes isolation and performance and depends on hardware and driver support.) Because GPU nodes are scarce and costly, the usual pattern is a dedicated node pool with a taint (repels Pods without a matching toleration, so a stray log shipper does not eat a GPU node's CPU) plus a nodeSelector or affinity (pulls the GPU Pod to the right hardware generation). Taints repel; selectors attract; you generally need both.
What breaks: a GPU Pod is Pending with "Insufficient nvidia.com/gpu" when none are free, and the cluster autoscaler adding a GPU node can take minutes if capacity exists at all; multi-GPU tensor-parallel models need the GPUs on the same node, and fragmentation (three nodes with one free GPU each) cannot host a 2-GPU Pod; and a GPU node running out of host CPU or RAM leaves GPUs idle, exactly as the demo shows. Cold starts are the other big one: a multi-GB container image pull, then model weight download or load into GPU memory, then CUDA and kernel warm-up can total minutes, during which the Pod is not ready. Mitigations include pre-pulled images or node-local caches, weights on a fast shared volume, keeping a warm minimum replica count, and a startup probe sized to reality (see GPU hardware utilization basics and model serving architectures).
HorizontalPodAutoscaler control loop (roughly every 15 s by default):
desiredReplicas = ceil( currentReplicas * currentMetricValue / targetMetricValue )
example: 4 replicas, CPU 90% of request, target 60% -> ceil(4 * 90/60) = 6
example: 6 replicas, in-flight requests/pod 5, target 10 -> ceil(6 * 5/10) = 3
guard rails: min/max replicas, tolerance (~10% dead band), scale-down stabilisation
window (default 5 min) so brief dips do not cause flapping.
# Labelled illustrative manifests (YAML), no output shown
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: llm-api}
spec:
scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: llm-api}
minReplicas: 3
maxReplicas: 12
metrics:
- type: Pods
pods:
metric: {name: inflight_requests} # custom metric, needs a metrics adapter (e.g. Prometheus adapter, KEDA)
target: {type: AverageValue, averageValue: "10"}
---
apiVersion: v1
kind: ConfigMap
metadata: {name: llm-config}
data: {MAX_TOKENS: "1024", MODEL_NAME: "example-model"}
---
apiVersion: v1
kind: Secret
metadata: {name: llm-secrets}
type: Opaque
stringData: {PROVIDER_API_KEY: "REPLACE_ME"} # base64-ENCODED at rest in etcd unless encryption at rest is enabledHPA percentages are relative to requests, so "60% CPU" is meaningless if requests are wrong. CPU is also a poor scaling signal for GPU inference: the GPU may be saturated while the host CPU is quiet, so you scale on queue depth, in-flight requests or tokens/s through a custom-metrics pipeline. Pod scale-up is slow for large models (schedule, pull, load, warm up), so autoscaling reacts minutes after a spike; keep headroom or a warm pool, and see autoscaling for inference workloads for the signals in detail. For configuration, a ConfigMap holds non-secret settings and a Secret holds credentials, both injectable as environment variables (fixed at container start, so changes need a rollout) or mounted files (updated eventually in the running Pod). A Kubernetes Secret is base64-encoded, not encrypted by default: real protection comes from RBAC, encryption at rest, and often an external secret manager. Never bake API keys into images, and do not commit Secret manifests to git.
rollout undo) is manual, and a stalled rollout is only flagged, not reverted.kubectl apply a manifest; the API server validates it and stores the desired state in etcd. Nothing runs yet.Pending.A toy scheduler with the filter and score phases, node taints and tolerations, a nodeSelector and a GPU resource. It runs the same workload twice, once with a spread policy and once with a bin-packing policy. It is a teaching model, not the real kube-scheduler: no affinity, priorities, preemption or topology spread.
from dataclasses import dataclass, field
@dataclass
class Node:
name: str
cpu: int # allocatable millicores
mem: int # allocatable MiB
gpu: int = 0
labels: dict = field(default_factory=dict)
taints: set = field(default_factory=set)
used: dict = field(default_factory=lambda: {"cpu": 0, "mem": 0, "gpu": 0})
pods: list = field(default_factory=list)
def free(self, r):
return getattr(self, r) - self.used[r]
@dataclass
class Pod:
name: str
cpu: int
mem: int
gpu: int = 0
node_selector: dict = field(default_factory=dict)
tolerations: set = field(default_factory=set)
def fits(node, pod):
"""Filter phase: hard constraints only (like the scheduler's predicates)."""
if not node.taints <= pod.tolerations: # untolerated taint -> reject
return False
if any(node.labels.get(k) != v for k, v in pod.node_selector.items()):
return False
return all(node.free(r) >= getattr(pod, r) for r in ("cpu", "mem", "gpu"))
def score_spread(node, pod):
"""Score phase, 'least allocated': prefer the emptiest node (spreads load)."""
return sum(node.free(r) / getattr(node, r) for r in ("cpu", "mem"))
def score_binpack(node, pod):
"""'Most allocated': prefer the fullest node that still fits (packs tightly)."""
return -score_spread(node, pod)
def schedule(pods, nodes, score):
pending = []
for pod in pods:
candidates = [n for n in nodes if fits(n, pod)]
if not candidates:
pending.append(pod.name) # stays Pending, like a real Pod
continue
best = max(candidates, key=lambda n: score(n, pod))
for r in ("cpu", "mem", "gpu"):
best.used[r] += getattr(pod, r)
best.pods.append(pod.name)
return pending
def cluster():
return [
Node("cpu-1", 4000, 16384),
Node("cpu-2", 4000, 16384),
Node("gpu-1", 8000, 65536, gpu=2, labels={"accel": "a100"}, taints={"nvidia.com/gpu"}),
]
def workload():
api = [Pod(f"api-{i}", 1000, 2048) for i in range(5)] # small CPU pods
llm = [Pod(f"llm-{i}", 4000, 24576, gpu=1,
node_selector={"accel": "a100"}, tolerations={"nvidia.com/gpu"}) for i in range(3)]
batch = [Pod("batch-0", 3000, 4096)] # arrives after the small pods
return api + batch + llm
for label, score in (("spread (least allocated)", score_spread), ("bin-pack (most allocated)", score_binpack)):
nodes = cluster()
pending = schedule(workload(), nodes, score)
print(f"== {label} ==")
for n in nodes:
print(f" {n.name}: cpu {n.used['cpu']}/{n.cpu}m mem {n.used['mem']}/{n.mem}Mi gpu {n.used['gpu']}/{n.gpu} {n.pods}")
print(" Pending:", pending)
# Output:
# == spread (least allocated) ==
# cpu-1: cpu 3000/4000m mem 6144/16384Mi gpu 0/0 ['api-0', 'api-2', 'api-4']
# cpu-2: cpu 2000/4000m mem 4096/16384Mi gpu 0/0 ['api-1', 'api-3']
# gpu-1: cpu 8000/8000m mem 49152/65536Mi gpu 2/2 ['llm-0', 'llm-1']
# Pending: ['batch-0', 'llm-2']
# == bin-pack (most allocated) ==
# cpu-1: cpu 4000/4000m mem 8192/16384Mi gpu 0/0 ['api-0', 'api-1', 'api-2', 'api-3']
# cpu-2: cpu 4000/4000m mem 6144/16384Mi gpu 0/0 ['api-4', 'batch-0']
# gpu-1: cpu 8000/8000m mem 49152/65536Mi gpu 2/2 ['llm-0', 'llm-1']
# Pending: ['llm-2']Read the output: under "spread", batch-0 (3000m CPU) stays Pending even though the cluster has 3000m of free CPU in total, because it is split 1000m and 2000m across two nodes. Bin-packing leaves one node with a contiguous 3000m gap and places it. Both policies leave llm-2 Pending: only 2 GPUs exist, and each LLM Pod also consumes 4000m of the GPU node's 8000m CPU. Spreading has its own benefits (fault isolation, less noisy-neighbour risk), so the point is the trade-off, not that packing always wins. For the real thing, these commands show the same information (no output is invented here):
kubectl describe pod llm-2 # Events section explains why it is Pending ("Insufficient nvidia.com/gpu")
kubectl get pod llm-api-xyz -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' # e.g. OOMKilled
kubectl top pod # needs metrics-server; live CPU/memory, compare with requests
kubectl get hpa llm-api -w # watch current vs target metric and replica countQ: What is the difference between resource requests and limits, and who enforces each?
A: Requests are what the scheduler reserves when placing a Pod: it sums requests per node and refuses to over-commit allocatable capacity. Limits are enforced at runtime by the kubelet and cgroups: exceeding a CPU limit throttles the container, exceeding a memory limit gets it OOM-killed. Requests decide where you run and what you pay for; limits decide how you fail, and the ratio between them determines QoS class and eviction order.
Q: A container keeps restarting with exit code 137 and reason OOMKilled. How do you debug it?
A: Confirm with kubectl describe pod that the last state is OOMKilled, then compare the memory limit against real usage over time from metrics. Common causes are a limit set below the working set (for a model server, weights plus KV cache plus framework overhead), a leak, or unbounded per-request memory such as very long prompts. Fix by raising the limit and request together, bounding request size or concurrency, or reducing model memory; simply removing the limit only moves the kill to the node under memory pressure.
Q: Explain liveness, readiness and startup probes and when each is wrong.
A: Readiness controls whether the Pod receives traffic, liveness restarts a wedged container, and the startup probe suspends the other two until the app has booted. Liveness is wrong when it checks downstream dependencies (a database blip restarts every replica) or when its timeout is shorter than a slow start; readiness is wrong when it returns OK before the model is loaded. For a model server I would use a generous startup probe, a readiness that checks the model is really loaded, and a shallow liveness.
Q: How does a Deployment rolling update work, and how do you make it safe?
A: A new ReplicaSet is scaled up while the old one is scaled down, within maxSurge and maxUnavailable, gated by readiness. Safety comes from an accurate readiness probe, maxUnavailable: 0 when you have spare capacity, graceful shutdown so in-flight requests finish, pinned image tags and revision history for kubectl rollout undo. Kubernetes does not judge whether the new version is good, only whether it is ready, so error-rate checks or canary analysis are needed on top.
Q: How are GPUs scheduled in Kubernetes?
A: A device plugin advertises GPUs as an extended resource such as nvidia.com/gpu; a Pod asks for whole units in its limits and the scheduler places it only on a node with enough free ones. In practice GPU pools are tainted, and workloads carry a matching toleration plus a nodeSelector or affinity for the GPU type. The costs are integer-only allocation by default, fragmentation for multi-GPU Pods, and slow scale-up, so capacity planning has to be explicit.
Q: Why is CPU-based HPA a poor fit for LLM inference, and what would you scale on?
A: The bottleneck is generally the GPU (memory bandwidth in decode, batch capacity), so host CPU can stay low while requests queue. I would scale on in-flight requests per replica, queue depth or a saturation signal exposed as a custom metric via Prometheus adapter or KEDA, with a scale-down stabilisation window. Because a new replica can take minutes to become ready, I would also keep a minimum replica count and some headroom rather than rely on reactive scaling alone.
Q: When would you choose something simpler than Kubernetes?
A: When the workload is a handful of services with modest scale and no one on the team owns cluster operations: a managed container service or a single server has far fewer failure modes than a cluster with a control plane, CNI, ingress controller, autoscalers and upgrades. Kubernetes pays off with many services, multi-team platforms, heterogeneous hardware like GPUs and a need for declarative rollouts and self-healing. A managed Kubernetes offering reduces, but does not remove, that operational burden.
Follow-up: A GPU Deployment rollout hangs with a new Pod Pending. What happened?
A: With maxSurge: 1 the update wants one extra Pod, but if every GPU is already allocated to old Pods the surge Pod cannot be scheduled, and old Pods will not be removed until a new one is ready. Either allow maxUnavailable: 1, maxSurge: 0 (capacity dips by one replica during the update) or keep a spare GPU for the surge. I would also check that the cluster autoscaler is allowed to add GPU nodes.
Follow-up: p99 latency spikes every few seconds but node CPU is only 40%. Why?
A: A CPU limit is enforced per scheduling period (commonly 100 ms) using a quota, so a multi-threaded burst can exhaust the quota early in the period and stall until the next one even when the node has idle cores. The throttling metrics (for example cgroup throttled periods, exposed by cAdvisor) confirm it. Raise or remove the CPU limit, keep the request accurate, or reduce thread count.
Follow-up: Your traffic doubles in 30 seconds and the HPA is configured correctly, yet users see errors for several minutes. Why?
A: HPA evaluates on an interval, needs metric data, then the new Pods must be scheduled (possibly waiting for a new node), pull a large image, load the model and pass readiness; for big models each step is measured in tens of seconds to minutes. In the meantime the existing replicas are overloaded. The fixes are headroom, a warm minimum, a lower scale-up threshold, request queuing with load shedding (see rate limiting and throttling) and faster cold starts.
In an LLM serving design, Kubernetes is the placement and lifecycle layer beneath the gateway: an ingress or API gateway terminates TLS and routes to Services fronting stateless API Pods on cheap CPU nodes, which call model-server Deployments on a tainted GPU node pool, with an HPA or KEDA scaling replicas on queue depth. When an interviewer asks for numbers, estimate in the open: assume a 7B-parameter model in 16-bit precision needs roughly 14 GB for weights plus KV cache, so it fits one 40 to 80 GB GPU with room for batching, which means one Pod per GPU and node count equals GPUs divided by GPUs per node; assume a 10 GB image and 15 GB of weights at an effective 500 MB/s gives about a minute of transfer alone, so a cold replica is realistically several minutes and your autoscaler cannot absorb a spike faster than that. All of this is ILLUSTRATIVE and must be replaced with measured values. The design should state a minimum replica floor across at least two zones (a zonal outage otherwise removes all capacity, see VPC and subnets for how subnets map to zones), PodDisruptionBudgets so node drains do not remove every replica at once, and an SLO-based sizing story linked to capacity planning and SLOs.
Kubernetes buys a uniform, declarative platform with self-healing, rolling updates, autoscaling and a huge ecosystem, at the price of a large surface area to operate: control-plane upgrades, networking plugins, ingress, RBAC, node pools, and a steep learning curve where most outages come from misconfigured requests, probes and rollouts rather than from the platform itself. For GPU workloads the costs are sharper: whole-GPU allocation wastes money on small models, bin-packing versus spreading trades utilisation against blast radius, and slow cold starts force you to pay for idle warm capacity. It is the wrong choice for a small team with one or two services, for latency-critical single-tenant workloads where a single well-tuned VM or a managed inference endpoint is simpler, or when nobody owns the platform. Even when it is right, treat manifests as code: pin image versions, set requests from measured usage, rehearse rollbacks, and load test the cold-start path, not just the steady state.