Kubernetes is the natural home for agentic workloads, but agents break several assumptions baked into the typical stateless web deployment. They run for minutes not milliseconds, they burst unpredictably, and if you self-host models they demand scarce, expensive GPUs. This is a reference architecture that survives real production load.
Why agents stress a normal deployment
A standard web service handles short, stateless, uniform requests. Agent workloads violate all three assumptions. A single run can execute for minutes across many LLM calls and tool invocations. State must persist across steps and survive a pod restart. Load is spiky and correlated — a batch job or a viral moment can 50x your traffic in seconds.
Design for these properties explicitly. Retrofitting them onto a request-response mindset is how teams end up with dropped runs, OOM-killed pods, and GPU bills that make finance call a meeting.
Separate the control plane from inference
Split your architecture into two tiers. The agent orchestration layer — the LangGraph runtime, tool execution, business logic — is CPU-bound and runs on standard nodes. The model inference layer, if you self-host, is GPU-bound and runs on a dedicated GPU node pool. Coupling them wastes expensive GPU capacity on CPU work and makes scaling clumsy.
This separation lets each tier scale on its own signal and its own hardware. Your orchestration pods scale on queue depth; your inference pods scale on GPU utilization and token throughput. It also lets you swap a self-hosted model for a managed API (Bedrock, Azure OpenAI) behind the same internal interface without touching orchestration.
Run long tasks as jobs, not request handlers
Do not hold an HTTP connection open for a five-minute agent run. Accept the request, enqueue the work, and return a handle the client can poll or subscribe to. Process runs from the queue with worker pods, and persist state to a durable checkpointer (Postgres) after each step so a pod eviction resumes rather than restarts.
This asynchronous, queue-backed pattern is what makes agents robust to the routine chaos of Kubernetes — rolling deploys, node autoscaling, spot instance reclamation. Combine it with a message queue or a durable task system and graceful shutdown handling so in-flight work drains cleanly on termination.
- Use a queue (SQS, Redis streams, or a task framework) to decouple request intake from execution.
- Honor SIGTERM: stop accepting new work, checkpoint current step, and exit within the termination grace period.
- Set idempotency keys so a resumed run never repeats a side effect.
GPU scheduling and serving
If you self-host models, use a purpose-built inference server — vLLM is the common choice — for continuous batching and PagedAttention, which together lift GPU throughput dramatically over naive serving. Run it on a tainted GPU node pool so only workloads that tolerate the taint land there, and request GPUs explicitly through the device plugin.
GPUs are your dominant cost, so drive utilization up. Continuous batching packs concurrent requests into the same forward pass. For multiple smaller models, GPU time-slicing or MIG partitioning shares a card across workloads. Keep a small warm pool to absorb bursts without cold-start latency, and autoscale the rest.
Autoscaling on the right signal
CPU utilization is the wrong scaling signal for both tiers. Scale the orchestration layer on queue depth or pending-run count using KEDA, which scales on external metrics and can scale to zero when idle. Scale the inference layer on GPU-relevant metrics — utilization, batch size, or time-to-first-token latency — exported to Prometheus.
Pair pod autoscaling with cluster autoscaler or Karpenter for node capacity, and prefer spot or preemptible GPU instances for interruption-tolerant batch work to cut cost substantially. Because your runs are checkpointed and queue-backed, a reclaimed spot node is an inconvenience, not a data-loss event.
Reliability, cost, and observability
Protect against the failure modes unique to this workload. Set token and step budgets per run so a looping agent cannot drain your budget. Add circuit breakers around model and tool calls so a downstream outage sheds load instead of piling up retries. Set resource limits carefully — LLM serving is memory-hungry and OOM kills are brutal.
Instrument cost as a first-class metric. Track tokens and GPU-seconds per run and per feature, not just aggregate spend, so you can see which workflow is expensive and why. Export traces (OpenTelemetry) across the orchestration and inference tiers so a slow run is debuggable end to end.
- Per-run token and step budgets to bound runaway cost.
- Circuit breakers and retry-with-backoff on every external model and tool call.
- Per-feature cost attribution and distributed tracing across both tiers.
A pragmatic starting point
You do not need this full architecture on day one, and you probably should not self-host models at all until scale justifies it. Start with a managed model API and a simple queue-backed worker deployment. Add the GPU tier only when your token volume makes self-hosting genuinely cheaper, which is a real inflection point but a later one than most teams assume.
Build the queue, checkpointing, and observability from the start, though — those are architectural decisions that are painful to retrofit. The GPU sophistication can come when the economics demand it.
Key takeaways
- 1.Separate CPU-bound orchestration from GPU-bound inference so each scales on its own signal and hardware.
- 2.Run agent tasks asynchronously from a queue with durable checkpointing to survive pod churn.
- 3.Serve self-hosted models with vLLM on a tainted GPU node pool and maximize utilization with continuous batching.
- 4.Autoscale on queue depth (KEDA) and GPU metrics, not CPU, and use spot GPUs for interruption-tolerant work.
- 5.Enforce per-run token budgets, circuit breakers, and per-feature cost tracing from the start.