Deploy on Kubernetes

Agent runs are long. A single request can hold an LLM call, several tool calls, and a stream open for minutes. That makes the default Kubernetes lifecycle hostile: a 30-second termination grace period kills a pod mid-run on every rolling deploy.

10xgraph build --k8s generates a manifest with those numbers already set correctly.

Prerequisites

• A working container image
• A cluster and kubectl context
• Shared Redis and Postgres, reachable from the cluster. Multiple replicas without shared persistence is the most common production mistake. See checkpointing.

Generate the manifest

10xgraph build --docker-compose --k8s --service-name my-agent --port 8000

This writes k8s.yaml next to the Dockerfile, containing a Deployment and a Service. Use --force to regenerate over an existing file.

The generated Deployment sets three things that matter and are easy to get wrong:

The readiness probe is what takes a pod out of rotation, and it is tighter (periodSeconds: 10) because that is the safe direction to be aggressive in. Both probes hit /ping.

What you must edit before applying

The generated manifest is a correct skeleton, not a finished deployment. Change these:

image: agentflow-cli:latest        # -> your registry and an immutable tag
env:
  - name: ORIGINS
    value: "https://your-frontend.example.com"   # -> your real origin

• Pin the image. latest makes rollbacks meaningless. Use a digest or a version tag.
• Set real origins. Production refuses to start with wildcard CORS and credentials enabled.
• Add your secrets. The manifest carries no API keys by design. Mount them from a Secret, never bake them into the image:

envFrom:
  - secretRef:
      name: agentflow-secrets

with, for example, GOOGLE_API_KEY, OPENAI_API_KEY, JWT_SECRET_KEY, DATABASE_URL, and REDIS_URL. See environment variables.

Apply and verify

kubectl apply -f k8s.yaml
kubectl rollout status deployment/my-agent
kubectl port-forward svc/my-agent 8000:80
curl http://127.0.0.1:8000/ping

Then verify the property that actually matters: that a deploy does not truncate a run.

# Start a long streaming run against the service, then in another shell:
kubectl rollout restart deployment/my-agent

The in-flight stream should finish normally. If it dies, your grace period is shorter than the run, or the app is not receiving SIGTERM as PID 1.

Scaling

The default is replicas: 2. Before raising it:

1. Shared state is mandatory. Every replica must point at the same Postgres and Redis, or a thread will resolve differently depending on which pod answers.
2. Threads are not sticky and do not need to be. State lives in the checkpointer, not in the pod. You do not need session affinity for REST calls.
3. Streaming connections are long-lived. Set your ingress and load balancer idle timeouts above your longest expected run, or the proxy will cut the stream even though the pod is healthy.
4. Concurrent writes to one thread now conflict loudly. With optimistic concurrency, the losing write raises StaleStateError and the API returns 409. Clients that fan out against one thread_id should serialise or retry.

For horizontal autoscaling, scale on CPU only if your agents are compute-bound. Most are latency-bound on the model provider, so queue depth or request concurrency is the better signal.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-agent
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-agent
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      # Long runs need time to drain before a pod disappears.
      stabilizationWindowSeconds: 900

Common mistakes

Related

• Deployment for the general production model
• Generate Docker files
• Backup and restore
• Logging and metrics