Deploy on Kubernetes

The problem: long runs and short timeouts

A typical agent invocation chains several operations: the client calls the server, the server awaits an LLM response (up to 10 minutes for some models), the LLM might trigger tool calls, tools execute, results return to the LLM, and the response streams back to the client. The entire request can stay open for 20 minutes or more.

Kubernetes’ default pod lifecycle is hostile to this. When you trigger a rolling deploy, the kubelet gives a pod 30 seconds to shut down gracefully before sending SIGKILL. If the app is still in an LLM call, the pod dies mid-request, and the client’s stream breaks.

10xgraph build --k8s generates a manifest that fixes this: it sets the termination grace period, pre-termination delay, and probe timeouts so that in-flight runs finish normally, even during deploys.

Generate the manifest

Create a Dockerfile and Kubernetes manifest in one command:

10xgraph build --k8s --service-name my-agent --port 8000

This produces three files:

• Dockerfile with a production-ready image that starts Gunicorn with a 600-second graceful timeout
• .dockerignore
• k8s.yaml containing a Deployment and a Service

The --force flag overwrites existing files. Do not add --docker-compose for an image you will run on Kubernetes: with that flag the Dockerfile omits its CMD and relies on the compose file’s command. Generate the compose file in a separate run if you need it.

What the generated manifest includes

The manifest is a working skeleton. It sets these values, which are easy to get wrong:

The readiness probe is strict (10-second intervals) because false positives only temporarily remove a pod from rotation. The liveness probe is slack (30-second intervals, 5 consecutive failures required) because a false positive kills the pod mid-run. Both hit the /ping endpoint, which does not require authentication.

Default resource requests and limits are also set (500m CPU and 512 Mi memory requested; 2 CPU and 2 Gi memory as limits), but you should tune these based on your agents and deployment environment.

Customize before applying

The manifest is not production-ready as-is. You must edit these values:

containers:
  - name: my-agent
    image: 10xgraph-api:latest        # Replace with your own pinned image reference
    env:
      - name: ORIGINS
        value: "https://your-frontend.example.com"   # Set to your actual origin
      - name: MODE
        value: "production"             # Already set, but verify
      - name: IS_DEBUG
        value: "false"                  # Already set, but verify

Pin the image

The manifest uses 10xgraph-api:latest as a placeholder image name. Build your own image from the generated Dockerfile, push it to your registry, and reference it immutably. A moving latest tag makes rollbacks ambiguous and makes image-related issues hard to debug:

• A digest: my-registry/my-agent@sha256:abc123...
• A version tag: my-registry/my-agent:1.4.0
• A git-based tag: my-registry/my-agent:git-abc123def

Set CORS origins explicitly

The manifest sets ORIGINS to the placeholder https://your-frontend.example.com. Replace it with your actual frontend domain(s), comma-separated if there are multiple:

- name: ORIGINS
  value: "https://app.example.com,https://app-staging.example.com"

With MODE=production, the server refuses to start if ORIGINS is the wildcard * while credentials are enabled, which is the default (CORS_ALLOW_CREDENTIALS=true). If you truly need wildcard origins, set CORS_ALLOW_CREDENTIALS=false, which is rare.

Mount secrets, do not bake them

API keys, database passwords, and JWT secrets must come from Kubernetes Secrets, never from plain values in the manifest or baked into the image. The generated manifest does not reference a Secret, so add an envFrom entry to the container yourself:

envFrom:
  - secretRef:
      name: my-agent-secrets

Create the secret in your cluster before deploying. The server reads JWT_SECRET_KEY, JWT_ALGORITHM and REDIS_URL itself; names such as the Postgres DSN variable below are whatever your graph module reads when it builds the checkpointer:

kubectl create secret generic my-agent-secrets \
  --from-literal=GOOGLE_API_KEY="..." \
  --from-literal=OPENAI_API_KEY="..." \
  --from-literal=JWT_SECRET_KEY="..." \
  --from-literal=JWT_ALGORITHM="HS256" \
  --from-literal=POSTGRES_DSN="postgresql://..." \
  --from-literal=REDIS_URL="redis://..."

Alternatively, use your favorite secrets management tool (Sealed Secrets, External Secrets, Vault, etc.) to manage these safely.

Apply and verify

Once you have customized the manifest, apply it:

kubectl apply -f k8s.yaml

Monitor the rollout:

kubectl rollout status deployment/my-agent
kubectl get pods -l app=my-agent

Verify the service is reachable:

kubectl port-forward svc/my-agent 8000:80
curl http://127.0.0.1:8000/ping

If /ping returns a JSON body with "data": "pong", the deployment is live.

Test graceful termination

The true test is that a long-running request survives a deploy. Start a streaming request:

# Terminal 1: Stream a request (this will take minutes)
curl -X POST http://127.0.0.1:8000/v1/graph/stream \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": [{"type": "text", "text": "explain quantum physics in detail"}]}]}'

While that request is in flight, trigger a rolling restart in another terminal:

# Terminal 2: Rolling restart
kubectl rollout restart deployment/my-agent

The stream should continue and complete normally. If the connection closes or times out, one of these is true:

• terminationGracePeriodSeconds is shorter than your longest run
• The container is not running as PID 1 and not receiving SIGTERM
• Your load balancer’s idle timeout is shorter than the grace period

Scaling horizontally

The manifest defaults to 2 replicas. Scale up for higher concurrency:

kubectl scale deployment my-agent --replicas=5

Before scaling, understand how 10xgraph handles distributed state:

1. Shared checkpointer is mandatory. All replicas must read and write to the same Postgres and Redis. If they do not, the same thread will resolve differently depending on which pod answers, causing data loss. Use PgCheckpointer in production (see Set up checkpointing).

2. Threads are not sticky. State lives in the checkpointer, not in the pod. You do not need (and should not use) session affinity. A client can invoke a thread on any replica.

3. Streaming connections are long-lived. If your load balancer (or ingress) has an idle timeout shorter than your longest run, the connection will close mid-stream even though the pod is healthy. Set idle timeouts to at least 15 minutes. For AWS ALB/NLB, increase deregistration_delay. For Nginx, increase proxy_read_timeout.

4. Concurrent writes conflict. With optimistic concurrency, two overlapping writes to the same thread raise a StaleStateError, and the API returns 409. Clients that fan out against a single thread_id should retry or serialize. Most clients do not do this; it only matters if you deliberately parallelize writes to the same thread.

Horizontal Pod Autoscaling (HPA)

For autoscaling, create an HPA resource:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-agent
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-agent
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 900  # Give long runs time to drain

CPU is the provided metric, but it is a poor signal for agent workloads. Agents are typically latency-bound on the model provider, not compute-bound. Scale on queue depth (requests pending in your ingress) or request concurrency (active requests per pod) instead, which your ingress or load balancer can expose.

Troubleshooting deployments

Related

• Deployment for the general container strategy and production decisions
• Backup and restore for Postgres and Redis backup procedures
• Observability for logging, metrics, and traces in production