Scaling and reliability

You scale a served 10xGraph agent by running more stateless API workers behind a load balancer, with Postgres for durable state and a shared cache store (Redis by default) for the things workers must share: locks, caps, rate limits and caches. Without one, those features still work but count per process. This page lists exactly what the cache store shares, how reliability works under failure, and where the limits are. For the request path and state layers, read Architecture first.

What the cache store shares across workers

The API keeps its shared state in one BaseCacheStore. Set cache_store.type to redis in 10xgraph.json to use RedisCacheStore (the URL comes from cache_store.url or REDIS_URL; Valkey, Dragonfly and KeyDB work through the same URL), or to custom with a path to your own class. With no cache_store the store is InMemoryCacheStore, in one process; REDIS_URL alone does not select Redis. See Bring your own backend. The tables below say “Redis” for the shared case.

Feature	With Redis	Without Redis (per process)
Thread ownership cache	L1 in each worker (30 s), L2 shared in Redis (1 h). Only known owners are cached, never “no owner”. Deleting a thread evicts both and leaves a tombstone for 30 s so another user cannot claim the id while a worker still holds the old owner.	L1 only; the tombstone exists only on the worker that handled the delete.
Rate limit	One sliding window for all workers.	Each worker counts alone, so the real limit is requests x workers.
Per-thread run lock	One-slot lease shared by all workers. Renewed while the run is alive, expires if the worker dies.	Lock holds only for runs on the same worker. Two workers can run one thread at once, and the Postgres version check then rejects the later write with a 409.
Per-user run cap (runs.max_concurrent_per_user)	Leases shared by all workers.	Counted per process: up to limit x workers.
WebSocket caps (websocket.max_connections, max_connections_per_user)	Shared through the same lease limiter.	Counted per process.
Idempotency (running lease and stored records)	Shared, so a retry on another worker sees the first request.	Seen only by the worker that handled the first request.
Token revocation	cache_store.token_revocation: true: a revoked token is refused on every worker.	A revocation reaches only the process that holds it, and the server warns about it in production or when WEB_CONCURRENCY is above 1. 10xgraph revoke refuses to run against an in-memory store.

The leases expire on their own, so a crashed worker cannot hold a thread or a user’s slots forever.

Workers and sizing

10xgraph build generates a Dockerfile that runs Gunicorn with Uvicorn workers. The worker count comes from WEB_CONCURRENCY, set to 2 in the image; override it at deploy time, for example -e WEB_CONCURRENCY=8. The image runs TenxGraphUvicornWorker and sets a graceful timeout of 600 seconds and a worker timeout of 660 seconds, because an agent run can last minutes and the default 30 seconds would kill runs during a rolling deploy. With --k8s, the generated manifest sets terminationGracePeriodSeconds above that and a preStop sleep so the load balancer stops sending new traffic first.

Each in-flight run holds a connection open and waits on model calls, so size and autoscale on concurrent runs, not CPU alone. Choose the worker count and the per-user cap together: without a shared cache store the cap is per worker.

Snowflake IDs

By default the framework generates IDs with UUIDs. If you want time-ordered numeric IDs, pass a SnowFlakeIdGenerator (from tenxgraph_api, needs the snowflakekit package) to StateGraph(id_generator=...). When several instances generate IDs, give each a unique SNOWFLAKE_NODE_ID and SNOWFLAKE_WORKER_ID so two instances never produce the same ID. See Environment variables.

Events out: publishers

A publisher sends run events (node start and end, tool calls, errors) to systems outside the worker, so you can build dashboards, audit logs and alerts without touching the graph. Core ships Redis Pub/Sub, Kafka, RabbitMQ and console publishers, plus an OpenTelemetry publisher for traces (also Logfire and LangSmith through it). Several can run at once through a composite publisher. On shutdown the server waits up to 30 seconds for in-flight background work, such as event publishing, then exits. See Events and observability.

Reliability mechanisms

Mechanism	What it guarantees
Replay-safe tool ledger	A tool call recorded in the checkpointer’s ledger is not executed again when a run resumes after a crash. A crash between the tool finishing and its record being written can still repeat it, so pass idempotency keys to payment and email providers. Details
Versioned state writes	With PgCheckpointer, each write is a new version under a row lock. A write based on a stale version is rejected, and the API returns 409 state_conflict instead of silently discarding another run’s work.
Per-thread run lock	One run per thread, so a double click or a second tab does not interleave history.
Timeouts	runs.timeout bounds a whole run. Set node_timeout and tool_timeout (no default) so one stuck tool cannot hold a run forever.
Parallel tool cap	Tools run in parallel up to 10 by default. Change it with runs.max_parallel_tool_calls or TENXGRAPH_MAX_PARALLEL_TOOL_CALLS.
Graceful shutdown	On SIGTERM the worker marks the app as draining, so /health/ready answers 503 and the load balancer stops sending new requests. Open runs get up to TENXGRAPH_GRACEFUL_SHUTDOWN_SECONDS (540 by default) to finish. Open streams then end with a final “interrupted” chunk. The app calls graph.aclose() with SHUTDOWN_GRACE_SECONDS, SHUTDOWN_STOP_SECONDS and SHUTDOWN_TIMEOUT_SECONDS (10, 10 and 30 seconds by default), which also drains background work, and closes limiters, revocation and authorization backends. A run that was cut off leaves a thread the next message can use. See Graceful shutdown.
Readiness probe	GET /health/ready calls aping() on the checkpointer, the store and the cache store (each time-boxed to 2 s, cached for 3 s) and returns 200 or 503. The response keys are checkpointer, store and cache. The cache store fails readiness only when token revocation or a fail-closed rate limit uses it. It is public and returns only ok or fail per component. /ping is the liveness probe.

Failure behaviour

Failure	What happens
Redis down: rate limit	Follows rate_limit.fail_open. Default true allows the request; set false to answer 503 RATE_LIMIT_UNAVAILABLE with Retry-After: 5.
Redis down: run lock, run cap, WebSocket caps, idempotency	Fail open. The request proceeds and the error is logged, because refusing every request would be an outage of its own. The Postgres version check still protects state.
Redis down: ownership	Falls back to the checkpointer lookup in the database. A request is never failed by the cache.
Redis clock	Lease expiry for the run cap uses the Redis clock (TIME), not each pod’s own, so clock skew between pods cannot trim a live lease.
Redis not answering	Every call fails after REDIS_SOCKET_TIMEOUT (5 seconds by default) instead of hanging the request. See environment variables.
Redis down: token revocation	Fails closed: a 503, not a possibly revoked token served.
Redis down: readiness	/health/ready reports cache: fail. It returns 503 only when token revocation or a fail-closed rate limit uses the cache store; otherwise the pod stays in rotation.
Postgres down	Runs that need the checkpointer fail. /health/ready returns 503, so Kubernetes stops routing to the pod. /ping stays 200 so the pod is not restarted for a dependency outage.
Worker dies mid-run	Locks and caps held in Redis expire. Resume by sending the thread another request; finished tool calls are skipped.

Honest limits

• Runs execute in the worker that received the request. There is no separate queue or worker tier, so a long run ties up that worker’s connection and a deploy waits on the graceful timeout.
• A tool can still run twice if a run crashes in the short gap between the tool finishing and its ledger record being written, or if the run was cut off while the tool was running. Use idempotency keys for tools with side effects.
• With the in-memory cache store, every cap and lock in the table above is per process.
• There are no published benchmarks for throughput or latency. Load-test your own graph, model and tools.
• Pin versions and read the changelog before upgrading. 1.0 is a release candidate until the final ships; from 1.0 breaking changes come only in a major release.

Related

• Architecture
• Production checklist
• Kubernetes
• Rate limiting