API Server Troubleshooting
In shortSymptoms, causes, and fixes for 10xGraph API server problems at startup, at request time, and in production deployments.
- 15 min read
- 8 sections
- Updated
- v1.0.0
- Markdown
Use this page when 10xgraph dev fails to start, rejects requests, or behaves differently in production than locally. Find your symptom under startup, request-time or production issues, then apply the fix. Each entry lists symptoms, the cause and a concrete fix verified against the server code.
Server troubleshooting map
flowchart TD
A[Server problem] --> B{Startup or request-time?}
B -->|Startup| C[Check config import, env, port, dependencies]
B -->|Request-time| D[Check auth, graph errors, persistence, logs]
C --> E[Apply fix]
D --> E
Startup issues
Issue: server does not start
Symptoms
- command exits immediately
- stack trace appears before the server binds
Likely causes
- invalid
10xgraph.json - graph import failure
- missing required environment variables or provider keys
Fix
- verify the config file path
- test the graph import manually:
python -c "from graph.react import app; print(app)" - verify required provider keys and dependencies are installed before startup
Issue: backend unreachable at startup
Symptoms
- the server exits at boot after about 30 seconds (
STARTUP_CHECK_TIMEOUT_SECONDS) with a message like:
Refusing to start: 1 backend(s) unreachable after 30s:
- cache (RedisCacheStore, cache_store.url, 127.0.0.1:6399, required by token_revocation): ConnectionError: Connection refused
Start the backend, fix its URL, or set STARTUP_CHECKS=false to skip this check.- each line names the component (
checkpointer,storeorcache) and its class, for examplecheckpointer (PgCheckpointer). The cache line also names where the setting came from (cache_store.urlorcache_store.path ...) and the feature that needs it (required by token_revocationorrequired by rate_limit.fail_open=false) - earlier log lines say
Startup check: ... is unreachable (attempt N)while it retries - only the checkpointer, the store and (when
cache_store.token_revocationis on, orcache_store.rate_limitis on withfail_openset tofalse) the cache store stop the boot. With any other setup an unreachable cache store logs one WARNING and the server starts, with locks and caps per process. A leftoverREDIS_URLtherefore never blocks a server that does not need it
Likely causes
- the backend is not running or not published on the host and port named in the message
- running
10xgraph devon the host with.envURLs that use Docker service names (postgres,redis,qdrant), which resolve only inside the Docker network - a wrong password, port or database in the connection string. The message shows
host:portonly, never credentials
Fix
- start the backend, or correct the URL. For Docker backends and
10xgraph devon the host, publish the ports on127.0.0.1and uselocalhostin.env(see Run10xgraph devagainst Docker backends) - raise
STARTUP_CHECK_TIMEOUT_SECONDSif the backends are slow to come up STARTUP_CHECKS=falseskips the check; use it only when you accept that a down backend shows up at the first request instead- with
10xgraph dev(reload on), the reloader keeps waiting for a file change after the failure; fix the cause and save a file, or restart the command - in production the orchestrator restarts the container until the backends are up. The same checks run behind
/health/ready, which answers 503 at runtime while the checkpointer or store is down (and the cache store, when a fail-closed feature uses it)
Issue: RuntimeError: Refusing to start: the following routes are not protected
Symptoms
- startup aborts with a
RuntimeErrorlisting one or more routes
RuntimeError: Refusing to start: the following routes are not protected by RequirePermission.
Add the dependency, or add the path to the public allowlist if it is intentionally open:
- POST /v1/my-new-endpointCause
Every non-public route must carry a RequirePermission guard. The check runs once at startup and walks the entire dependency subtree of every route. This is deliberate: a forgotten guard would otherwise ship a silently open endpoint.
Fix
Add the dependency to the handler (the import path is tenxgraph_api.src.app.core.auth.permissions):
from typing import Any
from fastapi import APIRouter, Depends
from tenxgraph_api.src.app.core.auth.permissions import RequirePermission
router = APIRouter()
@router.post("/v1/my-new-endpoint")
async def my_endpoint(
user: dict[str, Any] = Depends(RequirePermission("graph", "read")),
):
return {"ok": True}Only /ping is public. Modifying the public allowlist requires editing the source code frozenset, which is intentional. Do not use it to silence this error on a route that handles user data.
Issue: InsecureCorsConfigError on startup
Symptoms
- the server refuses to start in production with a message beginning “Refusing to start: CORS is configured with ORIGINS=‘*’”
Cause
ORIGINS="*" combined with CORS_ALLOW_CREDENTIALS=true while MODE=production. Starlette reflects the caller’s Origin back alongside Access-Control-Allow-Credentials: true, which makes every origin a trusted, credentialed one.
In development the same combination only logs a warning, so this failure usually appears the first time a working local config is promoted to production.
Fix
Pick one:
# Name the origins explicitly (the usual answer)
ORIGINS=https://yourapp.com,https://api.yourapp.com# Or serve a public, non-credentialed API (production also needs the strict checks off)
CORS_ALLOW_CREDENTIALS=false
STRICT_PRODUCTION_CHECKS=falseIssue: Refusing to start: MODE=production with insecure settings
Symptoms
- the server exits at startup in production and lists one or more lines that start with a warning sign
Cause
STRICT_PRODUCTION_CHECKS is on by default in production. It stops startup on a wildcard ALLOWED_HOST (including * with spaces around it), a wildcard ORIGINS, or an enabled DOCS_PATH or REDOCS_PATH. Generated .env.example, compose and Kubernetes files use placeholder values, so a file you did not edit also trips it.
Fix
- set
ALLOWED_HOSTandORIGINSto your real hosts and origins, and leaveDOCS_PATHandREDOCS_PATHempty - or set
STRICT_PRODUCTION_CHECKS=falseto start with warnings only
Issue: Invalid MODE on startup
Symptoms
- the server, or
10xgraph dev, exits with a message that starts withInvalid MODEand lists the accepted values 10xgraph devexits with code 2
Cause
MODE is trimmed and lowercased, then must be one of production, prod, development, dev or local. Anything else, such as prodution, is refused so a typo cannot run with the development defaults. Unset means development.
Fix
- correct the spelling, or remove the variable for development
Issue: runs fail with 504 or a runs.timeout error
Symptoms
- invoke returns
504withGraph run passed the runs.timeout limit of 300s and was cancelled - a stream, WebSocket or AG-UI run ends with an error chunk that carries the same text
Cause
In production, runs.timeout is 300 seconds when you do not set it. Development has no limit.
Fix
- set a larger number in
10xgraph.json, for example"runs": {"timeout": 900} - or set
"timeout": nullto remove the limit
Issue: port already in use
Symptoms
- startup fails with address already in use
Cause
- another process already listens on the chosen port
Fix
- change the port with
--port 8001 - stop the conflicting process:
lsof -i :8000 | tail -1 | awk '{print $2}' | xargs kill
Issue: auto-reload causes unstable behavior
Symptoms
- repeated restarts
- duplicate workers
- unstable behavior in Docker or remote filesystems
Cause
- reload watcher is not appropriate for that environment
Fix
10xgraph dev --no-reloadUse reload only for active local development.
Request-time issues
Issue: /ping works but graph routes fail
This is the most common request-time symptom: the server is up, but the graph fails when it runs. The production entry “deployment starts but requests fail immediately” below covers the deploy-specific causes.
Symptoms
- health check succeeds
/v1/graph/invokereturns 500 or import-related errors
Likely cause
- the server itself started, but graph dependencies fail when invoked (bad provider keys, network issues with external services, tool initialization errors)
Fix
- inspect runtime logs: the error appears there
- verify graph dependencies, provider keys, and tool integrations work outside the server
- test with a minimal invoke payload (e.g., no tools)
Issue: 403 Missing required permission: <resource>:<action>
Symptoms
- an authenticated user is rejected on a specific endpoint while others work
Cause
The caller’s permissions do not include this endpoint’s "<resource>:<action>" pair. With no roles configured, every authenticated caller holds everything except admin:*. With roles, the caller holds what their roles grant. A token scope or scopes claim can only narrow that set. A custom BaseAuth.authorize can also refuse.
A common case: adding a role table. A user whose role is absent from roles, with no default_role, holds no permissions and is denied everything.
Fix
- Add the permission to the role’s
permissionsin theauthobject, or setdefault_role - Confirm the token’s
roles_claimmatches a key in therolestable - Call
GET /v1/mewith the token to see the permissions and reach the server resolved
A 404 is a different problem: the thread, message, memory, file or run is not visible to the caller (unknown, or owned by someone else).
Issue: requests are unexpectedly public
Symptoms
- protected routes work without credentials
Cause
- auth is disabled in
10xgraph.jsonor config change did not reload into the running process
Fix
- set
authin10xgraph.json(for example"auth": "jwt") - set
JWT_SECRET_KEYin your environment - restart the server
- verify with an unauthenticated curl request:
curl -v http://localhost:8000/v1/graph/invoke 2>&1 | grep 401
Issue: WebSocket closes immediately
Symptoms
- the socket opens and closes before any application frame is exchanged
Cause and fix by close code
| Code | Cause | Fix |
|---|---|---|
1008 |
Auth or authorization rejected at the handshake, or the wrong socket for the graph type | Check the token transport; browsers should offer ["10xgraph-bearer", token] as the subprotocol. Call GET /v1/graph and read info.is_realtime to pick between /v1/graph/ws (turn-based) and /v1/graph/live (realtime). |
1013 |
The global rate limit or websocket.max_connections was exceeded |
Handshakes share the REST rate-limit bucket. Raise the limit, increase max_connections, or back off and retry. |
1003 |
Invalid init frame on /v1/graph/live: not JSON, or JSON that is not an object |
Send the init control frame as a JSON object first, before any audio |
1011 |
Unexpected server error during the session | Check server logs; this is not an auth or config rejection |
A 404 on POST /v1/ag-ui means ag_ui.enabled is not true in 10xgraph.json. The threads, store, and files routers are always mounted.
A 1008 accompanied by an error event with code: "not_live" or "THREAD_NOT_FOUND" tells you which of the two 1008 causes applies.
Issue: 422 with initial_state keys not allowed by this server
Symptoms
- invoke, stream, the WebSocket or AG-UI answers
422withinitial_state keys not allowed by this server: <key>. List them in runs.state_keys in 10xgraph.json to allow them PUT /v1/threads/{id}/stateanswers the same withstate keys not allowed ...
Cause
runs.state_keys denies by default. A client may set only the state keys it lists, and an unset or empty list allows none. AG-UI state follows the same rule.
Fix
- list the keys your client sends in
10xgraph.json:
{
"agent": "graph.react:app",
"runs": { "state_keys": ["preferences"] }
}- do not list a key that a tool or node trusts for access decisions, such as
role contextinPUTstate is always allowed and needs no entry- the setting can also be edited in
10xgraph config, under Client state keys
Issue: 429 Too Many Requests
Symptoms
- requests fail with status
429after a burst of calls - the body has
error.codeRATE_LIMIT_EXCEEDEDand the message ends withRetry after <n>s., or the message says there are too many runs in progress for this user
Cause
Two different limits answer 429:
- the
cache_store.rate_limitwindow: the response carriesRetry-After(seconds) andX-RateLimit-*headers. Behind a reverse proxy, every client may share one bucket iftrusted_proxiesis not set (see theX-Forwarded-Forentry below) - the per-user run cap
runs.max_concurrent_per_user(default 10): one verified user already has that many invoke, stream or AG-UI runs in flight. The body is the standard error envelope with codeHTTPExceptionand there is noRetry-Afterheader
Fix
- wait for the
Retry-Afterinterval, or for a running request to finish, then retry - raise
cache_store.rate_limit.requestsorruns.max_concurrent_per_userin10xgraph.jsonif the traffic is legitimate - set
trusted_proxieswhen the server sits behind a proxy - a rate limiter whose Redis is down answers
503 RATE_LIMIT_UNAVAILABLE(not 429) whenfail_openisfalse; see the entry below - on the TypeScript client this is a plain
TenxGraphErrorwithstatusCode429; see Error handling
Issue: 503 RATE_LIMIT_UNAVAILABLE
Symptoms
- requests fail with status
503,error.codeRATE_LIMIT_UNAVAILABLE, the messageRate limiter unavailable: its backend could not be reached.andRetry-After: 5
Cause
cache_store.rate_limit.fail_open is false and the API’s cache store (Redis by default) raised an error. This is an outage, not a limit hit; a real limit hit stays 429 RATE_LIMIT_EXCEEDED.
Fix
- restore the cache store (the Redis from
cache_store.urlorREDIS_URL) and retry afterRetry-After - clients that retry on 429 should retry on 503 too
- set
fail_opentotrueto let requests through during an outage./pingand/health/readyare never rate limited
Issue: the error message is generic in production
Symptoms
- a 5xx response says “An error occurred processing your request.” or “An error occurred executing the graph.” instead of the real cause
- the same request shows the real exception text locally
Cause
With MODE=production the server replaces the message of every 5xx error with a generic one, because exception text can contain internals. The full error is written to the server log with the request id. Messages of 4xx HTTPException errors (401, 403, 404, 413, 429) are written for the client and are kept. Request validation errors (422) use the generic text “The request data is invalid. Please check your input.” in production, except size-limit messages, which are kept. execution_meta.internal_data.error in GET /v1/threads/{id}/state also carries the generic text in production.
Fix
- take the
request_idfrom the response metadata and search the server log for it - reproduce in development (
MODE=development) to see the raw message - handled 4xx errors are logged at
WARNING(one line, no traceback), notERROR. If an alert rule matchedERRORfor 401 or 404 responses, change it to matchWARNING
Issue: thread endpoints fail
Symptoms
/v1/threadsreturns errors or empty results unexpectedly
Likely causes
- no checkpointer configured (the thread routes answer
503withCheckpointer is not configured) - checkpointer backend unavailable (Postgres, Redis, or SQLite)
- inconsistent
thread_idusage - a
thread_idthat does not exist, which answers404by default (runs.thread_idsisserver) - a state update whose
contextis not a list of messages, or that sets built-in fields or keys not listed inruns.state_keys(an empty list by default), which answers422 - a message id that is not in the thread, which answers
404(Message not found)
Fix
- pass a checkpointer to
graph.compile(checkpointer=...). The server uses that one; acheckpointerkey in10xgraph.jsonis not read - verify backend connectivity and health
- use the
thread_idthe server returned on the first call (do not generate a new one per request, and do not make one up: the server refuses ids it did not create)
For more details, see Checkpointing and threads.
Production and deployment issues
Issue: deployment starts but requests fail immediately
Symptoms
/pingworks but graph routes fail- first invoke request returns 500
- logs mention import or dependency errors
Likely causes
- graph import path is wrong in
10xgraph.json - environment variables required by the graph are missing
- production image does not include all dependencies
Fix
- verify the graph import works:
python -c "from graph.react import app; print(app)" - verify all deploy-time secrets and API keys are present
- verify the image or runtime installed all required Python packages (including provider extras like
google-genai)
Issue: threads vanish after restart
Symptoms
- conversation history works until the process restarts
/v1/threadsbecomes empty after deployment recycle
Likely cause
InMemoryCheckpointeris still being used
Fix
- switch to a durable shared checkpointer such as
PgCheckpointer - verify restart behavior before re-releasing
Issue: one replica sees thread history and another does not
Symptoms
- state appears inconsistent across instances
- one request remembers context, the next does not
Likely cause
- instances are not sharing the same persistence backend
Fix
- point all replicas to the same Postgres/Redis-backed checkpointer
- confirm the same
thread_idis being used by the caller - verify the backend (Postgres, Redis) is reachable from all replicas
Issue: auth works in curl but fails in browser clients
Symptoms
- curl with bearer token succeeds:
curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/v1/graph/invoke - frontend requests fail or never send credentials
Likely causes
- browser client is not attaching the auth header
- reverse proxy strips or corrupts the
Authorizationheader - CORS configuration blocks browser requests
Fix
- inspect the browser network tab to confirm the header is sent
- verify frontend client config (token is set correctly)
- verify reverse proxy configuration forwards
Authorizationheader unchanged - verify
ORIGINSincludes the real frontend origin - confirm
CORS_ALLOW_CREDENTIALS=trueif the frontend sends credentials
Issue: X-Forwarded-For is ignored behind a proxy
Symptoms
- rate limiting sees the proxy’s IP instead of the client’s
- requests through a reverse proxy are affected
Cause
Rate limiting ignores X-Forwarded-For unless you opt in. cache_store.rate_limit.trusted_proxy_headers defaults to false, so behind a proxy every client appears to share the proxy’s IP.
Fix
Enable the option in 10xgraph.json only when the app sits behind a proxy you control, and list the proxy networks so direct clients cannot spoof the header:
{
"cache_store": {
"rate_limit": {
"trusted_proxy_headers": true,
"trusted_proxy_hops": 1,
"trusted_proxies": ["10.0.0.0/8"]
}
}
}trusted_proxies takes the IPs or CIDR ranges the proxy connects from. X-Forwarded-For is ignored for any other source. See Configure the server for the full cache_store.rate_limit block.
Issue: requests time out only in production
Symptoms
- local requests are fine
- deployed requests are slow or timing out
Likely causes
- external tools or providers are slower in the deployed environment
- reverse proxy timeouts are too aggressive
- graph is making too many sequential tool calls
Fix
- inspect server logs for slow nodes or tools
- tune reverse proxy timeout settings (for example Nginx
proxy_connect_timeout,proxy_send_timeout,proxy_read_timeout) - prefer streaming where appropriate
- reduce expensive tool-call chains if possible
Issue: production deployment exposes too much
Symptoms
- the Swagger and ReDoc pages (default paths
/docsand/redocs) are publicly reachable - cross-origin browser access is broader than intended
Likely causes
DOCS_PATHorREDOCS_PATHset explicitly (in production they default to empty, which disables the pages and the OpenAPI schema; the server logs a warning when either is set)MODEis notproduction, so the development defaults applyORIGINS=*still set (production refuses it unlessSTRICT_PRODUCTION_CHECKS=false)
Fix
- set
MODE=productionand remove explicitDOCS_PATHandREDOCS_PATHvalues, or set both to empty strings - replace wildcard origins with explicit domains:
ORIGINS=https://yourapp.com,https://api.yourapp.com
Issue: 10xgraph dev works locally but deployed users cannot connect
Symptoms
- local playground sessions are fine
- deployed frontend or shared users fail to connect reliably
Likely cause
10xgraph devwas used as a testing tool, but the deployed system needs a proper hosted API endpoint and browser-safe networking setup
Fix
- deploy the image from
10xgraph build, which runs Gunicorn, behind HTTPS and with correct CORS and auth settings - treat
10xgraph dev(a Uvicorn development server with reload) as an interactive test path, not the deployment architecture.10xgraph apiand10xgraph playno longer exist (removed in 0.9.0)
Error codes
For a complete list of error codes and their meanings, see Error codes reference.
Related docs
- Run the API Server
- Configure the server
- Auth and Authorization
- Observability and logging
- Checkpointing and threads
What you learned
- How to separate startup failures from request-time failures
- How to trace common API issues back to config, reload mode, auth, or persistence
- How to troubleshoot production failures by isolating runtime, config, network, auth, and persistence layers
Frequently asked questions
- Why does the server refuse to start with "routes are not protected"?
- Every route except /ping and the two eval report routes must carry a RequirePermission guard. The check runs at startup so a forgotten guard cannot ship as an open endpoint. Add the dependency to the handler.
- Why do /ping and the health check pass but graph requests fail?
- The server started, but the graph fails when invoked. Read the runtime logs for bad provider keys, missing packages or tool errors.
- Why do threads disappear after a restart?
- The in-memory checkpointer is still in use. Switch to a durable checkpointer such as PgCheckpointer.