API Server Troubleshooting

Use this page when agentflow api starts incorrectly, crashes, or serves requests unreliably.

Server troubleshooting map

flowchart TD
    A[Server problem] --> B{Startup or request-time?}
    B -->|Startup| C[Check config import, env, port, dependencies]
    B -->|Request-time| D[Check auth, graph errors, persistence, logs]
    C --> E[Apply fix]
    D --> E

Issue: server does not start

Symptoms

• command exits immediately
• stack trace appears before the server binds

Likely causes

• invalid 10xgraph.json
• graph import failure
• missing required environment variables

Fix

• verify the config file path
• test the graph import manually
• verify required provider keys and dependencies are set before startup

Issue: RuntimeError: Refusing to start: the following routes are not protected

Symptoms

• startup aborts with a RuntimeError listing one or more routes

RuntimeError: Refusing to start: the following routes are not protected by RequirePermission.
Add the dependency, or add the path to the public allowlist if it is intentionally open:
  - POST /v1/my-new-endpoint

Cause

Every non-public route must carry a RequirePermission guard. The check runs once at startup and walks the whole dependency subtree of every route. This is deliberate: a forgotten guard would otherwise ship a silently open endpoint.

Fix

Add the dependency to the handler:

@router.post("/v1/my-new-endpoint")
async def my_endpoint(
    user: dict[str, Any] = Depends(RequirePermission("graph", "read")),
):
    ...

Only three paths are public: /ping, /v1/evals/runs, and /v1/evals/runs/{run_id}. Adding to that allowlist means editing the frozenset in the source, which is intentional. Do not reach for it to silence this error on a route that handles user data.

Issue: InsecureCorsConfigError on startup

Symptoms

• the server refuses to start in production with a message about CORS and credentials

Cause

ORIGINS="*" combined with CORS_ALLOW_CREDENTIALS=true while MODE=production. Starlette reflects the caller’s Origin back alongside Access-Control-Allow-Credentials: true, which makes every origin a trusted, credentialed one. Warning about it would not have helped, since the server would still have served it.

In development the same combination only logs a warning, so this failure usually appears the first time a working local config is promoted.

Fix

Pick one:

# Name the origins explicitly (the usual answer)
ORIGINS=https://yourapp.com,https://api.yourapp.com

# Or serve a public, non-credentialed API
CORS_ALLOW_CREDENTIALS=false

Issue: 403 Missing required scope: <resource>:<action>

Symptoms

• an authenticated user is rejected on a specific endpoint while others work

Cause

The identity carries a scope list that does not include this endpoint’s "<resource>:<action>" pair. Scopes resolve from the authorization backend’s scopes_for first, and only fall back to the token’s own scopes claim when that returns None.

A common version of this: switching to the RBAC backend. RoleBasedAuthorizationBackend always returns a list, never None, so a user whose role is absent from the roles table and for whom default_scopes is empty resolves to [] and is denied everything. An empty list is not the same as no scopes.

Fix

• Add the pair to the role’s scopes, or to default_scopes
• Confirm the user’s roles or role claim actually matches a key in the roles table
• To go back to unrestricted, ensure scopes_for returns None rather than []

Issue: WebSocket closes immediately

Symptoms

• the socket opens and closes before any application frame is exchanged

Cause and fix by close code

A 404 on /v1/evals/* means the server runs with MODE=production: the eval report viewer has no auth, so it is mounted only outside production. A 404 on POST /v1/ag-ui means ag_ui.enabled is not true. The threads, store, and files routers are always mounted.

A 1008 accompanied by an error event with code: "not_live" or "not_authorized" tells you which of the two 1008 causes applies.

Issue: port already in use

Symptoms

• startup fails with address already in use

Cause

• another process already listens on the chosen port

Fix

• change the port
• stop the conflicting process

Issue: /ping works but graph routes fail

Symptoms

• health check succeeds
• /v1/graph/invoke returns 500 or import-related errors

Likely cause

• the server itself started, but graph dependencies fail when invoked

Fix

• inspect runtime logs
• verify graph dependencies, provider keys, and tool integrations
• reproduce with a minimal invoke payload

Issue: auto-reload causes unstable behavior

Symptoms

• repeated restarts
• duplicate workers
• unstable behavior in Docker or remote filesystems

Cause

• reload watcher is not appropriate for that environment

Fix

agentflow api --no-reload

Use reload only for active local development.

Issue: thread endpoints fail

Symptoms

• /v1/threads returns errors or empty results unexpectedly

Likely causes

• no checkpointer configured
• checkpointer backend unavailable
• inconsistent thread_id usage

Error codes: STORAGE_NOT_FOUND_000, STORAGE_TRANSIENT_000

Fix

• configure a checkpointer
• verify backend connectivity
• use a stable thread_id

Error Code Quick Reference

See Error Codes Reference for full documentation.

Issue: requests are unexpectedly public

Symptoms

• protected routes work without credentials

Cause

• auth is disabled or config change did not reload into the running process

Fix

• set auth in 10xgraph.json
• restart the server
• verify with an unauthenticated curl request

Related docs

• Run the API Server
• Production Troubleshooting
• Auth and Authorization

What you learned

• How to separate startup failures from request-time failures.
• How to trace common API issues back to config, reload mode, auth, or persistence.