# Production Troubleshooting

> A production-first troubleshooting guide for common 10xGraph deployment, runtime, and connectivity issues.

Source: https://10xgraph.com/docs/how-to/production/troubleshooting
Last updated: 2026-06-16

This page focuses on issues that usually appear after an agent leaves local development: deployment failures, state drift, auth mismatches, and cross-service connectivity problems.

If you need a narrower troubleshooting guide, use the dedicated pages:

- [Installation](/docs/troubleshooting/installation)
- [API Server](/docs/troubleshooting/api-server)
- [Client](/docs/troubleshooting/client)
- [Playground](/docs/troubleshooting/playground)

## Production troubleshooting workflow

```mermaid
flowchart TD
    A[Symptom appears] --> B[Confirm exact failing endpoint or client action]
    B --> C[Check server logs and runtime config]
    C --> D[Check auth, CORS, and environment variables]
    D --> E[Check persistence backends]
    E --> F[Check reverse proxy or network path]
    F --> G[Apply fix and re-verify]
```

## Issue: deployment starts but requests fail immediately

**Symptoms**

- `/ping` works but graph routes fail
- first invoke request returns 500
- logs mention import or dependency errors

**Likely causes**

- graph import path is wrong in `10xgraph.json`
- environment variables required by the graph are missing
- production image does not include all dependencies

**Fix**

- verify `python -c "from graph.react import app; print(app)"`
- verify deploy-time secrets are present
- verify the image or runtime installed all required Python packages

## Issue: threads vanish after restart

**Symptoms**

- conversation history works until the process restarts
- `/v1/threads` becomes empty after deployment recycle

**Likely cause**

- `InMemoryCheckpointer` is still being used

**Fix**

- switch to a durable shared checkpointer such as `PgCheckpointer`
- verify restart behavior before re-releasing

## Issue: one replica sees thread history and another does not

**Symptoms**

- state appears inconsistent across instances
- one request remembers context, the next does not

**Likely cause**

- instances are not sharing the same persistence backend

**Fix**

- point all replicas to the same Postgres/Redis-backed checkpointer
- confirm the same `thread_id` is being used by the caller

## Issue: auth works in curl but fails in browser clients

**Symptoms**

- curl with bearer token succeeds
- frontend requests fail or never send credentials

**Likely causes**

- browser client is not attaching the auth header
- proxy strips `Authorization`
- CORS configuration blocks browser requests

**Fix**

- inspect the browser network tab
- verify frontend client config
- verify proxy forwards `Authorization`
- verify `ORIGINS` includes the real frontend origin

## Issue: production deployment exposes too much

**Symptoms**

- `/docs` and `/redoc` are publicly reachable
- cross-origin browser access is broader than intended

**Likely causes**

- `DOCS_PATH` / `REDOCS_PATH` still enabled
- `ORIGINS=*` still set

**Fix**

- disable docs endpoints or restrict exposure intentionally
- replace wildcard origins with explicit domains

## Issue: requests time out only in production

**Symptoms**

- local requests are fine
- deployed requests are slow or timing out

**Likely causes**

- external tools or providers are slower in the deployed environment
- reverse proxy timeouts are too aggressive
- graph is making too many sequential calls

**Fix**

- inspect server logs for slow nodes or tools
- tune proxy timeout settings
- prefer streaming where appropriate
- reduce expensive tool-call chains if possible

## Issue: `10xgraph play` works locally but deployed users cannot connect

**Symptoms**

- local playground sessions are fine
- deployed frontend or shared users fail to connect reliably

**Likely cause**

- `10xgraph play` was used as a testing tool, but the deployed system needs a proper hosted API endpoint and browser-safe networking setup

**Fix**

- deploy with `10xgraph api` behind HTTPS and correct CORS/auth settings
- treat `10xgraph play` as an interactive test path, not the deployment architecture

## Secret redaction in logs

Debug logging may surface API keys, bearer tokens, signed URLs, or other credentials that appear in LLM request parameters and error messages. The `tenxgraph.utils` module provides helpers to redact common credential formats before they reach your log handlers.

### Patterns redacted

`mask_secrets` redacts the following credential formats:

- OpenAI keys (`sk-...`, `sk-proj-...`)
- Google API keys (`AIza...`)
- GitHub tokens (`ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`)
- Slack tokens (`xox...`)
- AWS access key IDs (`AKIA...`)
- `Bearer <token>` values in Authorization headers
- `key=value` pairs where the key is `api_key`, `access_token`, `secret`, or `password`
- Signed-URL credential query parameters (`?token=`, `&sig=`, `&X-Amz-Signature=`, etc.)

### Quick setup

```python
from tenxgraph.utils import install_secret_redaction

# Call once at application startup, after configuring your logging handlers.
install_secret_redaction()                   # covers the "tenxgraph" logger and its handlers
install_secret_redaction("root")             # or cover the root logger
```

### Attaching to a specific handler

For finer control, add `SecretRedactionFilter` directly to a handler. Handler-level filters apply to all loggers that propagate to that handler, including children:

```python
import logging
from tenxgraph.utils import SecretRedactionFilter

handler = logging.StreamHandler()
handler.addFilter(SecretRedactionFilter())
logging.getLogger("tenxgraph").addHandler(handler)
```

### Redacting arbitrary strings

```python
from tenxgraph.utils import mask_secrets

safe_text = mask_secrets(some_string_that_may_contain_keys)
```

This is a defence-in-depth measure. Prefer not logging secrets in the first place. `mask_secrets` is a heuristic and may miss novel credential formats.

---

## Quick production checklist

1. confirm exact runtime command
2. confirm active `10xgraph.json`
3. confirm environment variables in the live process
4. confirm auth and CORS behavior from a real client
5. confirm persistence with restart testing
6. confirm proxy and network path

## Related docs

- [Deployment](/docs/how-to/production/deployment)
- [Environment Variables](/docs/how-to/production/environment-variables)
- [API Server Troubleshooting](/docs/troubleshooting/api-server)

## What you learned

- How to troubleshoot production failures by separating runtime, config, network, auth, and persistence layers.
- Which failures are usually caused by development defaults leaking into production.