# Retries, Fallback Models and Circuit Breaker for a Python AI Agent in 10xGraph

> Make a 10xGraph agent survive provider outages with RetryConfig backoff, a circuit breaker, cross-company fallback models, and a recursion limit.

Source: https://10xgraph.com/docs/examples/resilience
Last updated: 2026-10-28

Model providers return 429s and 5xx errors, and agents sometimes loop without finishing. This example covers both. The first script answers on Anthropic `claude-sonnet-5-5`, retries transient failures, and fails over to OpenAI `gpt-5-mini` and then Gemini `gemini-2.5-flash`. The second needs no API key and shows `recursion_limit` stopping a runaway loop.

## What you will build

An agent that answers support questions on Anthropic `claude-sonnet-5-5`, retries transient failures with exponential backoff, and fails over to OpenAI `gpt-5-mini` and then Gemini `gemini-2.5-flash` if Anthropic stays down. A circuit breaker keeps the agent from hammering a provider that is already failing.

A second script, which needs no API key, shows the other half of resilience: capping a runaway loop with `recursion_limit` and handling the exceptions the library raises.

## Key concepts

- `RetryConfig`: `max_retries`, `initial_delay`, `backoff_factor`, `max_delay`, `retryable_status_codes`, and the circuit breaker fields `circuit_breaker_enabled`, `circuit_breaker_threshold`, `circuit_breaker_reset_timeout`.
- `retry_config=` on `ReactAgent` / `Agent`: `True` (default) uses `RetryConfig()`, `False` or `None` turns retries off.
- `fallback_models=`: ordered list of `"model"` strings (same provider as the primary) or `(model, provider)` tuples (any provider).
- `recursion_limit`: a run-config key that caps node steps.
- `tenxgraph.core.exceptions`: `GraphError` (base), `GraphRecursionError`, `NodeError`, `NodeTimeoutError`, `StorageError` and others. Each has `error_code`, `context` and `to_dict()`.

## Run it

```bash
git clone https://github.com/10xGraph/10xGraph.git
cd 10xGraph/agentflow
pip install "10xgraph[anthropic,openai,google-genai]" python-dotenv
export ANTHROPIC_API_KEY="..."
export OPENAI_API_KEY="..."
export GEMINI_API_KEY="..."

python examples/resilience/retry_and_fallback.py
python examples/resilience/recursion_limit.py
```

`recursion_limit.py` needs only `pip install 10xgraph`. You only need keys for fallback providers you expect to reach.

## How the code works

### `retry_and_fallback.py`

**Retry order.** For each LLM call the agent builds the attempt list: the primary model, then each fallback in order. For one model it makes up to `max_retries + 1` calls. After a retryable error it sleeps `initial_delay * backoff_factor ** n`, capped at `max_delay`.

```python
retry_config = RetryConfig(
    max_retries=2,  # up to 3 calls per model: 1 try + 2 retries
    initial_delay=1.0,  # wait 1s, then 2s (delay * backoff_factor ** n)
    backoff_factor=2.0,
    max_delay=20.0,  # cap on any single wait
    retryable_status_codes=frozenset({429, 500, 502, 503, 529}),  # the default set
```

With these settings the waits are 1 s, then 2 s. When a model runs out of retries, or raises an error that is not retryable (a 400 or 401, for example), the agent moves to the next model at once. If every model fails, the last exception is re-raised as the provider SDK raised it.

**Circuit breaker.**

```python
    circuit_breaker_enabled=True,
    circuit_breaker_threshold=3,
    circuit_breaker_reset_timeout=60.0,
)
```

Each `(provider, model)` pair gets a breaker, stored on the agent instance so its state lasts across calls. One failed attempt cycle counts as one failure. After `circuit_breaker_threshold` failures in a row the circuit opens and the agent skips that model without calling it. After `circuit_breaker_reset_timeout` seconds one trial call is allowed. Success closes the circuit, and another failure opens it again. The breaker is off by default.

**Fallback models across companies.**

```python
    agent = ReactAgent(
        model="claude-sonnet-5-5",
        system_prompt=[{"role": "system", "content": "You are a support assistant."}],
        tools=[get_ticket_status],
        retry_config=retry_config,
        # A bare string inherits the PRIMARY model's provider, so "gpt-5-mini" alone would be
        # sent to Anthropic. To cross companies, give a (model, provider) tuple.
        fallback_models=[
            ("gpt-5-mini", "openai"),
            ("gemini-2.5-flash", "google"),
        ],
    )
```

The provider for the primary model is detected from its name. A fallback given as a plain string inherits that provider, which is only right within one company. The tuple form is `(model, provider)` with provider `"openai"`, `"google"` or `"anthropic"`. The fallback client is created on first use, so the OpenAI and Google SDKs are only needed if a fallback is actually reached. Fallbacks also reuse the primary agent's `base_url`, so do not combine them with a custom endpoint unless it serves every model in the list.

**Error handling.**

```python
    except GraphError as exc:
        # Library errors carry a code and context you can log or return from an API.
        print("graph error:", exc.to_dict())
    except Exception as exc:
        print("all models failed:", type(exc).__name__, exc)
    finally:
        await graph.aclose()
```

Library errors derive from `GraphError`. Provider errors that survive every fallback come through unchanged. `aclose()` in `finally` releases the graph either way.

### `recursion_limit.py`

No LLM is called. The graph is two plain nodes that route back and forth, which is what a confused agent loop looks like.

```python
def route(state: LoopState) -> str:
    # A real agent would stop when the task is done. This one never is.
    return "THINK"
```

The conditional edge always returns `"THINK"`, so the run never reaches `END`.

```python
    config = {"thread_id": "loop-1", "recursion_limit": 6}
    try:
        await app.ainvoke(inputs, config)
    except GraphRecursionError as exc:
        print("stopped by recursion limit")
        print("  code:   ", exc.error_code)  # RECURSION_001
        print("  context:", exc.context)  # max_steps, current_step, current_node
        print("  dict:   ", exc.to_dict())  # structured form, safe to return from an API
```

`recursion_limit` counts node steps (default 25). Lower it for cheap, fast failure. `GraphRecursionError` carries `max_steps`, `current_step` and `current_node`. The library also logs these errors with a traceback at ERROR level, so expect log noise.

```python
    try:
        await app.ainvoke(inputs, {"thread_id": "loop-2", "recursion_limit": 4})
    except GraphError as exc:
        print("caught as GraphError:", type(exc).__name__, "-", exc.message)
```

One `except GraphError` catches any library error.

```python
    saved = await app._checkpointer.aget_state({"thread_id": "loop-1", "user_id": "anonymous"})
    print("attempts recorded before the stop:", saved.attempts if saved else None)
```

The checkpointer keeps the state saved before the stop. The script reads it back from the private `_checkpointer` attribute.

## Full code

**retry_and_fallback.py**

```python
"""Keep an agent answering when its LLM provider fails.

Primary model is Anthropic. RetryConfig retries transient errors with exponential
backoff and adds a circuit breaker. fallback_models switches to OpenAI, then Google.
Run: python examples/resilience/retry_and_fallback.py
"""

import asyncio

from dotenv import load_dotenv

from tenxgraph.core.exceptions import GraphError
from tenxgraph.core.graph import RetryConfig
from tenxgraph.core.state import Message
from tenxgraph.prebuilt.agent import ReactAgent
from tenxgraph.storage.checkpointer import InMemoryCheckpointer

load_dotenv()

def get_ticket_status(ticket_id: str) -> str:
    """Return the status of a support ticket."""
    return f"Ticket {ticket_id} is open and assigned to the billing team."

retry_config = RetryConfig(
    max_retries=2,  # up to 3 calls per model: 1 try + 2 retries
    initial_delay=1.0,  # wait 1s, then 2s (delay * backoff_factor ** n)
    backoff_factor=2.0,
    max_delay=20.0,  # cap on any single wait
    retryable_status_codes=frozenset({429, 500, 502, 503, 529}),  # the default set
    # Circuit breaker, tracked per (provider, model) on this agent. After 3 failed
    # attempt cycles in a row the model is skipped for 60s, so a dead provider is not
    # retried on every request. After 60s one trial call is allowed.
    circuit_breaker_enabled=True,
    circuit_breaker_threshold=3,
    circuit_breaker_reset_timeout=60.0,
)

async def main() -> None:
    agent = ReactAgent(
        model="claude-sonnet-5-5",
        system_prompt=[{"role": "system", "content": "You are a support assistant."}],
        tools=[get_ticket_status],
        retry_config=retry_config,
        # A bare string inherits the PRIMARY model's provider, so "gpt-5-mini" alone would be
        # sent to Anthropic. To cross companies, give a (model, provider) tuple.
        fallback_models=[
            ("gpt-5-mini", "openai"),
            ("gemini-2.5-flash", "google"),
        ],
    )

    graph = agent.compile(checkpointer=InMemoryCheckpointer())
    config = {"thread_id": "ticket-7", "user_id": "demo-user"}
    try:
        result = await graph.ainvoke(
            {"messages": [Message.text_message("What is the status of ticket T-884?")]},
            config,
        )
        print("assistant:", result["messages"][-1].text())
    except GraphError as exc:
        # Library errors carry a code and context you can log or return from an API.
        print("graph error:", exc.to_dict())
    except Exception as exc:
        # When the primary and every fallback fail, the LAST provider's exception is
        # re-raised unchanged (an SDK error such as openai.APIStatusError or a
        # google.genai error). Handle it here, or let the caller retry later.
        print("all models failed:", type(exc).__name__, exc)
    finally:
        await graph.aclose()

if __name__ == "__main__":
    asyncio.run(main())
```

**recursion_limit.py**

```python
"""Stop a runaway graph with recursion_limit and catch GraphRecursionError.

No LLM is called: the graph is two plain nodes that route back and forth forever,
which is what a confused agent loop looks like. Also shows the library exception
hierarchy. Run: python examples/resilience/recursion_limit.py
"""

import asyncio

from tenxgraph.core.exceptions import GraphError, GraphRecursionError
from tenxgraph.core.graph import StateGraph
from tenxgraph.core.state import AgentState, Message
from tenxgraph.storage.checkpointer import InMemoryCheckpointer
from tenxgraph.utils import END

class LoopState(AgentState):
    attempts: int = 0

def think(state: LoopState) -> LoopState:
    state.attempts += 1
    return state

def act(state: LoopState) -> Message:
    return Message.text_message(f"attempt {state.attempts}: not done yet", role="assistant")

def route(state: LoopState) -> str:
    # A real agent would stop when the task is done. This one never is.
    return "THINK"

def build_graph():
    graph = StateGraph(LoopState())
    graph.add_node("THINK", think)
    graph.add_node("ACT", act)
    graph.add_edge("THINK", "ACT")
    graph.add_conditional_edges("ACT", route, {"THINK": "THINK", END: END})
    graph.set_entry_point("THINK")
    return graph.compile(checkpointer=InMemoryCheckpointer())

async def main() -> None:
    app = build_graph()
    inputs = {"messages": [Message.text_message("solve it")]}

    # recursion_limit counts node steps (default 25). Lower it for cheap, fast failure.
    config = {"thread_id": "loop-1", "recursion_limit": 6}
    try:
        await app.ainvoke(inputs, config)
    except GraphRecursionError as exc:
        print("stopped by recursion limit")
        print("  code:   ", exc.error_code)  # RECURSION_001
        print("  context:", exc.context)  # max_steps, current_step, current_node
        print("  dict:   ", exc.to_dict())  # structured form, safe to return from an API

    # Every library error derives from GraphError, so one handler can catch them all.
    try:
        await app.ainvoke(inputs, {"thread_id": "loop-2", "recursion_limit": 4})
    except GraphError as exc:
        print("caught as GraphError:", type(exc).__name__, "-", exc.message)

    # The checkpointer still holds the work done before the stop.
    saved = await app._checkpointer.aget_state({"thread_id": "loop-1", "user_id": "anonymous"})
    print("attempts recorded before the stop:", saved.attempts if saved else None)

    await app.aclose()

if __name__ == "__main__":
    asyncio.run(main())
```

Source on GitHub: [examples/resilience](https://github.com/10xGraph/10xGraph/tree/main/agentflow/examples/resilience)

## Expected output

Illustrative. `recursion_limit.py` printed this when run locally (log lines omitted):

```text
stopped by recursion limit
  code:    RECURSION_001
  context: {'max_steps': 6, 'current_step': 6, 'current_node': 'ACT'}
  dict:    {'error_type': 'GraphRecursionError', 'error_code': 'RECURSION_001', ...}
caught as GraphError: GraphRecursionError - Graph execution exceeded recursion limit: 4
attempts recorded before the stop: 3
```

`retry_and_fallback.py` prints one line when it works:

```text
assistant: Ticket T-884 is open and assigned to the billing team.
```

## Using a different model

Change the `model` string and the first tuple entries. For example `model="gpt-5-mini"` with `pip install "10xgraph[openai]"`, and fallbacks `[("claude-sonnet-5-5", "anthropic"), ("gemini-2.5-flash", "google")]`.

## Next steps

- [Graceful shutdown](/docs/examples/graceful-shutdown): finish and flush work when the process is told to stop.
- [Human in the loop](/docs/examples/human-in-the-loop): pause risky steps for approval.
- [Context management](/docs/examples/context-management): keep long conversations under the token limit.

## Frequently asked questions

### How do I fall back to a model from a different provider?

Pass a (model, provider) tuple in fallback_models, for example ("gpt-5-mini", "openai"). A plain string reuses the primary agent's provider, so a bare "gpt-5-mini" on a Claude agent would be sent to Anthropic.

### Which errors are retried?

HTTP status codes in RetryConfig.retryable_status_codes (429, 500, 502, 503, 529 by default) and connection or timeout errors. Any other error skips the remaining retries for that model and moves straight to the next fallback.

### What does the circuit breaker do?

When circuit_breaker_enabled is true, consecutive failed attempt cycles for a (provider, model) pair open its circuit. While it is open the agent skips that model and goes to the next fallback, until circuit_breaker_reset_timeout passes and one trial call is allowed.

### How do I stop an agent that loops forever?

Set recursion_limit in the run config (default 25). When the step count is reached, the graph raises GraphRecursionError, which you can catch.