Send traces to an OTLP endpoint

If your tracing backend speaks OTLP over HTTP, you do not need a vendor-specific destination. 10xGraph sends its spans (graph execution, node transitions, LLM calls, token usage, tool execution) to any OTLP HTTP trace receiver: an OpenTelemetry Collector, Grafana Tempo, Jaeger, Grafana Cloud and others.

The receiver must accept traces. Prometheus does not: it takes metrics only. For metrics, see Metrics with Prometheus and Grafana.

The observability levels and user_id_mode options in Send traces to Logfire and LangSmith apply here too.

An OTLP destination is an observability publisher. It receives graph, node, LLM and tool spans, AI errors included. It does not receive task lifecycle events or API and worker errors: those are in metrics and logs. See Publishers, AI tracing and metrics.

Install

pip install '10xgraph[otlp]'

The otlp extra installs the OpenTelemetry API, the SDK and the OTLP HTTP exporter. 10xgraph[observability] includes it.

Use the helper

Call setup_otlp() before graph.compile():

from tenxgraph.core.graph import StateGraph, Agent
from tenxgraph.runtime.publisher import setup_otlp, ObservabilityLevel
from tenxgraph.utils import END

graph = StateGraph()
graph.add_node("MAIN", Agent(model="gpt-4o"))
graph.set_entry_point("MAIN")
graph.add_edge("MAIN", END)

setup_otlp(graph, endpoint="http://collector:4318", level=ObservabilityLevel.STANDARD)

app = graph.compile()

FULL (the default) records prompts, completions, tool input and output and the graph state, so message content reaches the collector and whatever stores traces behind it. STANDARD drops prompts, completions and tool I/O but still records the graph state on node spans, which can contain the conversation. SPANS records timing and structure only. The example above passes STANDARD explicitly; omit level to get FULL.

setup_otlp() raises ValueError when it has no endpoint and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is unset. OTEL_EXPORTER_OTLP_ENDPOINT is not used as a fallback, because it often points at a gRPC port (4317), which this HTTP exporter cannot use.

Authentication headers

Headers are not a function argument and not a config key. Set the standard OpenTelemetry variable:

export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer your-token"

OTEL_EXPORTER_OTLP_TRACES_HEADERS also works and applies to traces only.

Configure the API server

In 10xgraph.json:

{
  "observability": {
    "level": "standard",
    "otlp": {
      "enabled": true,
      "endpoint": "http://collector:4318"
    }
  }
}

level defaults to "full" when omitted. If the graph already has an observability publisher, the server reuses it, logs a warning and does not apply this block, including its level.

If the exporter package is missing or no endpoint is set, the server logs a warning (“Observability setup skipped”) and starts without AI tracing. Put tokens in .env, never in the config file. See configuration for all keys.

Share the endpoint with metrics

When one endpoint takes both signals, such as an OpenTelemetry Collector, Grafana Alloy or Grafana Cloud, set use_metrics_endpoint to true instead of repeating the URL:

{
  "observability": { "otlp": { "enabled": true, "use_metrics_endpoint": true } },
  "metrics": { "enabled": true, "export": "otlp", "otlp": { "endpoint": "http://collector:4318" } }
}

This needs metrics.enabled with export set to otlp and metrics.otlp.endpoint set. Without them the server logs a warning and turns the OTLP trace destination off. If the shared URL contains /api/v1/otlp (a Prometheus OTLP receiver), the server logs a warning that trace exports will fail, because Prometheus accepts metrics only.

HTTP requests are not AI traces

AI tracing carries graph, node, LLM and tool spans only. Spans for HTTP requests never go to the OTLP destination or to any other AI tracing destination: the server turns FastAPI’s own telemetry off. With the API and the worker both pointed at one collector, only graph and node spans arrive from either. HTTP timing is available as metrics instead. See Metrics with Prometheus and Grafana.

Combine with other destinations

logfire, langsmith, phoenix and otlp in the observability block share one TracerProvider. In code, use setup_observability(graph, {...}) with an "otlp" entry.

Troubleshooting

Nothing arrives. Check the endpoint is reachable and accepts traces on /v1/traces. Spans are exported in batches, so allow a few seconds and let the process exit cleanly.

401 or 403. Set OTEL_EXPORTER_OTLP_HEADERS with the credentials your backend expects.

ValueError: OTLP tracing needs an endpoint. Pass endpoint= or set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT.

ImportError for opentelemetry-exporter-otlp-proto-http. Run pip install '10xgraph[otlp]'.