Logs that are safe to ship

Logs leave your process. They go to a log aggregator, a trace backend, an error tracker, sometimes to a vendor in another country. Whatever the server writes ends up in every one of those places and stays there for the retention period. So the useful question is what the server refuses to write, not how much it can.

10xGraph is open-source, self-hosted infrastructure for AI applications. You write the agents, tools and tasks; 10xGraph runs them behind an API server with authentication, role-based access, tenant isolation, durable state in Postgres and Redis, background jobs, metrics and tracing, and generates Docker and Kubernetes deploy files. MIT licensed, by 10xScale. Formerly Agentflow.

This post lists what 10xGraph and 10xgraph-api keep out of logs, traces and error reports by default, and the one setting you choose yourself. The setup steps are on the observability page.

User ids in traces

A trace span carries the id of the user who made the run. That is useful when you debug and a liability when the id is an email address.

observability.user_id_mode in 10xgraph.json decides how the id appears on spans:

Value	What the span carries
raw (default)	The id as is
hash	16 hex characters of SHA-256, or HMAC-SHA256 when TENXGRAPH_TRACE_USER_ID_KEY is set
mask	The ends of the id with the middle hidden, for example al***42

Prefer hash with the key set. A plain SHA-256 of a short id can be guessed by hashing candidates, and the keyed form cannot. The setting applies to the observability block and to OTEL_ENABLED, and it is in the 10xgraph config editor. A core older than the option keeps the raw id and logs a warning, so check the version.

This changes the user id only. Message and tool content stay in traces at the full observability level, so choose the level deliberately.

Secrets in connection strings

RedisPublisher used to put the URL it failed to connect to in its error, password included. The error now shows the URL through redact_url, which removes the userinfo, query and fragment, so redis://:s3cret@host:6379/0 reads as redis://host:6379/0. The same helper keeps IPv6 hosts in brackets and returns <invalid url> for a string with no scheme or host. Media download warnings use it as well, so a signed URL does not leak its query string.

Tool failures: a reference, not the exception

An unexpected error in a local or MCP tool used to put its message into the tool result and into events: database URLs, file paths, SQL errors. The model saw it, and so did anything that stored the conversation.

Now the model gets Tool '<name>' failed with an internal error (reference <ref>). The full error, with the same reference, is logged on the server. When a user reports a failure, the reference takes you from their screen to the stack trace. Argument errors, “Tool not found” and timeouts still reach the model as before, because the model can act on those.

Client values cannot forge log lines

Thread ids, file names, request paths and similar values come from clients. A value with a newline in it can end a log entry and start a fake one; one with an escape character can act on the terminal of whoever tails the log.

The API writes client-influenced values through log_safe. It escapes CR and LF, writes the other control characters (ESC and the rest of C0, and DEL) as \xNN, and keeps tab. The upload route logs the client file name this way, and request logs escape thread ids. In production, the per-run observability, stop, fix and WebSocket run lines no longer log the thread id at all. Warnings that must stay in production escape it.

Errors in production

With MODE=production, the error text a client sees is generic. A node that raises answers An error occurred in a graph node. with its code, such as NODE_001. The same swap applies to invoke, stream, WebSocket and AG-UI error chunks, and to GET /v1/threads/{id}/state, which used to return the raw exception text in execution_meta. The tool failure reference above is kept, since it contains no detail. Development shows the full text.

Without this, a connection string inside an exception message travels to the browser.

Sentry

If you enable Sentry, the server starts it with request bodies off (max_request_body_size="never"), local variables off and personal data off.

Log levels follow the status code the handler returns. A 4xx (401, 403 for a missing permission, 404, 413, 422, 429 and so on) is one WARNING line with the method, path, status and error code, with no traceback and no body. A 5xx stays an ERROR with the traceback. Only ERROR logs become Sentry events, and WARNING and INFO stay breadcrumbs. A bad token or a probing scanner no longer pages you.

One thing to change when you upgrade: an alert that matched ERROR for client errors must match WARNING.

Structured logs and the audit log

Set LOG_FORMAT=json (the default is text; any value other than text or json is rejected) and every application log line becomes one JSON object, which a log pipeline can parse without regular expressions. The secret sanitizer still applies.

A resource that is outside the caller’s reach answers 404, the same as one that does not exist, so a probing client learns nothing from the status code. The server can also record those decisions in a metadata-only audit log. With audit enabled and the log sink, each event is a JSON line on stdout from the logger tenxgraph_api.audit. It holds ids and outcomes, never prompts, answers or file contents. 10xGraph stores no audit data: your log pipeline keeps it, with the retention you choose. See Audit log.

What you still own

• Anything your own code logs. Escape client values and strip secrets from URLs before you log them. redact_url is in tenxgraph.storage.media.url_policy.
• Prompts, tool arguments and results in traces at the full level. Pick a lower level if conversations are sensitive.
• Retention and access on the log backend itself.