Multimodal and Vision

Image, audio, and document input is a first-class capability of the run endpoints. You upload a file once, get a file_id, and reference that id from a content block on any message you send to POST /v1/graph/invoke, POST /v1/graph/stream, or WS /v1/graph/ws. The server resolves the reference before the graph runs, so nodes and model adapters receive resolved media without knowing anything about the upload API.

This does not apply to WS /v1/graph/live. That socket carries audio frames and JSON control frames only; see its scope section.

The flow

1. POST /v1/files/upload stores the binary and returns a file_id.
2. You send a message whose content includes an ImageBlock (or AudioBlock, or DocumentBlock) carrying that file_id.
3. The server rewrites the block before execution:
   • Images and audio become a URL reference, graph://media/{file_id}, which the media reference resolver expands at model-call time.
   • Documents are replaced with a plain text block containing the extracted text, when an extraction is cached for that file.
4. The graph runs against the rewritten messages.

Step 3 runs inside the same input-preparation path that invoke, stream, and the ws socket all share, so behaviour is identical across all three.

Ownership is enforced during the rewrite: referencing a file_id uploaded by another user fails the request. The rewrite is a no-op when no media service is configured.

Step 1: upload the file

curl -X POST http://127.0.0.1:8000/v1/files/upload \
  -H "Authorization: Bearer $TOKEN" \
  -F "[email protected]"

{
  "success": true,
  "data": {
    "file_id": "b3f1c9d2e4a5",
    "mime_type": "image/png",
    "size_bytes": 184320,
    "filename": "invoice.png",
    "extracted_text": null,
    "url": "/v1/files/b3f1c9d2e4a5",
    "direct_url": null,
    "direct_url_expires_at": null
  }
}

The uploader is recorded as the file’s owner in the file’s own metadata, so ownership survives as long as the file does. Every read path (GET /v1/files/{file_id}, /info, /url, and the message rewrite) checks it, and a file owned by someone else returns 404 rather than 403, so the API never confirms that a foreign file_id exists.

Two rejections to expect at upload:

Step 2: reference the file in a message

Send an image block whose media carries the file_id:

{
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What is the total on this invoice?"},
        {"type": "image", "media": {"kind": "file_id", "file_id": "b3f1c9d2e4a5"}}
      ]
    }
  ],
  "config": {"thread_id": "invoices-1"}
}

Post that to /v1/graph/invoke or /v1/graph/stream, or send it as the messages array of a fresh frame on WS /v1/graph/ws. In every case the block reaches the graph as graph://media/b3f1c9d2e4a5.

Blocks that already carry an graph://media/ URL are left alone, so re-sending a previously rewritten message is safe.

Documents

Documents follow a different path because most models take text, not a file.

When DOCUMENT_HANDLING=extract_text (the default) and the uploaded MIME type is extractable, text extraction runs at upload time and the result comes back in extracted_text. It is also cached, keyed by file_id.

Extractable types:

application/pdf, application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, text/html, text/xml, application/xml, text/markdown, text/csv, application/json, text/plain.

Later, when a message references that file_id in a document block, the block is replaced with the cached text. If no extraction is cached the block passes through untouched and the agent’s own converter handles it.

The cache has two tiers: an in-process dictionary, plus the configured checkpointer’s cache namespace (media:extraction, 24-hour TTL) when a checkpointer is available. Without a checkpointer the cache is per process, so a document uploaded through one worker may need re-extraction on another.

Extraction requires the media extra:

pip install "10xgraph-api[media]"

DOCUMENT_HANDLING accepts three values:

Reading the server’s media configuration

GET /v1/config/multimodal reports the settings the server is actually running with, so a client can size its uploads and pick file types without hard-coding assumptions. It requires the config:read permission.

{
  "success": true,
  "data": {
    "media_storage_type": "local",
    "media_max_size_mb": 25.0,
    "document_handling": "extract_text"
  }
}

The response deliberately does not include the content-type allowlist. Treat 415 as the signal that a type is refused.

Environment variables

All media settings are read from the environment at startup.

Core

MEDIA_ALLOWED_CONTENT_TYPES

The allowlist is empty by default, which accepts anything. That is a deliberate default for local development, and a deliberate decision you should revisit before exposing uploads to untrusted callers.

Entries may be exact (image/png) or a wildcard subtype (image/*). Matching is case-insensitive and ignores any ;charset= suffix on the request’s content type.

MEDIA_ALLOWED_CONTENT_TYPES=image/png,image/jpeg,image/webp,application/pdf

# Any image, plus PDFs
MEDIA_ALLOWED_CONTENT_TYPES=image/*,application/pdf

A rejected upload returns 415 Content type not allowed: <mime>.

Cloud storage

Used only when MEDIA_STORAGE_TYPE=cloud.

Signed URLs

GET /v1/files/{file_id}/url returns a signed direct URL when the storage backend can produce one, falling back to the API-relative /v1/files/{file_id}. A signed URL bypasses the API entirely, so ownership is checked before one is issued.

Production checklist

• Set MEDIA_ALLOWED_CONTENT_TYPES to the types your agent actually handles.
• Set MEDIA_MAX_SIZE_MB to the smallest value that works. It is independent of MAX_REQUEST_SIZE, which the middleware applies to non-chunked request bodies.
• Use cloud storage for multi-worker deployments. The local store assumes a shared filesystem and memory loses everything on restart.
• Configure a checkpointer so the document extraction cache is shared across workers instead of being re-extracted per process.
• Set MEDIA_REQUIRE_OWNER=true if you need a hard guarantee that no file without a recorded owner can be read. Files uploaded before ownership tracking existed carry no owner and are otherwise allowed through with a warning.

See also

• REST API: Files
• REST API: Graph
• Environment variables