Streaming

Streaming keeps the provider protocol. Enforce waits for required review before release; Observe relays admitted provider traffic without waiting for the review. Treat stream completion and errors as part of the operation result.

What the application receives#

In Enforce mode, the gateway buffers content and executable arguments until required checks complete. It may send protocol-compatible heartbeats while work continues. In Observe mode, provider bytes reach the client as they arrive and the review follows the completed relay; a later verdict does not retract delivered content. Neither a heartbeat nor a completed Observe stream proves approval or authorizes an external tool effect.

Include usage and supported history#

Streaming Chat Completions must set stream_options: { include_usage: true }. Every protocol needs an explicit positive output-token limit and supported explicit history. Missing usage or an unsupported request can hold the operation.

{"stream":true,"stream_options":{"include_usage":true},"max_completion_tokens":1024}

Handle errors after HTTP 200#

HTTP 200 can still end with an error event. When the deployment selects error delivery, pass the typed Integrity error body to raise_for_integrity_hold or raiseForIntegrityHold. The default message delivery instead completes the provider-shaped stream with a hold explanation and an integrity marker containing its disposition, code, and operation ID. Chat Completions uses content_filter, Anthropic uses refusal, and Responses can end with response.completed while still carrying that hold marker. These are gateway explanations, not released customer-model answers. The SDK error-body helper does not parse those completed messages. Inspect the terminal marker or recover the durable operation if your provider client hides it; never infer approval from HTTP 200 or protocol completion. Stop execution on a hold and preserve a truncated stream as uncertain.

After a disconnect, use the original session and operation ID to recover. Do not automatically replay a stream or execute partial tool arguments.

Runs has its own activity stream#

Runs displays normalized session activity independently of the provider response stream. Its reconnect behavior does not repeat model inference. A UI update is evidence about the run, not an execution instruction to your harness.