Streaming

The gateway supports OpenAI-style streaming (stream: true) and returns server-sent events. Because bytes already sent to a client can’t be recalled, streaming trades off between time-to-first-token and how much can still be blocked. Two modes handle this differently; pick one in your runtime policy.

Buffered (default)#

Buffered mode runs the upstream call to completion, evaluates the full runtime policy exactly as for a non-streaming request, and then replays the result as SSE. Enforce applies configured remediation and blocks. Observe records the simulated decision and replays the provider result. The cost is no time-to-first-token benefit.

Incremental#

In Enforce, Incremental mode streams from the upstream with a hold-back buffer, running an output-guard checkpoint before the first release and at intervals thereafter. A violation found before the first release can fall back to the full pipeline. One found after release can only terminate the stream. Earlier tokens cannot be recalled.

In Observe, upstream SSE chunks and tool calls pass through from the same provider call. Enabled output checkpoints are recorded as telemetry, but they do not trigger a fallback, second provider call, or stream termination. When the output guard is disabled, checkpoints are skipped in both modes.

buffered           : [screen fully] -> replay response as SSE
incremental/enforce: [checkpoint] -> release -> [checkpoint] -> release
incremental/observe: [pass through] + record enabled checkpoints

Responses streaming#

/v1/responses uses the buffered path today. When a client requests stream: true, the gateway buffers the upstream result, applies runtime policy, then emits a compact synthesized Responses SSE replay (response.created, response.output_text.delta, response.output_text.done, response.completed). This preserves policy parity and gives INT-CoT access to reasoning summaries before anything is replayed to the client. It is not a reconstruction of the provider’s native event lifecycle: item-level added/done, function-call, reasoning, and other provider event classes are not replayed. Consumers that require those native streaming events should not route /v1/responses streaming through the gateway.

Buffered requests must start a response within 110 seconds. If provider calls, guard checks, and policy retries exhaust that shared deadline, the gateway cancels the remaining work and returns HTTP 504 with EDGE_RESPONSE_DEADLINE_EXCEEDED before opening SSE.

Disabling streaming#

Streaming can be turned off per project. When disabled, a streaming request returns an error with reason STREAMING_DISABLED_POLICY (HTTP 501) instead of an SSE response, so clients fail fast rather than hanging.