Resources

Performance & latency

Classifier latencies, gateway overhead, and tuning levers.

The classifiers are small, purpose-built models chosen so screening fits in the request path. This page explains how to measure latency and which controls affect it.

Classifier latency#

There is no maintained production benchmark artifact that supports a published “typical” latency range for each classifier. Use the server-reported latency_ms value and end-to-end client timing from representative traffic in the deployment you plan to use.

Warm and cold inference differ

Latency varies with input size, load, and deployment. A GPU cold start, meaning the first inference on a cold replica, is slower than warm inference. Separate cold-start samples from steady-state samples when you set an SLO.

Measuring it yourself#

Every classifier response includes latency_ms (server-side inference time), so you can record it directly from an SDK result:

  • SDK: read result.latency_ms on any check result.
  • Gateway: each trace records request and per-span latency.
  • Dashboard: the Overview reports p50 / p95 / p99 across recent traffic.

Gateway overhead#

Through the gateway, added latency depends on mode and configuration:

  • Observe adds the classifier calls that are enabled for the request. It never adds a retry or a second upstream call.
  • Enforce adds the same classification, plus any recovery steps you enable (informed retry, model escalation) when a guard trips, each bounded by global_max_attempts.
  • Buffered streaming guards the full response before replaying it, so it has full parity but no time-to-first-token benefit. In Observe, incremental streaming passes provider chunks through without hold-back. In Enforce, it releases guarded chunks from a hold-back buffer for lower perceived latency. See Streaming.

Tuning latency#

  • Only enable the classifiers you need per project. INT-CoT and INT-Output are off by default in gateway policy.
  • Call classifiers concurrently. The SDKs expose async methods over a pooled client so input, CoT, tooling, and output checks can overlap when your code has the data.
  • Reuse the SDK client within a long-lived process so TLS and connections are amortized. Keep the Python async client on one event loop.
  • For chat traffic, prefer incremental streaming when time-to-first-token matters more than full pre-send guarding; Observe passes through while Enforce retains hold-back checkpoints.

Availability#

Classifier error and timeout verdicts are governed by your project’s fail-open or fail-closed setting. Other gateway and upstream failures have separate behavior. For contractual availability terms, contact info@triage-sec.com and identify the deployment you plan to use.