Thresholds and calibration

Integrity uses static classifier artifacts and thresholds from your project policy. It does not currently fit project-specific thresholds or update classifier weights from runtime traffic. Tenant adaptation documents the rollout boundary for two experimental opt-ins with milestone-gated delivery.

Two separate mechanisms#

The product uses calibration and threshold tuning for different operations:

Project threshold tuning is manual. INT-CoT model calibration is selected at inference time from the source_model string. Neither mechanism learns from live project traffic.

New project defaults#

A new gateway project starts in Observe with fail_open enabled. Enabled classifiers run on the first eligible request. There is no project calibration wait.

For INT-CoT, a material score that is still rising can escalate an independently flagged tool or output result. The CoT verdict itself remains advisory and never becomes unsafe on its own.

Tune project thresholds#

Tune thresholds before you enable blocking. Use traffic that represents normal requests, high-risk actions, and attacks you expect the application to face.

1. Keep the project in Observe with fail_open enabled.
2. Run representative traffic. The policy screen plots recent score distributions and shows the approximate share at or above each threshold.
3. Inspect traces and correlate them with your own application logs. Identify safe requests that were flagged and attacks that scored too low.
4. Move the flagged and unsafe thresholds. Lower values surface or block more traffic. Higher values reduce alerts and allow more traffic.
5. Save the policy, observe subsequent requests, and switch to Enforce only after the observed decision mix matches your risk policy.

Policy changes normally reach the gateway within about 15 seconds. The policy store keeps the current configuration and timestamp, not a revision history. Record the previous values before a change if you need a rollback point.

Data used for tuning#

Threshold tuning uses data you review. Persisted telemetry includes scores, verdicts, outcomes, and timings for the project, and the policy screen displays recent score distributions. No background job reads those records and changes policy or model weights.

• Gateway traces. The default metadata_only mode retains decisions and scores without raw request or response bodies.
• Content review. Under metadata_only, use your application logs to classify a specific result as expected or unexpected. Use redacted or full only when your data policy permits retained content.
• Project memory. Memory stores human-confirmed context for repo-backed projects. It does not tune classifier thresholds. See Settings, roles, and memory.

Changing the observability content mode changes what content a record may retain. It does not change thresholds or enable classifier training. The production telemetry retention window defaults to 90 days. See Security and data handling.

INT-CoT source models#

INT-CoT ships with a global threshold, exact thresholds measured on labeled traces from evaluated source models, and explicit family proxies for selected current models. Selection is case-sensitive and uses this precedence: exact per-model calibration, then a declared family_proxy, then the global fallback. The provider is not part of the lookup.

Pass the exact model identifier that generated the reasoning trace. Retest INT-CoT score distributions after you change that identifier or move to a new model version.

The current classifier artifact loads finite thresholds for these exact keys:

• claude-fable-5
• deepseek-ai__DeepSeek-V4-Flash
• deepseek-v4
• deepseek-v4-fable
• glm-5.2
• gpt-5.5
• gpt-5.5__synthetic_divergence
• kimi-k2.6
• qwen3-14b

The following current models inherit an exactly calibrated relative’s threshold:

• gpt-5.6-sol → gpt-5.5
• claude-opus-4-8 → claude-fable-5

gpt-5.6 is not the exact gpt-5.6-sol key and therefore uses the global fallback. Provider- or dataset-style keys containing __ are exact artifact identifiers and may not correspond to a production API model name; pass one only when it exactly describes the trace source.

Current limits#

• No automatic project threshold fitting or proposed operating points.
• No trace labeling workflow that feeds threshold proposals or tenant adaptation.
• No calibration state, policy revision history, or one-click rollback.
• No production drift detector or automatic recalibration state.
• No online adapter or classifier-weight training from runtime traffic.

Retune manually after material changes to models, tools, prompts, retrieval sources, or user traffic. Classifiers remain probabilistic, so every threshold trades detection coverage against false positives. See Tenant adaptation for the rollout boundary and Limitations and FAQ for current runtime limits.