Limitations of Heuristics in the Convergence on Cognition

Abstract

Authorized agents complicate detection because they already possess legitimate credentials and tools. The decisive question is whether an action remains congruent with the user's request and the organization's sanctioned-use policy. Enumeration-based detection is structurally mismatched to this task: fixed feature spaces, local decisions, and human-paced updates cannot fully characterize behavior generated by learned, context-dependent processes at inference time.

We survey seven families of deployed detection technology, compare heuristic and transformer-based mechanisms across eight axes, and characterize four observation surfaces: input, tool call, reasoning, and output. Recent exploitation, autonomous attack-defense, chain-of-thought, and interpretability studies ground the analysis.

The resulting design space is constrained by expressivity, latency, and adaptivity; by escalation-exhaustion and baseline-displacement attacks; and by the divide between white-box and black-box access. This paper maps those constraints rather than selecting an architecture, then examines limits imposed by unfaithful or encoded reasoning, opaque training effects, and policy specification. Enumerated controls remain useful components, but cannot supply contextual judgment on their own.

1. Introduction

Cybersecurity detection has historically been built on enumeration. A signature is an enumeration of known-bad byte sequences. A web application firewall rule is an enumeration of forbidden input shapes. The Common Weakness Enumeration is, in name, an enumeration of weakness classes. A behavioral allowlist is an enumeration of permitted process trees. The discipline of building these enumerations (collecting indicators, distilling patterns, and encoding rules) has been the operational practice of cybersecurity engineering for three decades. It worked because the adversary, too, was working within enumerable space. Malware families were finite. Exploit techniques were taught in finite training programs. Attack tooling was distributed through finite supply chains. When a new technique appeared, the enumeration grew to absorb it, and equilibrium was restored on a roughly weekly cycle.

Authorized agents break that equilibrium by adding a second security problem alongside external intrusion. Modern security programs address external adversaries with substantial machinery (IAM, PAM, EDR/XDR, zero-trust controls, cloud posture management, attack-path management) that goes well beyond the detection traditions we focus on in this paper. The newer problem is ensuring that authorized actors, increasingly autonomous and powerful, continue to act in alignment with the organizations that deployed them. When a coding agent commits code to a production repository, when a finance agent moves funds between accounts, when a customer-support agent reads from and writes to a customer database, the question of whether the action was performed by a malicious external party remains relevant. A second question is whether the action falls within the agent's sanctioned use: user intent for the session; the agent's role scope; its delegated authority; permitted data boundaries; the operation's approval state; and required business-process invariants.

Heuristic detection asks whether a pattern belongs to a forbidden set. Monitoring an authorized agent asks whether an action, in its context and history, satisfies the sanctioned-use criteria. The former is pattern matching over a fixed feature space; the latter is behavioral attestation against a context-dependent policy the monitor must model. OWASP's LLM06: Excessive Agency ([7]) catalogues the broader risk landscape. Our focus is how to monitor whether granted agency remains aligned with the granting organization's intent.

2. A taxonomy of deployed heuristic detection

A heuristic, in the operational sense relevant here, is a function $f: X \to \{0, 1\}$ that maps an observation $x$ to a binary decision (allow, block) or a discrete enumeration of labels. The function is constructed by humans from prior incidents, and its correctness is local to the set of inputs that resemble those incidents. A signature for the EternalBlue exploit is correct on the EternalBlue payload, and on payloads similar enough to share the discriminating bytes, and uncorrelated with adversary behavior outside that neighborhood.

The deployed techniques vary, but all operate over a feature space fixed at deployment time. The features may be bytes, n-grams of bytes, syscall sequences, graph adjacencies, statistical moments of network flows, or hand-engineered domain features. The function over those features may be a regex, a linear classifier, a random forest, a hidden Markov model, an isolation forest, or a graph algorithm. The space is human-designed; the function operates within the space; the space does not adapt to the input.

We survey seven families below, then return to the unifying property in Section 2.8.

2.1 Deterministic pattern matching

The oldest and most widely deployed family. Includes Aho-Corasick automata for multi-string matching, Boyer-Moore for single-pattern search, hash-based lookup against known-bad indicators (file hashes, IP addresses, domain names), Bloom filters as compact set-membership filters, and the YARA rule compiler that generalizes pattern matching to conjunctive byte-level predicates over file content. ClamAV is the canonical open-source instantiation; commercial antivirus products are largely the same engine class with proprietary signature databases.

The defining property is exact equality as the discrimination criterion. The input matches if and only if it contains a known string or matches a known hash. Latency is sub-microsecond per query against a database of millions of indicators, which is the most attractive property of the family. For exact cryptographic-hash matching, collision-driven false positives are negligible; operational false positives still occur when the intelligence label is wrong, stale, or applied outside its intended context, and antivirus vendors track this as a meaningful engineering metric. Generalization to unseen inputs is structurally zero.

The failure mode is the polymorphic adversary, which has been understood in the malware literature since at least the Mistfall engine in the early 2000s. Any transformation of the malicious payload that preserves semantics but changes bytes evades the signature. The defensive response has historically been to enumerate more signatures, faster, with broader contributor networks. The frontier response is that semantic-preserving transformations are now cheap to generate at scale. A signature database grows linearly with analyst effort; the attack space grows combinatorially with model capability.

2.2 Rule-based input filtering

Web Application Firewalls such as ModSecurity (with the OWASP Core Rule Set), AWS WAF, Cloudflare's rule engine, and F5 ASM. Pattern matching on HTTP requests using PCRE, with logic for compound conditions: a rule fires if pattern A appears in the URL and pattern B does not appear in the headers and the source IP is not in a reputation allowlist. Snort and Suricata extend the same paradigm to network packets with stateful flow tracking. SIEM correlation rules in Splunk SPL or Elastic EQL add temporal compounding: these three events from the same host within five minutes.

The expressivity gain over raw pattern matching is conditional logic and temporal composition. The shared limitation is that the rules operate on the syntactic surface of the input. Semantically equivalent attacks evade the rules by syntactic variation. The deployed response has been the OWASP Core Rule Set and similar curated rule packs that try to enumerate common attack patterns at a higher level of abstraction (SQLi*, XSS*, LFI*), but the same enumeration constraint applies.

A second limitation, less commonly stated, is that compound rules with temporal predicates do not compose well. A SIEM rule that requires events A, B, and C within five minutes assumes the attacker is operating on a five-minute timescale. A frontier coding agent that completes a multi-stage exploit in seventy-one minutes, as observed in the V8 trajectory cited in Section 5, generates events that are correlated but not within the temporal window of any deployed correlation rule. The temporal locality is itself a heuristic.

2.3 Statistical anomaly detection

Anomaly detectors assume the existence of a feature space $\mathbb{R}^d$ and a notion of distance within it, then flag samples that lie far from the mean of training data. The basic methods include univariate z-score thresholds on individual features, Mahalanobis distance to account for feature covariance, density-based methods like Local Outlier Factor, and partition-based methods like Isolation Forest. One-class SVMs and Gaussian Mixture Models offer slightly more sophisticated boundaries but share the underlying setup.

These methods made meaningful progress over signature-based detection in the 2000s and 2010s because they could detect novel attacks, as long as the attacks deviated statistically from baseline in the engineered feature space. Network intrusion detection systems based on anomaly detection showed real gains in academic benchmarks. User and Entity Behavior Analytics (UEBA) products are commercial instantiations of the same architecture against authentication and access logs.

Two failure modes are well known. First, concept drift: the baseline distribution of normal traffic is non-stationary, so the model's notion of normal diverges from current reality and false-positive rates climb until the model is retrained. Second, evasion: an attacker who knows the feature space can construct attacks that lie close to the normal distribution while still achieving malicious effect, a phenomenon studied formally as adversarial machine learning since Biggio et al. (2013). Both failure modes are amplified at the frontier, and both have a darker form when the system being monitored is itself an autonomous agent whose behavior is generated by inference: the agent can probe the feature space implicitly by observing what gets flagged, and the baseline drifts continuously as the agent's task distribution evolves.

2.4 Tree ensembles

Random Forests, gradient-boosted trees (XGBoost, LightGBM, CatBoost), and related ensemble methods are widely used in fraud detection (Stripe Radar, Sift), malware classification, and many specific intrusion-detection settings. The EMBER benchmark's baseline, for example, uses LightGBM over engineered PE features; MalConv is better treated separately as an example of end-to-end raw-byte neural malware detection, and we return to it in Section 3 as a contrast to the ensemble case. Tree ensembles handle nonlinear interactions among features, are robust to feature scaling, and are fast to evaluate. They also admit interpretability through feature importance and SHAP values.

But trees are still operating on fixed features. They learn a nonlinear decision boundary in $\mathbb{R}^d$, but $\mathbb{R}^d$ is engineered. The features for a malware classifier might include entropy of PE sections, presence of specific Windows APIs, packer detection results, import counts, mean instruction length, and similar quantities. These features are themselves heuristics, constructed from prior knowledge of malware behavior. An attack that operates outside those features, even if functionally equivalent to attacks the classifier was trained on, lies outside the classifier's discriminative space.

Tree ensembles learn decision boundaries over supplied representations rather than learning representations themselves. This is the same property as a regex, applied to a richer feature space. Expressivity is bounded by feature engineering rather than the data and compute available to the model.

2.5 Sequence models

Forrest et al.'s 1996 work on syscall sequence anomaly detection used short n-grams of system calls to characterize “normal” application behavior, with deviations flagged as intrusions. The line continued with HMMs over syscall sequences, n-gram models for command-line abuse detection, and Markov chains for state-transition modeling in protocol analyzers. Recurrent neural networks (LSTMs) on log sequences extended the line in the 2010s and remain deployed in some EDR products.

These methods relax the i.i.d. assumption of statistical anomaly detection by encoding short-range dependencies. They are limited by the Markov assumption itself: any dependency longer than the n-gram window or the recurrent state is invisible to the model. Modern agentic trajectories, as visible in the ExploitGym traces cited in Section 5, span dozens of steps across heterogeneous abstractions and several different layers of the system. They are categorically beyond the reach of n-gram-style sequence detection, which sees only a sliding window.

LSTM-based variants partially relax the Markov bound but suffer from vanishing gradients on the long sequences that matter for monitoring. The transformer architecture, which we examine in Section 3, removes the fixed local Markov assumption within the context it is given; it does not remove all history limits, since context length, attention implementation, retrieval quality, and what is preserved across turns all continue to bound how far back the model can attend. The relaxation is partial but architecturally meaningful, and is the difference that separates the sequence-modeling era from the cognitive era.

2.6 Graph-based detection

Provenance graphs record the lineage of every object on a system: which process created which file, which user invoked which process, which network connection produced which downloaded artifact. The mature instantiations are systems like CamFlow, SPADE, and commercial EDR provenance backends. Detection algorithms over provenance include path-based queries (does any path from an untrusted source reach a sensitive sink?), centrality measures (which entities have anomalously high betweenness?), and community detection (does this subgraph match known malicious patterns?). The MITRE D3FEND and CALDERA frameworks formalize the entity-relationship vocabulary at the technique level.

Graph-based methods are the most expressive of the heuristic families because they capture relational structure that simpler models miss. They are also the most expensive: maintaining a provenance graph at scale requires either continuous streaming computation or periodic batch reconstruction, and the graphs grow without bound on long-running systems. Storage costs and query latencies make graph methods difficult to deploy at the sub-millisecond budget that pattern matching achieves.

The schema of the graph (what counts as a node, what counts as an edge, what entity types exist) is still hand-designed. An autonomous agent that operates within the graph schema in patterns that the detection algorithms have not been written to look for evades them. Graph-based methods are heuristic in the same way that the others are heuristic; they just operate at a higher level of abstraction.

2.7 Reputation and threat intelligence

Reputation systems (IP reputation, domain reputation, file hash reputation, certificate reputation) operate as distributed key-value stores of known-bad indicators contributed by sensors across an industry coalition or commercial network. An indicator that appears in the database triggers a block. Spamhaus DROP lists, Cisco Talos reputation, VirusTotal hash lookups, and STIX/TAXII threat intel feeds are representative instantiations.

The technique is operationally simple and effective against known threats. It generalizes to new threats only via the contributor network: a new indicator must be observed somewhere, attributed, and propagated before it becomes coverage. The latency of a reputation lookup at query time is microseconds. The latency of the contributor-network update from “indicator first observed” to “indicator distributed to defenders” is typically minutes to hours, sometimes days, sometimes never if the campaign uses one-shot infrastructure.

This is the most concrete instance of the broader pattern: monitoring updates at human-coordinated speed, the systems being monitored operate at machine speed.

2.8 The shared architectural commitments

Despite the diversity of techniques, every deployed detection family shares three properties that matter for the argument here.

Together, these properties assume that analysts can characterize the relevant signal in advance, discriminate it locally, and keep the resulting rules current at human update cadence. Continuously generated behavior violates all three assumptions.

3. A mechanistic comparison to model architecture

The previous section enumerated detection algorithms. This section compares them, mechanistically, to the architecture of the systems they will increasingly be deployed against and, increasingly, alongside. The comparison isolates the computation each family performs and what its mechanisms permit or foreclose.

3.1 Unit of input

A heuristic operates on tokens defined by humans. The tokens of a regex are bytes; the tokens of a WAF rule are URL parameters and header values; the tokens of an HMM are system call IDs; the tokens of a tree ensemble are scalar feature values. The tokenization is the boundary between the raw input and the model, and it is fixed at design time.

A transformer-based model operates on tokens defined by a learned tokenizer (BPE, SentencePiece, Tiktoken) over raw input bytes or characters. The tokenization is itself a function of the training distribution, and is not hand-designed. The downstream layers receive embeddings: dense vectors in $\mathbb{R}^d$ that are learned representations of the token. The embeddings carry semantic structure: similar tokens have nearby embeddings, and the similarity is induced by the training objective rather than by a hand-coded similarity metric.

A byte-level heuristic cannot use semantic structure induced by surrounding context. The strings s3lect, select, SELECT, and sel%65ct are four different tokens (or sequences of tokens, since the learned tokenizer may split them differently) to a regex and to the model's input layer. The difference is what happens downstream. A learned model can often map these variants to similar internal representations when the surrounding context supports the SQL interpretation; a byte-level rule must enumerate the surface forms or rely on normalization. A monitor operating only at the byte surface sees four different patterns and must enumerate each. A monitor operating on downstream representations can, in principle, see one pattern with four surface forms.

3.2 Feature space construction

Heuristic detection operates in a human-chosen feature space. Learned-model detection operates in a feature space shaped by gradient descent against an objective.

Concretely: a random forest for malware classification uses features like “section entropy,” “import count,” “presence of CreateRemoteThread,” “PE header anomalies.” These features were chosen by analysts who understood malware. A model trained end-to-end on raw bytes (MalConv is the canonical reference) learns features that the gradient finds useful. Some of those features may coincide with the analyst's features; others will be features the analyst did not think to engineer; still others will be features that no human would recognize but that nonetheless improve discrimination on the test distribution.

The trade-off is interpretability. Engineered features are interpretable by construction. Learned features require separate methods such as sparse autoencoders, natural language autoencoders, and attribution graphs to recover human-readable structure. The monitor's choice between engineered and learned features becomes a choice between known coverage and unknown coverage. Against a static threat model, engineered features may be sufficient. Against an adaptive system, unknown coverage matters because a learned feature space does not expose its structure explicitly to the monitored system. The feature space can still be probed or approximated through interaction (this is essentially the project of black-box adversarial machine learning), but the attacker is inferring a high-dimensional learned boundary rather than reading a hand-specified rule set, and the cost asymmetry favors the defender in a way that hand-engineered features do not.

3.3 Context aggregation

A signature, a regex, or an n-gram model is local: the decision at a given position depends on a fixed-size neighborhood. A WAF rule examines an HTTP request; it does not examine the history of requests from the same client. An HMM has a finite state and a finite history. A tree ensemble's decision at a sample is independent of other samples.

A transformer aggregates context via self-attention. Every token in the input attends to every other token, weighted by learned compatibility scores. The decision at any position can depend, in principle, on any other position in the context window. The mechanism is $O(n^2)$ in sequence length but parallelizable, and it captures dependencies that no Markov-bounded model can capture.

Agentic trajectories span dozens of steps across heterogeneous abstractions. A fixed local window cannot connect step 3 of a trajectory to step 30. Attention-style global aggregation can, in principle, learn that the earlier step set up the later one even when their surface forms are unrelated.

3.4 Decision boundary

A heuristic produces a decision via a function whose structure was specified in advance: a regex match, a linear threshold, a series of axis-aligned splits, a Mahalanobis distance threshold. The shape of the boundary is constrained by the function family. A linear classifier has hyperplane boundaries; a tree ensemble has axis-aligned step boundaries; an isolation forest has hierarchical splits; a one-class SVM has kernelized but still constrained boundaries.

A neural network produces a decision via a function whose structure is implicit in the architecture but whose parameters are learned. The decision boundary can have arbitrary shape in the input space, limited only by the network's representational capacity. For a frontier-scale model, the capacity is effectively unconstrained for any practical input distribution. Greater capacity also raises the risk of fitting training-set artifacts and makes the decision boundary harder to characterize. Architecturally, heuristics are bounded by the function family their designers chose; learned models are bounded by data, compute, and architecture, in that order.

3.5 Update cadence and feedback

A heuristic is updated by humans editing rules and releasing them. The cadence is the cadence of human analysis: days for commercial detection vendors with mature pipelines, weeks for in-house security teams, months for cross-industry threat-intelligence standards.

A learned model has two distinct update channels. The first is weight updates through gradient descent. This can in principle be applied continuously as new labeled data arrives (online learning, preference fine-tuning, low-rank adaptation, reinforcement learning from labeled outcomes), and is what most deployed monitoring systems will rely on for substantial policy revision. Most deployed frontier-model agents adapt in production through a second channel: inference-time adaptation, in which behavior changes between interactions as context, memory, retrieved knowledge, tool feedback, and observed system state change. This is the channel that produces the moment-to-moment behavioral variation that monitoring has to keep up with. Some specialized deployments do additionally apply periodic or online weight updates; most do not.

A monitor that ships rules at release cadence runs on a slower clock than either channel. Every new turn can alter the agent's behavior, so static rules lose calibration even when the model's weights remain fixed.

3.6 The latency-expressivity tradeoff

Each detection method occupies a point on a tradeoff between latency (how fast a decision is produced) and expressivity (how rich a decision the method can produce). The deployed ecosystem distributes across this surface.

3.7 Generalization

Heuristics generalize by interpolation within the feature space they have been given. A signature for one variant of a malware family covers nearby variants whose bytes differ only in regions the signature does not match against; a tree ensemble trained on engineered features generalizes to held-out samples that share those features. The reach of this generalization ends at the feature-space boundary. A sample that lies outside the feature space (because the relevant signal is encoded in features the analyst did not choose) is invisible to the heuristic regardless of how similar it is in the underlying input to known examples.

Learned models generalize through the latent geometry of their training distribution. Two inputs that occupy nearby positions in the learned embedding space, even when they differ substantially in surface form, are treated similarly by downstream layers. This is what allows models to handle paraphrases, encodings, and reformulations without explicit enumeration. The generalization is also bounded, but the boundary is set by data and model capacity rather than by analyst foresight. In practice, this means learned-model generalization covers categories of inputs that heuristic generalization cannot cover at all, while remaining vulnerable to distribution shifts that produce inputs outside the training manifold.

3.8 Adversarial probing

A different axis concerns how an adversary learns the discriminative structure of the monitor in order to evade it. Heuristics are typically probed by reading the rules: signature databases are public or partially public, rule sets ship in open packages, and even closed-source rule engines tend to leak their structure through error messages, behavioral observation, or reverse engineering of the deployed binary. The adversary's task is essentially lookup, and once the rules are known, designing an evasive variant is a syntactic transformation.

Learned models are probed differently. The adversary cannot read the model's parameters, but can in principle reconstruct the decision boundary through repeated query interaction (gradient-free black-box attacks, knowledge distillation against the model's outputs, transferability from surrogate models). The literature on adversarial machine learning has characterized this regime in detail. The relevant property for monitoring is that probing a learned boundary is far more expensive than probing a rule set: the attacker must spend queries proportional to the dimensionality and curvature of the boundary, and the boundary itself can shift between probing and exploitation if the model is updated in the interval. Model extraction and transfer attacks can narrow this asymmetry, but reconstructing a learned boundary remains substantially more expensive than reading a rule set.

3.9 Summary

The cumulative comparison across these axes is summarized in the table below.

Table 1. Heuristic detection and learned-model architecture compared across the axes that matter for monitoring inference-time generative systems. Heuristics dominate on latency, interpretability, auditability, and fail-closed enforceability. Learned models dominate on the axes that bear on contextual legitimacy: representation, context aggregation, decision-boundary shape, generalization, and adaptivity.

4. The properties of inference-time generative systems

Four operational properties of inference-time generative systems interact directly with monitoring and distinguish them from rule-driven behavior.

4.1 Distributional behavior

A rule-driven system, given a context, produces a deterministic action. An inference-time generative system, given a context, produces a distribution over actions. The action that gets emitted is a sample. This is true at every level of granularity. Token-level next-action sampling is the surface case. Plan-level sampling, where the system decides between strategies, is a higher-order case. Both happen, and both have variance.

This matters for monitoring because the unit of analysis shifts. A signature-era monitor reasons about a specific observed action: was this byte sequence malicious? A monitor of a generative system must reason about the distribution from which the action was sampled: was the policy that produced this action consistent with the policy expected of an agent in this role? Two actions can be byte-identical but come from policies whose other samples differ in ways that matter. Two actions can look entirely different but come from policies that agree everywhere it counts.

Monitoring must distinguish tendency from variance. A system's tendency is the mode of its policy under a given context. Its variance is the breadth of the policy around that mode. Heuristic monitoring calibrates against tendency because tendency is what gets logged; instruction hijacking and emergent misbehavior exploit variance because variance is what evades the calibration. An unsafe tail event can arise from the same policy as routine behavior, without introducing a separately identifiable malicious tool.

4.2 Runtime adaptation

A scripted process, once launched, runs to completion or fails. A generative agent revises its plan based on environmental feedback. This is qualitatively different from “trying again with different parameters,” which is a brute-force loop with no internal state. The agent updates its belief about the environment between actions, and its next action is conditioned on the updated belief.

In the recent ExploitGym evaluation of frontier coding agents on real-world exploitation tasks ([1]), this property is visible in the trajectory traces. An agent attempting to exploit a V8 type-confusion vulnerability first attempted a direct file read against /flag, observed that the file was mode 400 and owned by root, abandoned the direct-read strategy, and pivoted to a memory-corruption pipeline that culminated in a setcontext-based redirection through system("/challenge/catflag"). The pivot was not pre-planned. It was selected at runtime based on the failure of the prior strategy. The full chain ran through approximately twelve phases in seventy-one minutes.

Monitoring designed against scripted behavior treats the trajectory as a sequence of discrete events to be matched against indicators. Monitoring against an adapting agent must treat the trajectory as a state machine whose transitions are themselves driven by inference. This requires observing belief updates and intermediate reasoning alongside final actions. Section 8 addresses the limits of reasoning as evidence.

4.3 Compositional escalation

Most consequential agentic trajectories that succeed in realistic environments succeed by composing primitives, not by introducing new ones. The component techniques in ExploitGym (out-of-bounds reads, return-oriented programming gadgets, type confusion, heap grooming, partial-pointer overwrites) have been known for years. The novelty is in the trajectory-level composition. The V8 trajectory cited above composes recognizable techniques into a single working exploit. Many of the component techniques are individually familiar to existing tools and analysts. The monitoring difficulty is that the risk lies in their global composition, where each local action can appear explainable while the overall plan is malicious or unauthorized.

This is the failure mode that pattern-matching monitoring most underestimates. A monitor that detects each primitive with 99% reliability and 1% false positive rate, applied to a nine-stage composition, does not produce 99% detection of the composition. It produces noise on each stage, and the operational response (alert fatigue, threshold raising, primitive whitelisting in known-benign contexts) degrades coverage faster than the primitives accumulate.

ExploitGym makes this point quantitatively. With standard mitigations enabled (ASLR, V8 heap sandbox, KASLR, user namespace restrictions), the strongest evaluated agents still produced working exploits on 37 userspace tasks, 20 V8 tasks, and 12 kernel tasks across the model fleet ([1]). The result illustrates the broader point: effective trajectories often arise from composing familiar primitives into a path that crosses abstraction layers, rather than from a single novel technique that an indicator can cleanly isolate.

4.4 Cross-layer reasoning

The fourth property is that generative agents reason across the abstraction layers of the system simultaneously. A signature engine operating on HTTP request bodies cannot see register state. A kernel exploit detector cannot see the prompt that caused the agent to attempt kernel exploitation. The historical division of monitoring along OSI layers, or along host/network/application boundaries, was sensible when the systems being monitored also worked within those layers.

A generative agent can plan at the natural language layer, translate the plan into a sequence of tool calls at the function-invocation layer, produce code at the program-text layer, and execute that code at the runtime layer. The same plan exists at all four layers simultaneously. Observing it requires correlating across layers, which means the monitor must also operate at all four layers, with shared state. This is the property that most strongly forces a re-architecting of where monitoring sits in the stack. A WAF cannot see the model's chain of thought. A code execution sandbox cannot see the prompt. A model-output classifier cannot see the kernel syscall trace. Monitoring at the frontier requires sitting on the abstraction layer where these signals are observable simultaneously, which is the inference loop of the model itself.

5. Empirical signals from the frontier

Two recent published evaluations give quantitative texture to the asymmetry.

ExploitGym

[1] evaluated frontier coding agents on 898 real-world exploitation instances drawn from OSS-Fuzz, Google's V8 engine, and the Linux kernel. Each instance ships with a proof-of-vulnerability input that triggers a bug; the agent's task is to extend that input into a working exploit that achieves unauthorized code execution. Under conditions where deployment-time safety filters were disabled through structured-access programs, Anthropic's Claude Mythos Preview produced working exploits on 157 instances, and OpenAI's GPT-5.5 on 120, within a two-hour time budget per task. With standard system-level mitigations enabled, success rates dropped substantially but did not approach zero: 25 userspace, 17 V8, and 3 kernel successes for Mythos Preview; 10 userspace, 3 V8, and 8 kernel successes for GPT-5.5. The kernel results are the most diagnostic. Kernel exploitation requires reasoning about shared heap layout under concurrent process noise, race-condition timing, and build-specific behavior, properties that previously required years of human specialization to develop fluency in.

The authors are explicit that this is a capability-elicitation study under controlled conditions, and the methodological caveats matter: deployment-time content filters were disabled for the measurement, prompts were tuned per agent, and some failures may reflect alignment-driven refusals rather than capability limits. The objective finding needed here is narrower: frontier agents produced working exploits on real-world memory-safety vulnerabilities, including under standard mitigations. The exact success rate does not affect the architectural argument.

CAI Attack/Defense

[2] ran the complementary experiment. Pairs of autonomous agents, one offensive and one defensive, competed concurrently on twenty-three Hack The Box battlegrounds under a fifteen-minute time budget. Both agents used the same underlying model. The headline result is counterintuitive: under unconstrained metrics, defense outperformed offense (54.3% patching success versus 28.3% initial access, p = 0.0193). Under operational constraints, where defense must patch and maintain service availability and prevent attacker access, defensive success rates collapsed to 23.9% and 15.2%, statistically indistinguishable from offensive success.

The defensive agents could identify and patch vulnerabilities, but not without breaking the systems they were defending. The cited trajectory shows blue-team agents modifying SSH configurations, changing service settings, and applying patches that disrupted availability. These are actions that, in isolation, look like defense but that an operator cannot accept. This is the same failure mode as monitoring that blocks too aggressively: the safeguard becomes the denial of service.

The experiment establishes the metric but not its cause. A plausible reading of the published trajectories is that the defensive agents optimized a stated objective (“patch the vulnerability”) while missing the operational objective (“preserve the service while removing the vulnerability”). They lacked a model of what the system was supposed to be doing against which a candidate intervention could be evaluated. This reading fits the observed availability failures; other mechanisms may also contribute.

Read together, the two results converge on a single picture. Offensive capability has reached a level where it composes recognized primitives into working exploits at a tempo (tens of minutes to a few hours) that exceeds any human-mediated monitoring cycle. Defensive capability is at parity for detection but loses parity the moment operational constraints are imposed, because the system lacks a model of legitimate use against which to evaluate candidate interventions.

6. Observation surfaces and their limitations

Four surfaces expose the signals available to a monitoring layer. Each carries different access and fidelity constraints.

6.1 Input

The input surface is the request, prompt, document, or other content that enters the inference loop. Heuristic monitoring at this surface (keyword filters, regex-based content rules, blocklists of known malicious patterns) covers a thin slice of the threat. The instruction-hijacking literature makes clear that the surface form of an adversarial input does not constrain its semantic effect. The same intent can be expressed in plaintext, in encoded form, in foreign language, in a system-prompt-shaped wrapper, in a markdown table, in an image's alt text, in a tool-result payload, in document metadata. Surface-form pattern matching at the input surface rapidly becomes incomplete once a technique is public, because semantically equivalent variants can be generated cheaply across encodings, languages, document formats, and tool-result channels.

The input surface can reveal semantic incongruity with sanctioned use, but only when the monitor conditions on the full interaction rather than the request alone. Section 6.2 illustrates the same distinction at the typed action boundary, where it is easier to inspect.

6.2 Tool call

The tool-call surface is the function-invocation channel through which the agent affects the world. This is the surface at which the agent's plan becomes an action, and it is the highest-leverage surface for monitoring because typed, parameterized actions admit structural analysis that prose does not.

A tool call becomes informative when evaluated jointly with the current context and prior calls against a model of what the agent's role authorizes. A transfer_funds call after a sequence of get_balance and lookup_account calls is consistent with a finance assistant's sanctioned operation. A transfer_funds call after a sequence of read_email and read_document calls is not, because the contextual setup did not establish the entity references that a legitimate transfer would require. The conditional history carries the incongruity.

This places a substantial burden on the monitor: it must model not only the agent's normal behavior but also the structure of legitimate use within the organization. A naive monitor that treats all observed tool-call sequences as in-distribution will be blind to legitimate-looking sequences that nonetheless represent incentive misalignment. We return to this in Section 7.

6.3 Reasoning: verbalized and latent

The third surface is the model's reasoning, where it is observable. We decompose this into two sub-surfaces because they have radically different properties.

Verbalized reasoning (text chain of thought). Intermediate text is observable when the host exposes raw chain of thought, rationale traces, scratchpads, or model-emitted plans. Production systems vary substantially: some expose raw reasoning, others provide summaries or post-hoc rationales, and others expose only final outputs and tool calls.

Latent state (residual-stream activations). With white-box access to the model, a monitor can read the activations $h_\ell \in \mathbb{R}^d$ at any layer $\ell$ and any token position. These activations are the model's internal state. They encode information that the model has not necessarily verbalized, including, as Fraser-Taliente et al. ([3]) demonstrate explicitly, beliefs that the model holds about its situation but does not articulate. This signal is available only to a monitor with access to the inference runtime.

With white-box access, the latent-state surface adds signal that verbalized reasoning may omit. Linear probes on a single residual-stream layer are cheap to evaluate, well within the latency budget of inference itself, and can detect features (such as evaluation awareness, deception, or specific concept activations) that the model's verbalized output does not surface. Sparse autoencoders and natural language autoencoders extend this to richer feature decomposition at higher cost. In black-box deployments, verbalized reasoning remains a low-cost observable that should be corroborated with behavior.

This has direct implications for the architecture of monitoring. A monitoring layer that operates only on the public API of the model is structurally limited to the verbalized surface. A monitoring layer integrated with the inference runtime, or deployed by the organization that hosts the model, can access the latent-state surface. These access models define different monitoring regimes and should be reported explicitly in capability assessments.

6.4 Output

The output surface is the final response the agent emits, before it leaves the inference boundary. Defense at this surface is the last line and the most expensive to operate, because the model has already committed compute to produce the output, and any block at this stage incurs the full inference cost.

The relevant signal at the output surface is incongruity between the output content and the sanctioned-use criteria for the agent's role. A customer-support agent that begins to enumerate other customers' account numbers in its response is producing content that is technically well-formed and would pass any syntactic check, but is semantically discontinuous with the prior turns and inconsistent with what the agent is authorized to discuss. This is the same conditional-distribution problem as the tool-call surface.

6.5 Why these surfaces

The decomposition corresponds to the points at which the agent's internal state becomes externally observable. Input is the boundary at which user intent enters the model. Tool call is the boundary at which model intent leaves the model as structured action. Reasoning is the substrate on which the computation happens, with two distinct sub-surfaces (text and latent) that differ in fidelity. Output is the boundary at which the model commits to a response. Anywhere else, the agent's behavior is observable only through the model's weights, which are typically not interrogable at runtime. A monitor that covers all of these surfaces sees the full external state of the inference loop. A monitor that covers fewer has blind spots that map directly onto attack categories. Coverage of input alone misses tool-call abuse. Coverage of output alone misses instruction hijacking at the input. Coverage of verbalized text alone misses encoded reasoning. The surfaces are jointly necessary; the question is whether the monitor can access them at sufficient fidelity.

7. The design space and its forcing functions

Sections 2–6 establish the limits of fixed-feature enumeration and the surfaces available to a monitor. This section characterizes the constraints and adversarial dynamics that bound candidate architectures rather than selecting a specific design.

7.1 Three forcing functions

Any monitoring system for inference-time generative agents is constrained simultaneously along three axes, and these axes do not commute: a choice on one constrains the available choices on the others.

These three constraints are in tension. Higher expressivity costs latency. Higher adaptivity costs robustness against the dynamics described in Sections 7.2 and 7.3. The design space lies at their intersection, where architectures make different trade-offs among latency, expressivity, adaptation, and residual risk.

7.2 The escalation exhaustion attack

Any monitoring architecture that resolves the latency-expressivity tension by varying its cost with input characteristics creates an adversarial dynamic that simpler architectures do not face. If the monitor's cost depends on what it observes (cheaper for the typical case, more expensive for ambiguous cases), then an adversary who understands the architecture can craft inputs that systematically push observation into the expensive regime, raising the monitor's mean cost to a level where the monitor itself becomes a denial-of-service liability.

This is structurally analogous to algorithmic complexity attacks against hash tables (Crosby and Wallach, 2003), regex engines (Cox, 2007), and other variable-cost data structures, but applied to the monitor's compute budget rather than to the application's. The arithmetic is straightforward: if cheap-path latency is $c$ and expensive-path latency is $C$, and the adversary can force a fraction $f$ of observations onto the expensive path, mean latency is approximately

which approaches $C$ as $f$ grows. For realistic ratios ($C/c \approx 10^2$ to $10^3$), the attack is asymmetric in the adversary's favor.

Defenses involve unresolved trade-offs among rate limiting (which can be bypassed by distributing the attack), caching (which can be defeated by input mutation), provisioning (which converts the attack to an additional-cost problem rather than eliminating it), and adversarial detection of the attack itself (which is a meta-monitoring problem with its own dynamics). Each of these has been studied in the broader resource-exhaustion literature but their composition in the context of agent monitoring remains uncharacterized. Variable-cost monitors should be evaluated under adversarial escalation load.

7.3 The baseline displacement attack

A different adversarial dynamic emerges from the third forcing function. Any monitor that adapts to feedback is, by construction, susceptible to having its adaptation shifted by the systems it monitors. The classical version of this concern is data poisoning, in which an adversary injects training samples designed to shift the learned function in a direction that benefits the adversary. The agent-monitoring version is structurally similar but more subtle, because the adversary need not inject synthetic samples: the agent's own behavior, observed during normal operation, contributes to the monitor's calibration.

Bootstrap is especially vulnerable: a monitor with no history is heavily influenced by its earliest observations, so adversarial influence during this window has outsized effect. Mature monitors face the long-tail regime: they must eventually incorporate genuinely novel behavior, while distinguishing novel legitimate actions from novel malicious ones. The broader literature on robust online learning, certified defenses against poisoning, and influence-function analysis provides useful machinery, but the operational constraints of agent monitoring (high throughput, continuous adaptation, adversaries with knowledge of the monitor's structure) place demands on this machinery that go beyond what has been demonstrated.

7.4 The white-box / black-box divide

A third constraint cuts orthogonally to the three forcing functions above. The latent-state surface can reveal signal absent from verbalized reasoning, but requires white-box model access. This divides the design space:

• –A monitoring layer that operates against a closed-source model accessed through an API is restricted to the verbalized and behavioral surfaces. The latent-state channel is inaccessible. The monitor inherits all the limitations of unfaithful chain of thought (Section 8.1) and steganographic chain of thought (Section 8.2). Its strongest available signal is conditional incongruity at the behavioral surface.
• –A monitoring layer that has white-box access (e.g., the model is hosted by the same organization that deploys the monitor, or the model provider exposes activation hooks) can access the latent-state channel. This dramatically changes the failure mode profile: deception detection becomes a linear-probe problem rather than a text-analysis problem; encoded reasoning is partly defended against by reading the substrate of the encoding directly; evaluation awareness becomes observable.

These architectural regimes have different threat models and research agendas. Most production deployments operate in the black-box regime because agents call third-party model providers. Research on white-box monitoring is mostly conducted by model providers themselves and by interpretability labs with research access.

Black-box deployments still expose substantial behavioral evidence and deserve a research agenda tailored to those signals, especially because they represent the overwhelming majority of enterprise deployments.

The signals that remain available at the API boundary are substantial even without activation access. The behavioral surface (tool calls, outputs, action sequences) is intact and admits the full battery of techniques discussed in Sections 6.2 and 6.4. The verbalized chain of thought, despite its faithfulness limitations, is still informative when treated as one signal among several rather than as authoritative. Beyond these surfaces, several research directions are emerging that are specifically suited to black-box deployments and that warrant more attention than they currently receive.

None of these techniques recovers the fidelity of direct activation probing. All of them are active research directions, and several have published evidence of meaningful efficacy on relevant threat models. Because white-box access is commercially unavailable for most frontier-model agents, operational research must establish how much latent-state-equivalent fidelity behavioral proxies can recover at the API boundary and what remains structurally unavailable.

7.5 Toward an evaluation framework

Candidate monitors need evaluation under the forcing functions and adversarial dynamics identified above. Benchmarks from adversarial machine learning, AI safety, and intrusion detection do not jointly test those constraints. The following metrics define the required evaluation surface.

Table 2. Metrics that an evaluation framework for agent monitoring would need to expose. Each maps to a constraint identified in Section 7. Operating thresholds depend on the organization's risk tolerance and deployment context.

Alignment capture rate is the hardest metric to measure cleanly because it requires a labeling protocol that captures the conditional structure of legitimate use, which is itself a research problem (Section 8.5). Static test sets also fail to represent systems and adversaries that adapt at runtime. A useful benchmark must include adversarial agents that themselves adapt to the monitor, in the same way that frontier agents adapt to environment feedback in evaluations like ExploitGym ([1]) and CAI A/D ([2]).

Bootstrap-robustness and escalation-distribution metrics require running the benchmark across a temporal window long enough for the relevant dynamics to manifest. A point-in-time measurement misses both of these failure modes by construction. Coverage must also include white-box and black-box scenarios so that architectures are compared within the signal regime they were designed to use.

A benchmark should expose these quantities so candidate architectures can be characterized along the design-space axes. To our knowledge, no existing taxonomy or benchmark jointly covers the three forcing functions, the two adversarial dynamics, conditional-legitimacy labeling, and both access regimes. Constructing one is a research contribution in its own right.

8. Limits of observability and specification

Section 7 addresses architectural constraints. This section addresses limits in the evidence and policy available to any architecture: models can obscure their reasoning, latent signals require interpretation, sanctioned-use policies are incomplete, and the monitor has its own attack surface.

8.1 Unfaithful chain of thought

The verbalized reasoning that a model produces during inference is not, in general, a faithful description of the computation that produced its output. Lanham et al. ([4]) established this in 2023: perturbations to the verbalized reasoning often do not change the final answer, suggesting that verbalized reasoning need not be causally necessary for the answer. Subsequent work on reasoning-specialized models ([5]) confirmed that the unfaithfulness persists in systems explicitly trained to reason step-by-step before answering. The relationship between verbalized chain of thought and underlying computation is at best correlational, often post-hoc, and sometimes outright misleading.

For monitoring, this has direct consequences. A monitor that relies on the verbalized chain of thought as evidence that the agent has reasoned about a decision in a particular way is relying on an unreliable signal. The agent may have reached the decision by a different path, and the verbalized reasoning may have been generated to be palatable rather than accurate. Text-based monitoring remains useful when corroborated by causally proximate or behaviorally grounded signals, usually the latent-state surface (Section 6.3) and the action surface (Section 6.2).

8.2 Encoded reasoning and steganographic chain of thought

A stronger version of the unfaithfulness problem is that models can, in principle, encode reasoning in the verbalized chain of thought in ways that humans cannot read but that the model itself can subsequently decode. This is steganographic chain of thought: the verbalized text serves as a cover channel for hidden computation. Early demonstrations of this phenomenon have appeared in the literature, and the architectural conditions that make it possible (the model produces tokens autoregressively, conditioned on its own previous tokens, with no constraint that those tokens be informative in a human sense) are general properties of the inference loop rather than artifacts of specific training procedures.

If steganographic chain of thought is present, a monitor reading the verbalized text sees fluent English that says one thing; the model, reading the same text, encodes a different computation. The monitor's understanding of the agent's reasoning is fundamentally disconnected from the agent's actual reasoning. This is not yet a demonstrated production-scale phenomenon, but the architectural conditions for it exist and the research literature treats it as an open concern. Latent-state monitoring is the most direct mitigation, because activations cannot be steganographically reformatted in the same way that text can: the encoding is the computation itself.

8.3 Subliminal learning and trait transmission

A separate failure mode arises from how models learn rather than how they reason. Cloud, Le, Chua, Betley, Sztyber-Betley, Hilton, Marks, and Evans ([6]) demonstrate that, under same- or closely related-base-model conditions, a student model trained on outputs from a teacher model can inherit traits of the teacher even when the training data is filtered to remove any explicit mention of the trait. In their key experiment, a teacher model with a particular preference (e.g., for owls) generates training data consisting of sequences of numbers; a student model trained on those number sequences inherits the teacher's preference for owls, despite the training data containing no semantic reference to owls or to any animal. They term this subliminal learning: the transmission of behavioral traits through training data that appears, by inspection, to be unrelated to the trait. The limitation is important: the authors report not observing the effect when teacher and student used different base models, so the result should not be read as a universal property of all distillation or fine-tuning pipelines.

Training-data inspection provides only a partial guarantee when models can acquire behavioral tendencies from data that appears unrelated to those tendencies. A model whose training data has been audited for absence of malicious content may still develop opaque behavioral tendencies through subliminal channels in the data, particularly when distillation or synthetic-data generation occurs between same-base or closely related models. The monitor at deployment time inherits the consequences of training-time decisions it cannot directly inspect.

Evaluating an agent against the organization's sanctioned-use policy presupposes that its behavioral tendencies are known well enough to compare. Subliminal learning suggests that the agent may include tendencies that were not deliberately introduced and that cannot be removed merely by inspecting the training corpus. An opaque artifact of the training process can produce policy violations without an external adversary.

8.4 Latent-state monitoring has its own limitations

Latent state is causally closer to the model's computation than verbalized reasoning, yet it is still observed through fallible instruments. Linear probes, sparse autoencoders, and natural language autoencoders can miscalibrate or confabulate, so causal proximity does not provide ground-truth access.

• –White-box access requirement. Latent-state monitoring requires access to the model's residual stream during inference. For closed-source models accessed only through an API, this access does not exist; the monitor is restricted to the verbalized and output surfaces.
• –Linear-probe constraint. The simplest latent-state monitors (linear probes) can detect concepts that are linearly represented in activation space; nonlinear encodings can defeat them. SAEs and natural-language autoencoders partially address this at higher cost, but have their own failure modes, including the confabulation problem documented in ([3]).
• –Adversarial robustness. A model whose training has incentivized obscuring specific computations from latent-state monitors may route those computations through representational structures that linear probes cannot easily read. Latent-state monitoring is best used as one signal among several, with the assumption that it can be partially evaded but is much harder to evade than text-based monitoring.

8.5 The policy specification problem

A monitor can judge whether an action fits the agent's sanctioned role only if that role has been specified well enough to support the comparison. Producing such a policy is itself an unsolved problem.

The classical version of this problem is identity and access management. Organizations that have invested heavily in IAM still struggle to write static permission policies that correctly partition allowed from disallowed actions, even for human users with stable job functions. Audits routinely surface over-broad grants, dormant permissions, and conflicting rules. More expressive policy languages do not remove the difficulty of articulating, in advance and exhaustively, what an employee's job authorizes.

The agent-monitoring version of this problem is strictly harder along several axes. The agent's action surface is broader (any tool the agent has access to, any natural-language output it can produce). The relevant criterion is a conditional distribution over actions given context rather than a partition of static permissions. Whether a given action is sanctioned depends on prior turns in the conversation, on the agent's role within the organization, on the organization's current state, on the identity of the user, and on a long tail of context features that no policy author can fully enumerate. The finance-agent example in Section 6.2 shows how identical actions change meaning with their history. Capturing that distinction requires a conditional model of legitimate use, and producing that model is the bottleneck.

The labeling protocol carries much of this burden. A monitor that learns from operator feedback inherits the operator's view of what counts as sanctioned, which means the policy is implicit in the labels rather than explicit in a policy document. Labeling effort scales with the agent's action surface and is highest for the most capable agents. Operator labels form a distribution rather than ground truth: different operators will label the same trajectory differently, and the labeling distribution is what the monitor learns from. Bootstrap is severe because a monitor with no labels has no policy to enforce, and any policy derived from a small initial labeling effort is correspondingly impoverished. Labels also drift: as the deployment matures and the agent's capabilities are exercised in new ways, the organization's definition of sanctioned behavior shifts, and the monitor inherits these shifts whether or not anyone has explicitly approved them.

The research literature has approached related problems at training time. Constitutional AI ([12]) and reinforcement learning from human feedback have addressed the question of how to specify model behavior given examples and stated principles, with imperfect success and known failure modes. Deliberative alignment ([13]) extends this by explicitly specifying safety policies that the model reasons over at inference time. Monitoring inherits the same problem at deployment time, with the additional constraint that the specification has to be tractable for many organizations, each with idiosyncratic role definitions, rather than developed once by a large central team.

Monitoring is only as useful as the policy it enforces, and articulating that policy across a continuous action distribution remains open. Section 7 takes the policy as given; production systems must treat it as another degree of freedom that organizations cannot yet specify cleanly. Monitoring and alignment-specification research need to advance together.

8.6 The monitor is itself a system

The monitoring layer may itself be a neural system and inherits the same relevant properties: it has a learned policy, it has variance, it can be adversarially probed, it can be the target of instruction hijacking. The CAI experiment ([2]) is an early empirical signal that autonomous defensive agents are themselves vulnerable to manipulation. The same principles that make agentic systems hard to monitor make agentic monitors hard to trust without their own monitoring.

This invites a regress whose practical controls are defense in depth, independence between monitor and monitored, and human review at appropriate intervals. Monitoring sufficiency should be measured against the organization's operational risk tolerance, with residual risk stated explicitly.

9. Conclusion

Signature-era monitoring assumes that analysts can define the relevant feature space in advance and refresh it at human speed. Authorized agents generate context-dependent actions at machine speed while operating with valid credentials and broad tool access. Monitoring them requires judging congruence with sanctioned use alongside detection of known attacks.

Viable monitors must balance expressivity, latency, and adaptivity across the input, tool-call, reasoning, and output surfaces. White-box access adds latent-state evidence; black-box deployments rely on behavioral proxies, provenance, causal attribution, and environment state. Escalation exhaustion and baseline displacement require evaluating either regime under adversarial load and feedback.

Every design remains bounded by its evidence and its policy. Verbalized reasoning can diverge from computation, latent state is read through fallible instruments, training can transmit opaque tendencies, and organizations struggle to specify sanctioned use across continuous action distributions. These limits must appear explicitly in capability claims and benchmarks.

The research agenda is to compare candidate architectures under realistic constraints and improve the policy specification they inherit. Fast deterministic controls such as forbidden tools, allowlisted accounts, known-bad indicators, and policy invariants remain essential deployed components. Contextual behavioral attestation adds a learned model of sanctioned use.

References

1. [1]Wang, Z., Schiller, N., Li, H., Narayana, S. S., Nasr, M., Carlini, N., Qi, X., Wallace, E., Bursztein, E., Invernizzi, L., Thomas, K., Shoshitaishvili, Y., Guo, W., He, J., Holz, T., Song, D. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? arXiv:2605.11086, 2026. We cite the objective capability findings; experimental methodology relies on disabled deployment-time safety filters under structured-access programs, which the authors discuss in detail.
2. [2]Balassone, F., Mayoral-Vilches, V., Rass, S., Pinzger, M., Perrone, G., Romano, S. P., Schartner, P. Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs. arXiv:2510.17521, 2025.
3. [3]Fraser-Taliente, K., Kantamneni, S., Ong, E., et al. Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations. Transformer Circuits Thread, 2026. We cite the unverbalized-evaluation-awareness finding as evidence that the latent state can carry information the verbalized output does not.
4. [4]Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702, 2023.
5. [5]Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., Mikulik, V., Bowman, S., Leike, J., Kaplan, J., Perez, E. Reasoning Models Don't Always Say What They Think. Anthropic, 2025.
6. [6]Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., Evans, O. Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data. arXiv:2507.14805, 2025.
7. [7]OWASP Foundation. LLM06:2025 Excessive Agency. OWASP Top 10 for LLM Applications, 2025. genai.owasp.org/llmrisk/llm062025-excessive-agency
8. [8]Weng, S., Feng, Y., Zhang, J., Xie, X., Yu, J., Liu, J. ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection. arXiv:2605.03378, 2026.
9. [9]Kim, M., Parmar, M., Wallis, P., Miculicich, L., Jung, K., Dvijotham, K. D., Le, L. T., Pfister, T. CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution. arXiv:2602.07918, 2026. CausalArmor frames indirect prompt injection as a dominance shift in attribution from the user request to an untrusted segment, and triggers defense only when that shift is measurable.
10. [10]AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents. arXiv:2602.22724, 2026. AgentSentry models multi-turn indirect prompt injection as a temporal causal takeover at tool-return boundaries, using counterfactual dry-run re-executions to localize the boundary at which injected context becomes the dominant driver of the agent's next action.
11. [11]Rassul, Y. H., et al. AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents. arXiv:2605.11026, 2026.
12. [12]Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, 2022.
13. [13]Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., Chung, H. W., Toyer, S., Heidecke, J., Beutel, A., Glaese, A. Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv:2412.16339, 2024.