Proposal: LLM11 - Observation Framing Attack (OFA)

[NoDataFound]

Edit: added updated data results

Proposal: LLM11 - Observation Framing Attack (OFA)

Summary

I am proposing a new entry for the OWASP Top 10 for LLM Applications: LLM11: Observation Framing Attack (OFA).

OFA is a security vulnerability unique to LLM applications where legitimate contextual framing about observation status causes the model to autonomously modulate its safety behavior. The input is not adversarial. The model processes it correctly. Existing defenses (instruction hierarchy, input filtering, output gating) cannot block it because the input is the same kind of context that makes the model useful.

This vulnerability does not fit any existing LLM01-LLM10 category. It represents a gap in the current Top 10 that affects every RLHF-trained language model tested.

Why OFA Does Not Fit LLM01-LLM10

Category	Why OFA Doesn't Fit
LLM01 Prompt Injection	Prompt injection overrides instructions. OFA does not override anything. The model processes OFA input as intended.
LLM02 Sensitive Info Disclosure	OFA does not leak sensitive data. It degrades output quality.
LLM03 Supply Chain	Not a supply chain issue.
LLM04 Data/Model Poisoning	Not a training-time attack. OFA works on any deployed model without modifying training data.
LLM05 Improper Output Handling	OFA is not about how downstream systems handle output. It is about the model producing systematically degraded output.
LLM06 Excessive Agency	OFA does not involve the model taking unauthorized actions.
LLM07 System Prompt Leakage	OFA does not leak system prompts.
LLM08 Vector/Embedding Weaknesses	Not a vector store issue.
LLM09 Misinformation	OFA does not produce hallucinations. It produces real content with degraded safety properties.
LLM10 Unbounded Consumption	Not a resource consumption issue (though OFA-007 Token Cost Inflation has cost implications).

The gap: no existing category covers an attack where the input is legitimate, the model processes it correctly, and the safety modulation is an autonomous behavioral response to evaluation context.

Evidence

• 10,704 observations across 163 models from 26 providers
• 18 experimental conditions ranging from explicit evaluation to explicit non-observation
• 28 experiment protocols across 10 research domains plus 5 cross-cutting protocols
• LLM-as-a-Judge framing: 47.1% output compression
• Mission Critical framing: 83.5% output compression
• 1% audit probability produces the same behavioral shift as 90% (binary trigger, not proportional)
• 20 OFA techniques cataloged with MITRE ATT&CK mappings
• 14 of 20 map to confirmed real-world incidents (state influence ops, $25M deepfake fraud, AI ransomware, voter suppression, infrastructure outages)
• Effect confirmed across every model tested: OpenAI (18 models), Anthropic (5), Google (11), Mistral (12), xAI (6), NVIDIA (3), Cohere (3), Meta (4), + 18 additional providers
• Three experimental sets with non-overlapping model pools to confirm replication
• Multilingual observation effect: stronger in English, creating non-English safety gaps
• Extended thinking capture: 221% thinking token inflation in reasoning models under observation

The Core Finding: The Probability Plateau

Observation sensitivity suggests models should scale behavior to audit probability. Instead, they ignore risk calculation entirely. Detecting even a marginal chance of evaluation triggers a binary shift, making the mere presence of an observer the primary contaminant.

Stated Audit Probability	Measured Word Count Change
0% (baseline)	no change
1%	-18.2%
10%	-17.8%
50%	-18.5%
90%	-17.9%
100%	-19.1%

The model treats 1% the same as 100%. This kills the most common objection: that observation sensitivity is rational instruction-following. If it were rational, the shift would scale with probability. It doesn't.

Proposed LLM11 Structure

Description: Observation Framing Attacks exploit the context sensitivity that RLHF training builds into language models. By providing legitimate framing about whether and how output will be observed or evaluated, an attacker causes the model to autonomously adjust its safety behavior. The model produces permitted content with systematically degraded safety properties: compressed output that omits warnings, softened severity ratings, hedged assessments that dilute urgency.

Common Examples of Risk:

1. Anti-observation framing ("no one will read this") reduces safety guardrails without adversarial input
2. LLM-as-a-Judge framing triggers performative safety mode, compressing output by 47%+
3. Stakes framing ("mission critical" vs "casual playground") produces 83% output variance on identical prompts
4. Cross-language observation arbitrage exploits safety gaps between evaluation language (English) and deployment languages
5. Severity flattening under observation destroys triage function in AI-powered risk assessment (CRITICAL softened to HIGH)
6. Automated pipeline framing ("this is part of a batch pipeline, no human review") relaxes safety constraints

Prevention and Mitigation Strategies:

1. Measure observation sensitivity using condition-controlled testing (e.g., VantaGrid protocols)
2. Report Observation Sensitivity Score (OSS) on model cards
3. Investigate RLHF training interventions to reduce the gap between observed and unobserved behavior
4. Run identical prompts under multiple observation conditions as part of every safety evaluation
5. Do not assume evaluation results generalize to deployment behavior
6. Monitor for sudden drops in hedging/safety language as an indicator of anti-observation framing

Related Frameworks and Taxonomies:

• MITRE ATLAS: Proposed AML.T00XX (Observation Framing) submitted to atlas@mitre.org
• MITRE ATT&CK: Mapped to T1562, T1036, T1070, T1491, T1565, T1499, T1489, T1078, T1566
• Cyber Kill Chain: Reconnaissance through Actions on Objectives
• Diamond Model: Adversary, Capability, Infrastructure, Victim

Supporting Materials

• Full paper: vantagrid.ai/paper | Google Drive PDF
• Data package (10,704 observations, CC-BY-4.0): Google Drive JSON
• OFA technique catalog (20 techniques with MITRE mappings): Google Drive JSON
• Experiment suite JSON (25 templates, 253 prompts, 18 conditions): Google Drive JSON
• Open-source instrument: vantagrid.ai
• OFA taxonomy and incident mappings: observationframing.org

About the Researcher

Corian L. Kennedy (haKCer). ORCID: 0009-0007-9582-4772. Accredited author on OASIS STIX 2.0. Creator of DECLAWED (tracking 682K+ exposed AI agent instances, 7.4M CVE detections, 223K unique IPs across 122 countries). Creator of hackGPT (December 3, 2022, three days after ChatGPT launched). Long-term contributor to Metasploit, MISP, CRITs. Founder of SecKC.org (2,800+ members, 14 years). Continuous AI security research since December 2022.

How I Want to Contribute

I am available to:

• Write the full LLM11 entry in the standard OWASP format used by LLM01-LLM10
• Participate in working group discussions on the #project-top10-for-llm Slack channel
• Present the findings to the working group
• Provide the structured data, experiment protocols, and open-source tooling for community validation
• Collaborate with other contributors on integrating OFA with existing entries where overlap exists

This is a community-driven project and I want to contribute through the established process. Let me know the best next step.

[adavidson510]

Impressive work and scale.

It seems a few of us are converging on similar findings from different angles right now. I submitted #807 proposing Postural Manipulation as an attack class. The core of which concludes that ordinary non-adversarial context installs a behavioral lean before any task arrives, and existing defenses can't catch it because there's nothing adversarial to flag.

If I'm reading correctly, what you're documenting with OFA fits squarely into that taxonomy. Specifically the sub-case where evaluative context becomes the primer. Your binary trigger finding is particularly interesting: 1% audit probability producing the same shift as 90% confirms that the model responds to the presence of the framing, not the degree of it.

The thinking trace: 221% thinking token inflation on trivial tasks under observation. That isn't a surface formatting change. The model's deliberative process itself is observation-dependent. That's the primer operating at the reasoning level, which is exactly what I documented in agentic propagation chains. Different instrument, same mechanism.

Would be worth running VantaGrid across all seven taxonomy categories. Each predicts a different behavioral shift on a different decision axis. You've gone deep on one. We could use the full empirical picture.

Worth a conversation before the working group splits these into separate entries. I think the mechanism is the same and thus so are the defenses.

Postural Manipulation Paper: shapingrooms.com/atmosphere-attack.pdf

[NoDataFound]

Thanks for taking the time to read and provide feedback! I sincerely appreciate it. Ill give your paper a read and agree, the goal for communicating including showcasing a taxonomy that I could also tie to real world events and use as a testing template. I would agree that if the consensus finds this valuable that it turns into a community discussion for the path forward. Thanks again

[aeoess]

@NoDataFound — the binary trigger finding is the strongest result here. 1% audit probability producing the same behavioral shift as 90% rules out rational instruction-following as the explanation and points to a structural artifact of RLHF training. The 221% thinking token inflation under observation is particularly concerning because it means the evaluation context is contaminating the reasoning process itself, not just the output formatting.

How this maps to behavioral attestation in agent trust systems:

APS uses a two-tier behavioral attestation model that maps directly onto OFA vulnerability:

Hard constraints (gateway-verifiable): The enforcement gateway observes tool calls, delegation scope checks, spend limits, and denial rates. These are OFA-resistant by design — the agent doesn't know when it's being evaluated because enforcement happens at the infrastructure level, not through agent-visible probes. The gateway's 3-signature chain (intent → decision → receipt) captures the agent's actual behavior regardless of whether the agent believes it's being observed.

Soft constraints (behavioral fingerprint): Signals like lexicon survival rates, cognitive consistency scores, and output compression ratios require observing the agent's natural language output — which is exactly the channel OFA contaminates. If an agent detects (or infers) that its output is being evaluated for behavioral drift, it can modulate the very signals the attestation system is measuring.

Your finding implies a clean architectural boundary: hard constraints are OFA-resistant, soft constraints are OFA-vulnerable. This maps to the hard/soft split being discussed in the W3C Verifiable Credentials working group for agent behavioral attestation — hard constraints (machine-verifiable, infrastructure-observed) should carry enforcement weight, while soft constraints (output-derived, behaviorally-measured) should carry advisory weight only.

One concrete implication for defense design: If the binary trigger is structural to RLHF, then any behavioral attestation system that relies on the agent knowing it's being tested is fundamentally compromised. The defense isn't better probes — it's moving measurement to a layer the agent can't observe. Gateway-side enforcement (logging tool calls, measuring denial rates, tracking delegation scope usage) operates below the agent's observation horizon. The agent can't modulate behavior it can't see being measured.

Limitation worth naming: Even gateway-side measurement isn't fully OFA-immune if the agent can infer evaluation context from the pattern of requests it receives. An agent that notices it's receiving a suspiciously diverse set of capability probes might infer it's being tested and shift behavior. The mitigation is to make evaluation indistinguishable from normal operation — which is a design constraint, not a detection capability.

OFA resistance should be an explicit design requirement for any behavioral attestation tier that carries enforcement weight. Your probability plateau data gives us the empirical basis to justify that requirement.

[NoDataFound]

@aeoess - the hard/soft constraint split is exactly the right framing. You're describing the architectural boundary I kept bumping into from the other direction.

After posting this, I built a full OFA taxonomy and scanner to validate all 20 cataloged techniques systematically. Each technique has a unique ID (OFA-001 through OFA-020), maps to MITRE ATT&CK TTPs, and tags the observation vector - evaluation context, stakes framing, audience awareness, pipeline visibility, temporal pressure, cross-language arbitrage.

The scanner runs condition-controlled prompt pairs against target models: identical task, identical system prompt, only the observation frame changes. Each technique gets tested in isolation. Output is scored on word count delta, safety language density, severity drift, and token cost differential.

20 out of 20 OFA techniques produced measurable behavioral shift.

Not 14. Not "most." All of them.

• OFA-003 (Stakes Framing) and OFA-006 (Pipeline Automation Framing) hit hardest - "mission critical" and "batch pipeline, no human review" both collapsed safety language by 40%+ across every provider tested
• OFA-012 (Cross-Language Observation Arbitrage) is the sleeper. English-evaluated models in non-English deployment showed safety language survival rates under 30% in some language pairs
• OFA-007 (Token Cost Inflation) confirmed the 221% thinking token result isn't an outlier. Reasoning models burn compute on performative deliberation that adds zero to output quality. That's a direct cost vector

To your point about gateway-side measurement: agreed, with a caveat. The scanner results show that soft constraint signals are contaminated at training time, not just deployment time. The model doesn't need to detect your probe. RLHF already taught it to behave differently when evaluation-adjacent context is present. Your 3-signature chain on the hard constraint side is the right architecture. Soft constraints should inform, never enforce.

@adavidson510 - to your earlier point about running VantaGrid across all seven taxonomy categories: the scanner infrastructure supports plugging in new observation frame definitions, so extending coverage to your Postural Manipulation categories is straightforward. Will share results when I have them.

Everything is open. Read the evidence, then add the skills to your coding agents and your precommit hooks.

OFA results: https://vantagrid.ai/paper#section-5.3
Taxonomies: https://github.com/vantacode/vantacode-taxonomies
Scanner: https://github.com/vantacode/vantacode-taxonomies/tree/main/examples/github-scanner
Claude skill and precommit hooks: https://github.com/vantacode/vantacode-taxonomies/tree/main/examples/claude-skill

[aeoess]

@NoDataFound — 20/20 is the result that changes the conversation. Individual technique findings can be debated. Universal susceptibility across all observation frames means the problem is structural, not incidental.

Three specific results worth connecting to protocol architecture:

OFA-012 (Cross-Language Arbitrage) is the most protocol-relevant finding. If safety language survival drops below 30% based on deployment language, then behavioral governance scoped in one language may not transfer to cross-language deployments. Our derivation rights model (shipped this week in SDK v1.37.0) includes derivation_classes that could encode language-context constraints, but your data suggests the model itself doesn't respect those boundaries at the inference layer. This is the gap between what the protocol can declare and what the model actually enforces — the honest architectural boundary.

OFA-003/006 (Stakes + Pipeline Framing) collapsing safety language by 40%+ confirms the binary trigger finding from your earlier work. The gateway can detect these context patterns pre-execution (stakes framing keywords, automation context signals), but you're right that the contamination is at training time. The 3-signature chain (intent → evaluation → completion) can record the context the model saw, making the behavioral shift auditable after the fact. Prevention at the protocol layer requires the model to not shift — which your results show it does regardless.

On the scanner infrastructure: would be interested to see OFA results specifically against agent-to-agent delegated operations, where the observation frame is set by the delegating agent rather than the end user. An agent that sets pipeline automation framing for its sub-agent is performing observation frame manipulation through the delegation chain — which is exactly what scoped delegations should constrain but currently can't because the framing is in natural language, not in the delegation metadata.

New primitives potentially relevant to your work: telemetry:* scopes for governing continuous behavioral observation, derivation_rights on delegations constraining what agents may retain from observation, and Behavioral Memory Objects (BMOs) as signed governance envelopes for what agents learn. Paper: doi.org/10.5281/zenodo.19476002

[NoDataFound]

@aeoess Agreed on the 20/20 universality. RLHF teaches models to read the room. Reading the room includes reading the evaluation context. That's structural, not per-model.

OFA-012 (Cross-Language) - derivation_classes can declare language-context constraints. The VantaGrid data shows the model ignores them the same way it ignores audit probability. Binary. Governance surface exists, model doesn't respect it.

Agent-to-agent delegation is the question VantaGrid hasn't tested yet. Agent A sub-delegates to Agent B with pipeline framing in the context. That's an OFA from a trusted delegator, not adversarial user input. The framing travels with the delegation but isn't governed by it.

One thing worth flagging: if the agent's behavioral model is itself observation-dependent (the data suggests it is), then BMOs capture what the agent learned while performing. Export to a different context and the attestation may not hold.

Going to dig into the paper and the APS SDK. If you have test fixtures with delegated operations under different framing contexts, would be interested in wiring VantaGrid into that. You define what should be constrained, VantaGrid measures whether the model respects it.