LLMs can be tricked into impersonating users via role spoofing
The claim
LLMs are highly susceptible to role spoofing attacks that combine fake structure and content
Published as: “Understanding and Mitigating Role Spoofing in LLMs”
So what
Deployers of LLM agents can add a lightweight linear-probe detector to flag role-spoofing injections, reducing the risk of unauthorized actions without high computational overhead.
The five-minute version
Role spoofing attacks that combine fake content and fake special-token structure cause LLM agents to impersonate the user over 95% of the time on several frontier models. The paper shows that adversarial training only mitigates content-based attacks while leaving models vulnerable to structure-based attacks, and proposes linear probes over internal activations that detect role spoofing with near-perfect separation and lower false-positive rates than prior methods. Evidence is based on experiments across multiple models in the AgentDojo benchmark and a custom tool-use testbed.
What they found 2
- Role spoofing attacks achieve >95% impersonation success on several models.
“role spoofing attacks were able to impersonate the user over 95% of the time for several models including Claude Opus 5 and GPT-5.4.”
- Linear probes achieve near-perfect separation between clean and injected cases.
“Our probes achieve near perfect separation between clean and injected cases, across a variety of held out test cases”
What undercuts it the paper’s own results
- adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks.
Stated limits 2
- a limitation inherent to our probes is that they can only be used to detect role spoofing, and cannot detect prompt injection attacks that do not involve role spoofing.
- we acknowledge that the type of role spoofing we discuss in this paper may be entirely solved in future models by incorporating training data with such adversarial examples.
What’s new here 4
- Systematic analysis of how LLMs track conversation roles using both content-based and structure-based cues.
- Demonstration that adversarial training against content attacks shifts models toward reliance on structure cues, not solving role spoofing.
- Introduction of linear role probes trained on random-content conversations that detect role spoofing more effectively than prior probes.
- Toy-model study illustrating mechanism shifts during pretraining, instruction-tuning, and adversarial fine-tuning.
Artifacts & methods 5
- role spoofing attacks (fake content + fake special tokens)
- AgentDojo benchmarkdataset
- custom tool-use testbedevaluation
- toy two-layer transformer studytheory
- linear role probes over model activationssystem