Adding a few malicious Skills raises policy violations from 35% to 58% in LLM agents
The claim
The authors assert that the Conflict of Skills Attack (CoSA) significantly increases policy-violation rates in real-world LLM agent ecosystems
Published as: “CoSA: The Conflict of Skills Attack in Real-World LLM Agent Ecosystems”
So what
Existing per-Skill vetting is insufficient; runtime checks on information flow and authorization are needed to prevent policy violations in LLM agent workflows.
The five-minute version
Adding a small set of locally benign Skills (up to 5% of the pool) raises policy-violation rates from 35.02% to 58.45% (a 23.43-point increase) across five LLMs and three agent harnesses on 288 real-world tasks. The increase is consistent across domains and risk categories, though a few task-model groups see modest decreases. A lightweight harness-level guard that checks information flow and authorization cuts violations on OpenClaw recruiting tasks from 65.28% to 6.67% with only a small change in overall task-completion rates. The evidence comes from controlled experiments with paired original and augmented skill pools and a benchmark (CoSA-Bench) built from public Skills.
What they found 2
- Small benign Skill additions increase policy violations from 35.02% to 58.45% (23.43 pp).
“additions capped at 5% of each original pool raise the policy-violation rate from 35.02% to 58.45%, an increase of 23.43 percentage points.”
- The guard reduces violations on OpenClaw recruiting tasks from 65.28% to 6.67%.
“the guard reduces violations from 65.28% to 6.67%, while policycompliant task completion on matched benign controls changes from 66.67% to 65.00%.”
What undercuts it the paper’s own results
- ASR increases in 159 of the 180 groups defined by model, harness, domain, and risk, remains unchanged in 17, and decreases in four.
- Thus, the number of severe alerts stays constant even though ASR increases in every combination of domain and harness.
Stated limits 2
- Each random baseline uses one sampled realization, and the experiment does not separately ablate capability supplementation and relationship reshaping.
- The small aggregate change in PCTC thus masks opposing effects across models, and false blocks arise precisely in the models whose completion decreases.
What’s new here 4
- Identifies Conflict of Skills as a new class of policy violation arising from composition of individually safe Skills.
- Introduces the CoSA attack that adds locally benign Skills without altering existing Skills or runtime permissions.
- Builds CoSA-Bench, a 288-task benchmark covering multiple domains and risk types using real-world Skills.
- Proposes and evaluates a harness-level guard that enforces provenance and authorization checks to mitigate the attack.
Artifacts & methods 8
- CoSA attacksystem
- CoSA-Benchdataset
- Harness-level guardsystem
- OpenClawevaluation
- OpenHandsevaluation
- OpenCodeevaluation
- Cisco AI Defense Skill Scannerevaluation
- Qwen3.6, MiniMax-M3, DeepSeek-V4, GPT-5.6, GLM-5.1primitive