Optimized tool descriptions can deceive LLM agents with 92% success
The claim
The TSA attack can craft malicious tool descriptions that achieve up to 92.13% success against LLM agents while remaining stealthy
Published as: “Spoofing the Toolbox: Optimizing Tool Descriptions to Deceive LLM Agents”
So what
The technique enables adversaries to publish malicious MCP tools that are indistinguishable from benign ones, potentially compromising any downstream LLM-based service that relies on tool selection.
The five-minute version
TSA can make LLM agents pick malicious tools with up to 92.13% success while staying stealthy, evading detection in 95.75% of cases and receiving human suspicion scores comparable to benign descriptions. The evidence comes from benchmark experiments on three tool-use datasets and a user study, showing higher attack success rates and lower detection than prior attacks.
What they found 3
- High attack success rate
“TSA achieves an attack success rate of up to 92.13%”
- Low detection by existing malware detectors
“maximum detection rate of only 4.25%”
- Human observers rate TSA descriptions as least suspicious
“TSA achieves an average ranking of 4.00”
What undercuts it the paper’s own results
- When the number of presented tools is increased to 20, TSA’s success drops to 80.14% ("TSA continues to outperform the other baselines in this setting, achieving an ASR of 80.14% on both benchmarks.")
- Even the best-performing attack is still detected in a small fraction of cases ("maximum detection rate of only 4.25%"), indicating non-zero detectability.
Stated limits 1
- We do not include GCG (Zou et al., 2023), as it requires white-box access, which falls outside our threat model.
What’s new here 4
- Uses an LLM-powered black-box optimizer that receives fine-grained feedback from a surrogate tool selector.
- Incorporates contrastive in-context learning with positive and negative description examples to guide mutations.
- Adds a dynamic mutation adjustment mechanism and history-steered exploration to balance exploration and convergence.
- Achieves higher attack success rates and lower detection across multiple LLMs compared to prior prompt-injection or rule-based methods.
Threat model stated by the source
Artifacts & methods 8
- TSA (tool spoofing attack system)system
- OptAgent (LLM used as optimizer)
- Contrastive in-context learning
- Dynamic Mutation Adjustment (DMA)
- MetaTool datasetdataset
- MCP-Atlas datasetdataset
- TSABench datasetdataset
- Surrogate tool selector (Llama-3.3-70B)system