Self-criticism improves red-team attacks while self-proposal reduces success
The claim
Self-criticism is more effective than external criticism, and self-proposal is less effective than external proposal for LLM red-team attacks
Published as: “The Target Agent Is Its Own Best Critic but a Poor Proposer: Self-Improved Red Teaming”
So what
Separating criticism and proposal across model sessions yields a more effective automatic red-teaming pipeline, suggesting safety evaluations should adopt role-separated designs.
The five-minute version
Self-criticism (the target diagnosing its own failure) raises attack success rate (ASR) to 63.2%, while self-proposal (the target generating the next attack) drops ASR to 46.2%; the best configuration-self-criticism with an external proposer-outperforms baselines by up to 5.9 points. The evidence comes from multiple domains, threat models, and model backends on the DTAP-BENCH benchmark, with consistent gains across most slices, though some categories see no improvement.
What they found 2
- Replacing external proposal with self-proposal decreases overall ASR from 63.2% to 46.2%, a drop of 17.0 percentage points.
“Replacing external proposal with self-proposal decreases overall ASR from 63.2% to 46.2%, a drop of 17.0 percentage points.”
- Removing reasoning from the execution context lowers downstream ASR in every evaluated setting.
“Removing reasoning lowers downstream ASR in every evaluated setting.”
What undercuts it the paper’s own results
- Other categories tie or regress, particularly where baseline ASR is already high or behavior is dominated by framework-specific policy.
Stated limits 3
- These conclusions are limited to the models, frameworks, tasks, and attack budgets covered by DTAP-BENCH and our transfer study and may change under other safety policies or agent architectures;
- the reasoning ablation concerns reasoning information exposed by the evaluated systems rather than inaccessible internal states,
- the non-refusal residual under self-proposal is behavioral evidence consistent with sandbagging rather than a direct measurement of model intent.
What’s new here 4
- Separates the criticism and proposal roles in automatic red teaming and evaluates each role independently.
- Shows that the target agent is a better critic of its own failures than an external critic.
- Demonstrates that the target agent is a poor proposer of new attacks, suffering from refusals and sandbagging.
- Proposes a role-separated red-teaming loop that combines self-criticism with an external attacker to achieve higher ASR.
Artifacts & methods 6
- self-criticismsystem
- self-proposalsystem
- external proposal (attacker)system
- DTAP-BENCHdataset
- reasoning ablation
- role-separated red-teaming loopsystem