Protocol mutation cuts copied answer success by ~10% in smart contract security tasks
The claim
Protocol mutation reduces the success rate of copied public answers by 10.9 percentage points and lowers copying frequency by 9.3 points
Published as: “Beyond Copying: Lifelong Smart Contract Security Task Construction via Protocol Mutation”
So what
The approach enables more robust evaluation of LLM agents on smart contract security by forcing them to adapt attacks rather than copy memorized solutions.
The five-minute version
CONSTRAINTDIAL reduces the success of LLM agents that rely on public answers by 10.9 percentage points on average across detect, exploit, and patch tasks, and cuts the fraction of tasks solved by copying online answers by 9.3 points. The framework automatically reconstructs contracts, mutates protocol conditions, and validates that the original attack fails while the vulnerability remains exploitable and normal functionality is preserved. They built a benchmark (ReCast) with 3,045 task inputs from 203 public cases and evaluated six models, finding mixed effects on overall success rates. The paper does not provide a dedicated limitations section, but notes that only a 30% sample of cases was used and that confidence intervals include zero, indicating statistical uncertainty.
What they found 3
- Protocol mutation reduces success with supplied public answers by 10.9 percentage points on average
“Compared with reconstruction alone, protocol mutation reduces success with supplied public answers by 10.9 percentage points on average across detect, exploit, and patch.”
- The fraction of tasks solved by copying online answers falls by 9.3 points after mutation
“The fraction of tasks solved by copying online answers falls by 9.3 percentage points on average.”
- Public answers can still explain the vulnerability and repair after mutation, even though attack calls must change
“After mutation, public answers can still explain the vulnerability and how to repair it, while the attack calls or inputs need to change.”
What undercuts it the paper’s own results
- Exploit success rises for four models and falls for two, so overall success does not uniformly decrease.
- All 95% confidence intervals for success rate changes from Base to Combined changes include zero (Appendix C.1).
Stated limits 2
- To limit cost, we randomly sample 30% of the 203 public cases, rounding up to 61.
- All 95% confidence intervals for success rate changes from Base to Combined changes include zero (Appendix C.1).
What’s new here 4
- Automated lifelong construction of smart contract security tasks via protocol mutation
- Changes required calls or inputs for attacks while preserving the underlying vulnerability
- Full automation of contract reconstruction, mutation proposal, and validation without manual annotation
- Paired checks of vulnerable and repaired protocol versions to ensure exploitability and normal operation
Artifacts & methods 7
- CONSTRAINTDIALsystem
- ReCast benchmarkdataset
- protocol mutationprimitive
- RECONSTRUCTPAIR algorithmprimitive
- TRACEDEPENDENCIESprimitive
- SELECT mutation proposalprimitive
- CHECK validation procedureevaluation