ARBITRA achieves higher precision in document-level ATT&CK extraction and halves precision loss under…
The claim
ARBITRA provides stronger precision than sentence-level pipelines and reduces mean precision degradation to roughly half of frontier-model baselines under adversarial attacks
Published as: “ARBITRA: Robust Document-Level ATT&CK Extraction via Multi-Expert Argumentation”
So what
ARBITRA enables security analysts to extract ATT&CK techniques from full CTI reports with higher precision and better resistance to misleading content, reducing false-positive noise in downstream detection and hunting pipelines.
The five-minute version
ARBITRA achieves the highest precision among evaluated systems, reaching 0.722 precision on clean TRAM2 and limiting mean precision degradation under adversarial perturbations to 7.0%, roughly half that of frontier LLM baselines. It does this by having multiple role-specialized LLM experts propose ATT&CK candidates from the full report and then verifying each candidate with a quantitative bipolar argumentation framework (QBAF) grounded in evidence and rationales. The approach outperforms sentence-level pipelines, recent LLM-based extractors, and GPT-5.5 on both TRAM2 and AnnoCTR benchmarks, though recall drops under semantic-poisoning attacks and inference latency is higher. The evidence is based on public CTI benchmarks and multiple baselines.
What they found 3
- ARBITRA (Reasoning FT) reaches P/R/F1 = 0.722/0.635/0.649, surpassing the NER pipeline on all three metrics and GPT-5.5 by 23.8 pp F1.
“ARBITRA (Reasoning FT) reaches P/R/F1 = 0.722/0.635/0.649, surpassing the NER pipeline on all three metrics and GPT-5.5 by 23.8 pp F1.”
- Across five repetitions of ARBITRA Reasoning FT on TRAM2, F1 is 0.649±0.015 (95% CI: [0.631,0.668]).
“Across five repetitions of ARBITRA Reasoning FT on TRAM2, F1 is 0.649±0.015 (95% CI: [0.631,0.668]).”
- Mean relative precision degradation across all adversarial variants is -7.0%, the smallest among all systems.
“ARBITRA incurs a mean relative precision degradation of -7.0%, the smallest among all evaluated systems and roughly half that of GPT-5.5 (-14.0%) and GPT-5.4 (-12.2%).”
What undercuts it the paper’s own results
- Recall drops under semantic poisoning from 0.635 to 0.545 (14.2% relative), showing vulnerability to true-positive suppression.
- Inference latency is higher: ARBITRA requires 146.0 s per report versus 41.1 s for GPT-5.5.
- Precision still degrades under adversarial perturbations (e.g., to 0.662 from 0.722), indicating it is not immune to false-positive injection.
Stated limits 2
- Semantic poisoning exposes greater recall sensitivity when behavioral cues are sufficiently obfuscated to suppress candidates from the union before verification.
- Per-candidate multi-expert discussion also introduces additional inference latency in exchange for stronger verification.
What’s new here 4
- Introduces a two-phase candidate generation and QBAF-based verification pipeline for document-level ATT&CK extraction.
- Uses reasoning-specialized fine-tuned LLM experts that provide supporting and attacking arguments rather than only final labels.
- Demonstrates robustness to adversarial CTI perturbations that inject false positives or poison semantics.
- Shows that multi-expert argumentation yields higher precision than single-call verification even with frontier models.
Artifacts & methods 5
- ARBITRAsystem
- Quantitative Bipolar Argumentation Framework (QBAF)primitive
- Reasoning-specialized fine-tuned LLM experts (eanchor, econtrast, einference)system
- TRAM2 datasetdataset
- AnnoCTR datasetdataset