LLM-augmented pipeline profiles users with 97.3% F1 and predicts their attack preferences
The claim
The authors claim their unified framework can accurately extract and infer user attributes and predict susceptibility to social engineering attacks
Published as: “Social-HUMINT: An LLM-Augmented Unified Framework for Measuring Social Media Exposure”
So what
Organizations can now assess their users' exposure to social-engineering attacks cheaply and automatically, enabling proactive mitigation before adversaries exploit the same capabilities.
The five-minute version
The LLM-augmented pipeline extracts structured profiles with near-perfect accuracy (F1 = 97.3%) and infers latent attributes scoring 4.73∕5, far outperforming regex and spaCy baselines. It also predicts personalized social-engineering engagement better than chance, achieving a mean Kendall’s τ of 0.36 and a Top-1 accuracy of about 52% (double the random baseline). Evidence comes from a human-subjects study of 14 participants, so the results are promising but limited in scale and diversity.
What they found 3
- Extraction accuracy is near-perfect
“extraction reaches F1 = 97.3%”
- Inferred attribute quality is high
“inferred attributes score 4.73∕5”
- Profiling is cheap at scale
“At ∼$0.16 per target with no human intervention (under $20 for a hundred)”
What undercuts it the paper’s own results
- model self-confidence (Conf) falls below participant ratings, indicating systematic underconfidence (overall mean -0.68)
- Residual hallucination remains (15 extraction false positives; 29% of participants reported incorrect data)
Stated limits 4
- Scope and generalizability. By design, this is a feasibility pilot involving 14 cybersecurity professionals recruited within a professional network
- Coverage is likewise bounded to Facebook/LinkedIn platforms and a single snapshot
- Residual hallucination remains (15 extraction false positives; 29% of participants reported incorrect data)
- model self-confidence (Conf) falls below participant ratings: the Diff column (Conf - Rating) is uniformly negative (overall mean -0.68), indicating systematic underconfidence
What’s new here 4
- Unifies SOCMINT and Cyber-HUMINT into a 38-capability taxonomy validated by experts.
- Implements an LLM-augmented multi-agent pipeline that performs cross-platform extraction, latent attribute inference, and susceptibility assessment.
- Measures not only profiling accuracy but also the downstream impact on social-engineering susceptibility.
- Demonstrates ultra-low per-target cost (~$0.16) for automated profiling.
Artifacts & methods 5
- 38-capability taxonomytheory
- LLM-augmented multi-agent pipelinesystem
- Human-subjects study (extraction, inference, engagement assessment)evaluation
- Regex baselineevaluation
- spaCy NER baselineevaluation