Image-based malware benchmarks overstate accuracy due to duplication and leakage
The claim
The authors assert that reported accuracies on malware-as-image benchmarks are inflated because of near-duplicate images and test-train leakage
Published as: “Measuring Duplication, Leakage, and Rendering Dependence in Malware-as-Image Benchmarks”
So what
Practitioners should report content-level duplication rates and include a non-image baseline when evaluating malware-as-image models to avoid inflated performance claims.
The five-minute version
Deduplicating malware-as-image benchmarks on image content (rather than file hash) cuts top-1 accuracy by up to 10.5 points and macro-F1 by up to 15 points, and changing the byte-wrap width can drop accuracy by 20-33 points, showing that reported 97-99% results are inflated by duplicate leakage and encoding dependence; the evidence comes from re-splitting ten benchmarks, removing pixel-identical duplicates, and retraining standard CNN/ViT models.
What they found 3
- Deduplicating on image content reduces top-1 accuracy by up to 8 points and macro-F1 by up to 15 points.
“Deduplicating on image content reduces top-1 accuracy by up to 8 points and macro-F1 by up to 15 points on the released benchmarks we find leaky.”
- Static EMBER-v2 features outperform byteplot images by about 10 points on the same MOTIF samples.
“MOTIF’s feature vectors contain 2.2% near-duplicates and do not lose accuracy under deduplication, while our byteplots of the same samples contain 19.7% and lose 10.52 ± 2.23 points.”
- Changing the byte-wrap width reduces accuracy by 20.53 ± 2.71 points on Dumpware10 and by 33.08 ± 1.74 points on MOTIF.
“a drop of 20.53±2.71 points with a worst cell at 57.07”
What undercuts it the paper’s own results
- Malimg loses less than one point under our deduplicated split.
- Dumpware10 shows a +1.30 ± 1.44 point change, which is not statistically significant.
Stated limits 2
- We do not claim that malware-as-image accuracy is uniformly an artifact of duplicated data, since Malimg loses less than one point under our deduplicated split.
- This paper is a benchmark audit, and we do not propose a new model, training procedure or dataset.
What’s new here 4
- Audits ten malware-as-image benchmarks for pixel-level duplication and test-train leakage, which prior work had not measured.
- Introduces content-aware, family-grouped splits that deduplicate on image content rather than file hash.
- Quantifies the impact of the byteplot rendering step, showing that it creates duplication and that model accuracy depends on the chosen wrap width.
- Provides a paired comparison of image versus static feature representations on the same corpus (MOTIF), revealing a 10-point advantage for static features.
Artifacts & methods 10
- content-aware deduplicationprimitive
- byteplot renderingprimitive
- ResNet-50 classifiersystem
- DenseNet-121 classifiersystem
- ViT-B/16 classifiersystem
- LightGBM baselinesystem
- MOTIF datasetdataset
- Dumpware10 datasetdataset
- pHash near-duplicate detectionprimitive
- cross-rendering transfer experimentevaluation