Leakage-aware benchmark reveals existing models perform poorly on hybrid water CPS attack detection
The claim
Under a leakage-aware evaluation, no model exceeds a constant normal predictor for attack classification (macro-F1 = 0.465)
Published as: “AquaSense-ICS: A Hybrid Real-Simulated Cyber-Physical Dataset and Leakage-Aware Benchmark for Smart Water Quality Monitoring and Attack Detection”
So what
Current ML detectors for water-CPS attacks perform only marginally better than a naïve normal predictor, highlighting the need for more robust methods before deployment.
The five-minute version
Under a leakage-aware evaluation (chronological splits and leave-attack-type-out), no model beats a constant "normal" predictor (macro-F1 = 0.465) and the best detectors only achieve a mean AUROC of 0.64 on unseen attack types, indicating large headroom. Random row splits artificially inflate performance to near-perfect scores (macro-F1 = 0.9998 for Random Forest). The paper introduces AquaSense-ICS, a hybrid dataset of 23,688 real IoT samples (8 h recorded over 25.4 h) co-registered with a six-species EPANET-MSX simulation and a library of labeled attacks and faults.
What they found 3
- Random splits with feature-derived labels yield near-perfect classification
“macro-F1 = 0.9998 for Random Forest”
- Under leakage-aware protocol, no model exceeds the constant normal predictor (macro-F1 = 0.465)
“no model exceeds the constant “normal” predictor for attack classification (macro-F1 = 0.465)”
- Best detectors achieve a mean AUROC of 0.64 on attack types unseen during training
“the best detectors reach a mean AUROC of 0.64 on attack types unseen during training”
What undercuts it the paper’s own results
- The same dataset reports macro-F1 = 0.9998 under random splits, which dramatically overstates model performance compared to the realistic leakage-aware results.
Stated limits 4
- The real sensor acquisition phase covers a short (25.4 h recording window, 8.0 h recorded), limited geographic deployment involving a single indoor installation, which may not fully capture the spatial heterogeneity of largescale distribution networks or seasonal effects.
- The number of physical sensor modalities remains constrained by hardware availability. Only TDS is measured in the water; temperature, humidity and gas resistance describe the ambient air, and no immersed temperature probe was available for TDS compensation.
- The real node and the simulated network are independent systems aligned on a common clock; the dataset does not contain physically coupled real-simulated measurements.
- Water-quality labels are rule-based and have not been validated against laboratory grab samples.
What’s new here 4
- First publicly released dataset that combines real low-cost IoT water-quality sensing with multi-species EPANET-MSX hydraulic simulation.
- Provides a parameterised attack and fault library with sample-level ground truth applied to both real and simulated channels.
- Introduces a leakage-aware benchmarking protocol (chronological and leave-attack-type-out splits) that reveals true detection performance.
- Includes time-series augmentation (TimeGAN) and explicit origin labeling to separate synthetic from real data.
Artifacts & methods 9
- AquaSense-ICS datasetdataset
- EPANET-MSX simulationsystem
- Attack and fault generatorprimitive
- TimeGAN augmentationprimitive
- Random Forest classifiersystem
- XGBoost classifiersystem
- Isolation Forestsystem
- One-class SVMsystem
- SHAP explainabilityevaluation