Watermark basin detection identifies 79% of distilled LLMs at 0.1% false-positive rate

The five-minute version

WM-Basin detects 79% of distilled models at n-gram=1 and 45% at n-gram=3 with a 0.1% false-positive rate, far outperforming the prior token-level detector (29% and 1.3%). The method works across 24 student architectures, four teacher models, and four watermark schemes, and survives quantization, pruning, and reinforcement-learning post-training. Detection degrades when only a small fraction of training traces are watermarked or when the student is further fine-tuned on clean data. The evidence is based on 1,200 distilled models evaluated on public math benchmarks.

What they found 5

• WM-Basin achieves high true-positive rates at very low false-positive rates, far surpassing the existing token-level detector.

  “Among 1,200 student models distilled from DeepSeek-R1Distill-Qwen-14B (Guo et al., 2025) on OpenR1-Math reasoning traces (Hugging Face, 2025), at a 0.1% false-positive rate, WM-Basin detects 79% of distilled models at n-gram=1 and 45% at n-gram=3, against 29% and 1.3% for the black-box detector (Sander et al., 2024).”

• Detection performance is robust across a wide range of student architectures.

  “Pooled over all architectures (the Overall row of Tab. 1), WM-Basin outperforms the baseline at every window and operating point, and at the 5% and 1% thresholds its margin widens with the window.”

• The method works for multiple teacher models.

  “Tab. 3 shows that, at p ≤0.05, WM-Basin detects 96 to 100% of distilled models for every teacher.”

• WM-Basin transfers to different watermarking schemes without modification.

  “Tab. 4 shows that all three transfer: WM-Basin flags every student at p≤0.01 and n-gram=1 under each scheme, and it outperforms the baseline in every cell, by a margin that widens with the window at p≤0.05.”

• Standard model compression does not remove the watermark basin.

  “Tab. 6 shows that only int8 and 10% pruning leave a usable model, and both cost WM-Basin at most three points at any window and operating point.”

What undercuts it the paper’s own results

• WM-Basin performs at chance when at most 10% of the traces are watermarked and exceeds 90% TPR once 75% are.
• Continued SFT on clean data substantially reduces WM-Basin's power, whereas RL post-training leaves detection intact.

Stated limits 2

• A watermark can be present in a distilled model but unreadable in its text: continued SFT on clean data suppresses the text-level signal while an open-weight probe still recovers the residual, and a supplementary study (App. D.3) finds the same for rewriting the teacher’s traces before distillation.
• Both are limits on what a black-box verifier can observe, not on what distillation transfers.

What’s new here 4

• Detects watermark distillation via alignment of probe-gradient representations rather than token-level counts.
• Provides exact finite-sample p-values using a non-parametric rank test against decoy keys.
• Remains effective under model compression, pruning, and RL post-training.
• Works across multiple teachers, watermark schemes, and student architectures without needing the watermark’s own detection test.

Threat model stated by the source

Artifacts & methods 8

• WM-Basinsystem
• Token-level black-box detector (Sander et al., 2024)system
• Probe gradient mappingprimitive
• Watermark basin geometrytheory
• OpenR1-Math-220k datasetdataset
• GSM8K benchmarkdataset
• DeepSeek-R1-Distill-Qwen-14B and other teacher modelssystem
• Non-parametric rank test (Dwass, 1957)primitive

Topics