On-policy self-distillation reveals member data via sparse correction signals

The five-minute version

OP-Print can reliably infer membership in on-policy self-distilled language models, achieving up to 7.7% higher AUC and 26.5% higher TPR@5%FPR than the strongest prior black-box baselines on Qwen3-8B. The attack works in both gray-box (using teacher-student residuals) and black-box (using restart-based output-basin stability) settings, consistently outperforming likelihood-based and rollout-derived baselines across three model sizes. Evidence comes from controlled experiments on 1k OpenThoughts reasoning problems (500 members, 500 non-members) fine-tuned on Qwen3-1.7B/4B/8B, showing AUC gains from ~0.52 to ~0.61 in gray-box and ~0.55 to ~0.58 in black-box. However, absolute AUC values remain modest (≈0.55-0.60), indicating the signal is weak and the attack incurs extra inference cost.

What they found 3

• OP-Print improves black-box AUC by 7.7% and TPR@5%FPR by 26.5% over the strongest baseline on Qwen3-8B

  “For example, on Qwen3-8B (Yang et al., 2025) under the black-box setting, OPPrint improves AUC by 7.7% and TPR@5% FPR by 26.5% over the strongest baseline.”

• In gray-box attacks OP-Print-G-Max raises AUC from 0.517 to 0.594 (1.7B), 0.517 to 0.611 (4B), and 0.529 to 0.587 (8B)

  “In the gray-box setting, OP-Print-G-Max achieves the highest AUC across all model scales, improving the strongest non-OP-Print baseline from 0.517 to 0.594 on Qwen3-1.7B, 0.517 to 0.611 on Qwen34B, and 0.529 to 0.587 on Qwen3-8B.”

• Black-box OP-Print-B attains the best overall AUC (0.575) and accuracy (0.573) on Qwen3-4B

  “In the black-box setting, OP-Print-B achieves the best overall AUC and accuracy across model scales. For example, on Qwen3-4B, it reaches 0.575 AUC and 0.573 accuracy”

What undercuts it the paper’s own results

• Gray-box AUC values for OP-Print remain only modestly above random (e.g., OP-Print-G AUC 0.548)
• Quote: "OP-Print-G AUC 0.548"
• Some black-box baselines (e.g., RC-Rate) achieve higher TPR@5%FPR on specific models, showing inconsistent superiority
• Quote: "RC-Rate attains the highest TPR@5%FPR on Qwen3-4B, its gains are less consistent across scales"

Stated limits 2

• The black-box attack requires repeated restart generation, which introduces additional inference cost (but the adversaries can select the attack configuration themselves).
• We focus on attack effectiveness and leave defense design against correction-imprint leakage to future work.

What’s new here 4

• Exploits a sparse, token-level correction imprint unique to on-policy self-distillation
• Provides a unified attack that works under both gray-box (teacher-student residual) and black-box (restart-based output-basin stability) threat models
• Demonstrates that conventional likelihood-based MIAs are near random on these models
• Shows the correction imprint persists across different distillation objectives (JSD and tinker-style)

Artifacts & methods 5

• OP-Printsystem
• teacher-student residual discrepancyprimitive
• restart-based output-basin stabilityprimitive
• Qwen3-1.7B/4B/8B modelssystem
• OpenThoughts mathematical reasoning subsetdataset

Topics