Difficulty calibration yields near-perfect membership inference in diffusion models
The claim
The authors claim that calibrating image difficulty across training or diffusion time enables near-perfect membership inference attacks on diffusion models
Published as: “Difficulty Calibration Across Time for Membership Inference in Diffusion Models”
So what
These techniques enable more reliable privacy and copyright audits of diffusion models, helping stakeholders verify whether specific images were used in training.
The five-minute version
The paper shows that Temporal Self-Calibration (TSC) can double the true-positive rate at 1% false-positive rate compared to prior attacks, reaching 62.31% on CIFAR-10, and that Reference-Model Calibration (RMC) achieves near-perfect auditing of fine-tuned diffusion models without needing captions. These gains are demonstrated across multiple image synthesis and text-to-image datasets and hold for various diffusion model parameterizations. Evidence is strong: the authors report absolute TPR@1%FPR numbers and compare against several established baselines on public benchmarks.
What they found 1
- TSC doubles the low-FPR detection performance on CIFAR-10 compared to the strongest prior baseline.
“On CIFAR-10, TSC achieves a TPR@1%FPR of 62.31%, doubling the performance of the strongest baseline (PIAN at 30.90%).”
What undercuts it the paper’s own results
- The Poké mon fine-tune dataset is an exception where PIA outperforms TSC.
- When using raw loss instead of log-loss, TSC falls below the Naive baseline.
Stated limits 3
- The methods assume access to denoising outputs for candidate images at selected noise levels, which may not be available through black-box APIs.
- RMC requires the checkpoint from which the audited model was fine-tuned; if unavailable, the method cannot be applied.
- RMC measures loss changes between checkpoints but does not identify membership in the reference model’s pretraining corpus.
What’s new here 4
- Calibrates membership inference across training time using a reference checkpoint (RMC).
- Introduces Temporal Self-Calibration that uses a model’s own denoising trajectory to cancel image-difficulty effects.
- Achieves consistent improvements across different diffusion model prediction heads (x, v, ε) and training objectives.
- Demonstrates robustness to query budget and loss domain transformations.
Artifacts & methods 5
- Temporal Self-Calibration (TSC)system
- Reference-Model Calibration (RMC)system
- CIFAR-10, CIFAR-100, STL-10-U, Tiny ImageNet, MS-COCO, Flickr30k, Pokémondataset
- Naive, SecMI, PIA, PIANprimitive
- log-loss transformationevaluation