Full transparency of LLMs can aid attackers and offers limited trust benefits
The claim
Open weights, data, and pipelines do not increase LLM security or trustworthiness and can raise attack success rates
Published as: “Can Open Be Too Open? On the Security and Trustworthiness of Fully Transparent LLMs”
So what
Openness alone should not be assumed to improve LLM security; developers must weigh the added attack surface against the benefits of customization and independence.
The five-minute version
Open weights, open data, and open pipelines can increase the success rate of attacks on LLMs. The paper argues that because LLM training is non-deterministic and costly, users cannot verify that published weights actually stem from the released data and pipeline, so transparency does not remove the need to trust developers. It also notes that while openness enables customization and independent hardening, current verification and patching methods are limited, making the security benefits of full transparency uncertain.
What they found 2
- Open weights, data, and pipelines can increase attack success rates.
“Open weights in particular, but also open data and pipelines, can increase the success rate of attacks.”
- Users cannot verify that open weights correspond to the published data and pipeline due to limited reproducibility.
“due to the limited reproducibility of LLM pipelines, in practice, users cannot verify the correspondence between open weights and the associated open data and pipeline.”
What undercuts it the paper’s own results
- Transparency allows users to customize the model through finetuning, adding specialized capabilities and security guardrails.
- Open-weight models reduce inference costs and guarantee model availability for users with sufficient compute.
Stated limits 3
- The non-determinism and cost of LLM training make it infeasible for users to verify that the model they are running actually resulted from using the published training pipeline on the published training data.
- lightweight methods like unlearning and safety finetuning are not robust [73], [98]
- zero-knowledge proofs of training are too expensive for the model creator to be practical.
What’s new here 4
- Provides a systematic comparison between open-source software security assumptions and LLM transparency, highlighting key mismatches.
- Identifies reproducibility limitations of LLM training as a fundamental barrier to establishing trust from openness.
- Shows that openness of weights, data, and pipeline can directly raise attack success rates, a risk not emphasized in prior surveys.
- Catalogues existing verification techniques (proof-of-learning, attested training, etc.) and evaluates their practical feasibility for frontier models.
Artifacts & methods 7
- open weightsprimitive
- open dataprimitive
- open pipelineprimitive
- proof-of-learning
- zero-knowledge proofs of training
- verifiable training
- attested training