Open-Sourcing Highly Capable Foundation Models

Abstract

Open-sourcing has historically produced net benefits for software and AI, but for some highly capable foundation models likely to emerge in the near future the risks — irreversible diffusion of dangerous capabilities, removable safeguards, and proliferating unresolved flaws — may outweigh them, so such models should not be open-sourced at least initially, and open-source goals should instead be pursued through structured access and other alternatives.

Abstract (as published)

Recent decisions by leading AI labs to either open-source their models or to restrict access to their models has sparked debate about whether, and how, increasingly capable AI models should be shared. Open-sourcing in AI typically refers to making model architecture and weights freely and publicly accessible for anyone to modify, study, build on, and use. This offers advantages such as enabling external oversight, accelerating progress, and decentralizing control over AI development and use. However, it also presents a growing potential for misuse and unintended consequences. This paper offers an examination of the risks and benefits of open-sourcing highly capable foundation models. While open-sourcing has historically provided substantial net benefits for most software and AI development processes, we argue that for some highly capable foundation models likely to be developed in the near future, open-sourcing may pose sufficiently extreme risks to outweigh the benefits. In such a case, highly capable foundation models should not be open-sourced, at least not initially. Alternative strategies, including non-open-source model sharing options, are explored. The paper concludes with recommendations for developers, standard-setting bodies, and governments for establishing safe and responsible model sharing practices and preserving open-source benefits where safe.

Executive summary

Three factors underpin the paper’s concern:

1. Highly capable models have the potential for extreme risks. The primary worry is diffusion of dangerous capabilities that could cause significant physical harm or disruption to key societal functions — building new biological and chemical weapons, mounting cyberattacks on critical infrastructure and institutions, disseminating targeted misinformation at scale, or enacting coercive population surveillance. Current capabilities arguably fall short of the critical threshold, but nascent dangerous capabilities are emerging and the trend should continue as models become more capable and as deployment and fine-tuning require less expertise and compute.
2. Open-sourcing helps with some risks but could exacerbate extreme ones. For traditional software, open-sourcing favours defence over offence; for increasingly capable foundation models the balance likely skews toward offence because (i) safeguards can be disabled and new dangerous capabilities introduced via fine-tuning; (ii) attacker knowledge of exploits increases greatly; (iii) researching safety vulnerabilities is time- and resource-intensive, with fixes neither straightforward nor easily implemented; (iv) downstream adoption of improvements is hard to ensure, so flaws perpetuate given the general-purpose nature of foundation models.
3. There are alternative, less risky methods for pursuing open-source goals. Structured model access catered to specific research, auditing, and downstream development needs; proactive organisation of secure collaborations; and efforts to enable wider involvement in AI development, evaluation, and governance.

The five resulting recommendations:

1. Developers and governments should recognise that some highly capable models will be too risky to open-source, at least initially — they may become safe later as societal resilience grows and safety mechanisms improve.
2. Open-sourcing decisions should be informed by rigorous risk assessments that consider not only dangerous capabilities and immediate misuse but also how a model might be fine-tuned or amended to facilitate misuse.
3. Developers should consider alternatives that capture distributive, democratic, and societal benefits with less risk — staged release, structured access for researchers and auditors, and democratic oversight of development and governance decisions.
4. Developers, standards bodies, and open-source communities should collaboratively define fine-grained standards for when model components should be released, grounded in an understanding of the risks of releasing different combinations.
5. Governments should exercise oversight of open-source AI and enforce safety measures when stakes are high enough, via liability law, regulation, licensing, fines, or penalties — and should build the capacity to enforce them. Immediate work is needed on the costs, consequences, and legal feasibility of these interventions.

Definitions and the open-source disanalogy

• Highly capable foundation models: those exhibiting high performance across a broad domain of cognitive tasks, often at or above human level. The authors deliberately use “highly capable” rather than “frontier”, since some models of concern will sit behind the moving frontier.
• Extreme risks and harms: operationalised following Shevlane et al. by scale of impact — killing tens of thousands, hundreds of billions of dollars of economic or environmental damage, or comparable disruption to society and political order. The paper is primarily concerned with the near-future emergence of ASL-3-type models under Anthropic’s Responsible Scaling Policy.
• The term “open-source” has become disjointed from open-source licence requirements: Meta calls Llama 2 open-source although its licence bars commercial use by downstream developers with over 700 million monthly users and bars using outputs to train other LLMs — which the Open Source Initiative has criticised as misleading. “Openwashing” describes marketing an appearance of openness while continuing proprietary practices.
• The paper sets licensing aside, since licence breaches are hard to track and enforce for freely downloadable models and will not deter malicious actors; “open-source” here means publicly accessible at no cost.
• The software analogy breaks down because AI systems have separable components — inference code, training code, weights, training data — each shareable independently. Following Solaiman’s gradient of system access, “open-source” spans non-gated downloadable and fully open models.
• Operative definition used: models for which at least model architecture and trained weights are publicly available.
• Component definitions given: model architecture (structure and design, input/output types, how learning happens), model weights (numerical values transforming input to output), inference code (runs the trained model), training code (defines architecture and implements weight-optimisation algorithms).
• Releasing more components makes reproduction and modification easier: architecture plus weights plus inference code suffices to run a pre-trained model, and inference code is easily written or LLM-generated. Optimal hyperparameters and data cleaning/labelling/loading code further lower the bar.
• Weight leaks should be expected: weights are relatively small files (usually under 256 GB) that can be shared easily and untraceably, as LLaMA’s leak a week after gated release demonstrated.
• Box 1 argues further research is needed to define open-source gradients precisely, since benefits and risks depend on which component combinations are released; the Open Source Initiative has launched such an effort.

Risks

Malicious use

Three mechanisms by which open publication facilitates misuse:

• Removing safety filters. New or modified inference code can run the model without content filters. Stable Diffusion’s safety filter can be removed by deleting a single line, because filters are appended post-hoc rather than changing model behaviour.
• Unrestricted fine-tuning. API fine-tuning can be monitored (the owner can inspect the dataset); without monitoring, fine-tuning can reintroduce dangerous capabilities removed pre-release, or make models more dangerous than before safety measures. This is harder than removing post-hoc filters, requiring a curated dataset plus compute and expertise.
• Aiding jailbreaks, including of other models. Weight access enabled researchers to optimise automatic, unlimited production of “adversarial suffixes”; developed on Vicuna-7B and Llama 2, the attack transfers to GPT-4, Bard, and Claude — so open-sourcing one model can expose others’ vulnerabilities.

Varieties of malicious use surveyed:

• Influence operations — automating and scaling political or ideological campaigns; interactive deepfakes of politicians; fabricated verifiable-seeming histories. The CounterCloud experiment autonomously identified political articles, generated and published counter-narratives, and directed traffic via tweets and fake journalist profiles.
• Surveillance and population control — facial and voice recognition, affect recognition, predictive policing, and cheap analysis of unprecedented data volumes; nascent AI surveillance is spreading across political systems from closed autocracies to advanced democracies.
• Scamming and spear phishing — LLMs generate convincing targeted spear-phishing emails at negligible cost; malicious AI tools and jailbreaks proliferate on criminal forums; one report details $35 million stolen from a Japanese firm using AI voice cloning to impersonate an executive.
• Cyber attacks — early demonstrations show current coding abilities apply to malware development and attack design; AI-generated malware may increase the pace and variability of production and can evade detection systems built for less variable human-written programs.
• Biological and chemical weapons development — foundation models may reduce the expertise needed for dual-use research such as gain-of-function virology or synthesis of dangerous compounds. GPT-4 pre-release evaluation showed it could re-engineer known harmful biochemical compounds, and a red-teamer used it to generate the formula for a novel unpatented molecule and order it to their house; red-teaming of Claude 2 identified significant biosecurity potential. Specialised pharmaceutical AI tools can be repurposed to design potent novel toxins. US bipartisan legislation — the Artificial Intelligence and Biosecurity Risk Assessment Act — has been proposed in response.

Ease of malicious use:

• The pool of actors able to run and modify highly capable models is relatively small, but growing: computer science PhD enrolment and AI conference attendance are rising, supplemented by online tutorials.
• Inference needs far less compute than training. Llama-2-70B is estimated to have cost $1.7–3.4 million to train (1,720,320 A100 GPU-hours), while inference runs at 0.2–6 cents per 750-word prompt and about $4 per hour of GPU time. Impactful misuse may need only a few inferences — a single critical vulnerability can disrupt infrastructure.
• Although frontier training costs rise ($10–100 million now, projected $1–10 billion), algorithmic progress targets reducing compute demands for both training and fine-tuning, and compute costs per FLOP fall — so compute access should not be relied on as a lasting limiting factor. Targeted fine-tuning remains far cheaper than training from scratch.

Offence-defence balance:

• Shevlane and Dafoe’s factors: counterfactual possession, absorption and application capacity, resources for solution finding, availability of effective solutions, difficulty of propagating solutions.
• For software, the balance favours defence — vulnerabilities are easy to find, patches are relatively easy and usually complete, and automatic updates propagate them.
• For AI it likely skews toward offence: solutions are hard because internal workings are poorly understood, some risks such as bias are learned from data and may be impossible to remove, mitigation may require social as well as technical change, and AI-specific failure modes resist quick fixes (stochastic outputs, indistinguishability of prompt injections from regular inputs).
• The conclusion is tentative — the balance varies by model, application space, and released components, and defensive capabilities may improve.

Proliferation of unresolved model flaws

• Downstream actors can apply models to tasks never risk-assessed, fine-tune or alter them for additional capabilities, or combine them with other models, tools, and services (e.g. AutoGPT chaining prompts on top of GPT-3.5/GPT-4; ChemCrow integrating 17 computational chemistry tools).
• In all cases the base model’s flaws, vulnerabilities, and unresolved safety issues propagate downstream — biased behaviour, prompt-injection and adversarial-attack vulnerabilities, autonomous self-proliferation abilities.
• Adaptation can also unlock difficult-to-predict new risks, so developer safeguards may not survive downstream modification. Structured API access permits some monitoring; open release offers no such recourse and no take-backs.
• Even in traditional open-source software, flaws proliferate because downstream developers and users often fail to implement patches, even where licences require it; users are frequently unaware updates exist, or find them incompatible with existing infrastructure.
• Incentives differ from software: safety patches for foundation models typically reduce functionality, so downstream users who want the removed functionality are incentivised not to update and sometimes not to disclose vulnerabilities.
• The authors call for clear legislation holding developers and controllers liable for their systems’ impacts.

Benefits and less risky alternatives

External model evaluation

• The argument: open-sourcing enables independent evaluation by a much wider and more diverse community than any single developer could employ, catching bugs, biases, and safety issues and holding developers accountable.
• Evaluation: open-sourcing is most useful for complex safety challenges — emergent abilities, capability overhang, behaviours elicited months after release — and less so for discrete bugs, which developers can fix cheaply in-house and which community reporting and bug bounties can partly surface. Because some behaviours only emerge with weight modification, evaluators need fine-tuning access.
• Alternatives: staged-release impact testing behind an API (where needing many safety measures between stages is itself evidence open-sourcing would lead to misuse, though it costs market share and delays benefits); privileged access for trusted third-party auditors via gated download or research API; red-teaming, potentially extended to a wider community of red-team professionals; and safety bounties. An early OpenAI ChatGPT bounty trial drew over 1,500 submissions for $20,000 in API prizes with few genuinely new discoveries but useful insight into common attack routes.
• Caveats on alternatives: gated download risks leaks and requires careful vetting; developer-chosen auditors and red-teamers risk favouritism and capture, so standards, mandatory auditing regimes, and public or government-shared audit reports are needed.

Accelerating (beneficial) AI progress

The paper separates three kinds of progress:

• Integration progress — open-sourcing helps most here, letting more people tinker, innovate, and optimise for downstream applications. Alternatives: plugins reviewed by the developer or a third-party auditor before publication; gated access with Know-Your-Customer requirements and technical measures such as fingerprinting each copy of the weights.
• Safety progress — academic safety research is often curtailed by insufficient model access, but the benefit is dampened because external researchers frequently lack the computational infrastructure to run the largest models, and some agendas (e.g. emergent capabilities) require bleeding-edge models that smaller local models cannot substitute for.
• Capability progress — open-sourcing helps least, because frontier work is bottlenecked by compute, high-quality data, and talent concentrated in a few well-resourced labs. Frontier advances will nearly exclusively occur at leading labs, which can absorb open-source innovations. Open release does aid talent development, expanding the pool available to regulators, labs, universities, and auditors. Whether accelerating capability progress is even desirable is contested; it counts as a benefit only if it drives beneficial progress disproportionately to risk.
• Alternatives for safety and capability progress: privileged access for identified research groups via structured research APIs (combined with NDAs); actively sought collaborations with trusted parties plus gated download, as OpenAI did during GPT-2’s staged release; and a multistakeholder governing body to mediate research access, guarding against favoritism, in-group evaluation, and “independence by permissions” where developers’ access decisions determine research directions.
• Box 2 strategies for safety progress alongside model sharing: large rewards programmes on the scale of millions or billions for major safety breakthroughs such as interpretability; agreements among frontier labs to commit a percentage of profits or research hours to safety; and international institutions or collaborations akin to CERN or ITER — coupled with membership restrictions, information security, and involvement of lab safety researchers to avoid diffusing dual-use technology or disincentivising in-house safety work.

Distributing control over AI

• The argument: open-sourcing distributes influence away from major labs, which are unelected entities serving their own and shareholder interests, and promotes representation of more diverse interests — the “democratising AI” case made by, among others, Stability AI’s Emad Mostaque.
• Evaluation: the Linux precedent should not be expected to transfer cleanly. Of the four aspects of AI democratization, open-sourcing most directly promotes democratization of development, but with three limits:
  • participating at the cutting edge requires financial, compute, talent, and data resources few actors outside major labs and governments possess — “even maximalist varieties of ‘open’ AI don’t democratize or extend access to the resources needed to build AI from scratch”;
  • open-sourcing can entrench leading developers’ control, since downstream innovations built on their architecture feed back into their products and open-source communities become familiar hiring pools;
  • open-source communities are relatively homogeneous economically, culturally, by gender, and geographically, so distributing power to them is not the same as distributing power to communities generally — democratizing development also requires education, upskilling, and technical infrastructure such as high-bandwidth access and cloud compute.
• Democratization of use benefits from tailoring models to diverse needs, but also requires intuitive interfaces, which open-source communities can help build.
• Alternatives: participatory or representative deliberative processes to inform high-impact decisions about development, use, governance, and model access; institutionalised democratic structures within large labs (democratically selected boards, required deliberative processes for particular topics) to dissipate unilateral control; and support for appropriate regulatory intervention with safeguards against regulatory capture.