On the Societal Impact of Open Foundation Models

Abstract

Open foundation models — those with widely available weights — derive both their benefits and their risks from five distinctive properties, and the misuse debate should be settled by assessing marginal risk relative to closed models and pre-existing technologies, a standard that most existing studies of misuse do not meet.

Abstract (as published)

Foundation models are powerful technologies: how they are released publicly directly shapes their societal impact. In this position paper, we focus on open foundation models, defined here as those with broadly available model weights (e.g. Llama 2, Stable Diffusion XL). We identify five distinctive properties (e.g. greater customizability, poor monitoring) of open foundation models that lead to both their benefits and risks. Open foundation models present significant benefits, with some caveats, that span innovation, competition, the distribution of decision-making power, and transparency. To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk. Across several misuse vectors (e.g. cyberattacks, bioweapons), we find that current research is insufficient to effectively characterize the marginal risk of open foundation models relative to pre-existing technologies. The framework helps explain why the marginal risk is low in some cases, clarifies disagreements about misuse risks by revealing that past work has focused on different subsets of the framework with different assumptions, and articulates a way forward for more constructive debate. Overall, our work helps support a more grounded assessment of the societal impact of open foundation models by outlining what research is needed to empirically validate their theoretical benefits and risks.

Policy context

• The US Executive Order on Safe, Secure, and Trustworthy AI mandated a Department of Commerce report to the President on the benefits and risks of open foundation models.
• In the EU, open foundation models are partially exempt from obligations under the AI Act.
• Consideration of widely available model weights is a stated priority of the UK AI Safety Institute.
• Named examples of open foundation models: BERT, CLIP, Whisper, BLOOM, Pythia, Llama 2, Falcon, Stable Diffusion, Mistral, OLMo, Aya, Gemma.

Definition

• Open foundation model = foundation model with widely available model weights; any non-open model is called closed. This tracks the Executive Order’s notion of “foundation models with widely available model weights”.
• Under this definition, such a model (i) must provide weights-level access; (ii) need not release code, data, or compute; (iii) must be widely available, though some user restrictions (e.g. age) may apply; (iv) need not be released in stages; (v) may carry use restrictions.
• The dichotomy is acknowledged as reductive but useful, because the central issue is that developers relinquish exclusive control over downstream use once weights are released — restrictions are hard to enforce and easy for malicious actors to ignore, and open release is irreversible.

Five distinctive properties

• Broader access. Weights must be widely available; user restrictions are difficult to enforce or verify (as Meta’s LLaMA 1 release demonstrated), so weights may effectively reach the public. Functional barriers — expertise, compute affordability — remain.
• Greater customizability. Weights (plus activations and gradients) permit quantization, fine-tuning, and pruning. Closed-model adaptation, where offered, is more restrictive, costly, and bounded by the developer’s implementation. Customizability prevents alignment interventions from holding, since users can fine-tune them away — though similar issues arise for fine-tunable closed models.
• Local adaptation and inference ability. Deployment on local hardware removes the need to transfer data to the developer, which matters in confidentiality- or regulation-bound sectors such as healthcare and finance.
• Inability to rescind model access. Developers and distribution channels can stop further access but cannot revoke existing copies; users can redistribute weights peer-to-peer over the developer’s objections.
• Inability to monitor or moderate usage. Inference may run locally, on generic cloud platforms, or on dedicated model hosts; in all cases developers do not observe inference by default. Coordination with dedicated hosts offers partial monitoring, but local inference does not.

Benefits (with caveats)

Distributing who defines acceptable model behaviour

• Closed-model developers exercise unilateral control over what counts as acceptable behaviour — a consequential decision as foundation models increasingly intermediate information access and interpersonal communication.
• Downstream developers using open models can specify alternative behaviour, allowing greater diversity rather than a monolithic developer-determined view.

Increasing innovation

• Aggressive customizability plus local adaptation and inference lets application developers fine-tune on large proprietary datasets without data-protection concerns, and supports advances such as pushing the state of the art in additional languages.
• Caveats: open developers generally lack the user feedback and interaction logs closed developers use to improve models; heavier customisation fragments usage and weakens economies of scale (though model merging may recover some of this). Usability — model capability, quality of inference APIs — shapes innovation beyond openness itself.

Accelerating science

• Broader access enables greater inclusion in research, and weights are essential for several forms of interpretability, security, and safety research.
• Ongoing access supports reproducibility, undermined to date by closed developers regularly retiring models. Safety instrumentation on closed models can render some research impossible — e.g. simulating human behaviour including toxic speech.
• Caveat: weights alone are insufficient. Understanding how biases propagate requires comparing data biases to model biases, hence training-data access. Even benchmark validity depends on data transparency, since contamination can produce overoptimistic results. Projects prioritising asset accessibility exist but are not the norm among open models.

Enabling transparency

• Widely available weights let external researchers, auditors, and journalists scrutinise models more deeply — valuable because developers often underrepresent the marginalised communities most exposed to harms, and because the history of digital technology shows broader scrutiny surfaces concerns developers miss.
• The 2023 Foundation Model Transparency Index found developers of major open models tend to be more transparent than closed counterparts.
• Caveat: weights make some transparency possible without guaranteeing it, and say nothing about upstream resources (data sources, labour practices, energy) or downstream impact (affected markets, adverse events, policy enforcement).

Mitigating monoculture and market concentration

• Foundation models act as infrastructure across sectors, contributing to algorithmic monoculture where many applications depend on one model. Monocultures yield poor societal resilience and systemic risk — the paper cites Meltdown and Spectre, whose impact followed from widespread dependence on Intel and ARM microprocessors — and have been conjectured to produce correlated failures and cultural homogenisation.
• Easier customisation may yield more diverse downstream behaviour, reducing the severity of homogeneous outcomes; broad weight access and customizability enable downstream competition and limit vertical cascading of concentration.
• Scale of barriers: Llama 2 required 3.3 million A100-80GB GPU hours, roughly $6 million at February 2024 cloud rates of $1.8/GPU hour.
• Caveat: open models are unlikely to reduce concentration in the highly concentrated upstream markets for computing and specialised hardware.

Risks and the marginal-risk framework

• Surveyed misuse vectors specifically associated with open foundation models: biosecurity, cybersecurity, voice-cloning scams, spear phishing, disinformation, non-consensual intimate imagery (NCII), and child sexual abuse material. Speculative AI-takeover existential risk is explicitly out of scope.
• Central question: what additional risk does society bear because of open foundation models, relative to pre-existing technologies, closed models, or other reference points?
• The framework is based on threat modelling from computer security and emphasises communicating assumptions and uncertainty, since misuse vectors involve complex supply chains and the offence-defence balance is unstable.

The six steps

1. Threat identification — name the misuse vector and detail how misuse would be executed, including the malicious actors and their resources (individual hackers differ from state-sponsored entities).
2. Existing risk (absent open foundation models) — establish the pre-existing level of risk for the vector, to baseline any new risk.
3. Existing defenses (absent open foundation models) — describe technical defences (e.g. spam filters) and regulatory ones (e.g. laws against distributing CSAM), which inform how effectively new risks will be absorbed.
4. Evidence of marginal risk of open FMs — with threat, existing risk, and existing defences established, evaluate the marginal risk against the status quo. Awareness of existing risk identifies cases where open models merely duplicate it (e.g. an open language model supplying biological information already on Wikipedia); awareness of existing defences identifies concerns already well addressed (e.g. by email and OS-based filters detecting spear-phishing emails). Marginal risk may also be assessed relative to a more restricted release such as a closed model’s API, which requires reasoning about that release’s own defences — ensuring closed releases are not assumed intrinsically safer, but interrogated for the fallibility of their safeguards.
5. Ease of defending against new risks — existing defences are only a baseline; new defences can be built and old ones modified. Characterisations of marginal risk should anticipate how defences evolve in reaction, including the possibility that (open) foundation models contribute to those defences, such as better disinformation detectors or code fuzzers.
6. Uncertainty and assumptions — articulate what underpins the assessment: the trajectory of technological development, the agility of threat actors in adapting, and the potential effectiveness of novel defence strategies. Forecasts of capability improvement or falling inference costs, for instance, change assessments of misuse efficacy and scalability.

Illustrative worked examples

• Cybersecurity / automated vulnerability detection. Existing risk already favours attackers, who need one working vulnerability while defenders must cover all; sophisticated attackers already use automated tools such as fuzzers and Metasploit, and MITRE’s ATLAS matrix includes techniques using closed models. Defences are layered (defence-in-depth), and bug bounties shift the balance toward defence. The authors are unaware of evidence that malicious users have successfully used open models to automate vulnerability detection; dark-web advertisements exist but their reliance on open models is unclear. Monitoring closed models is also imperfect because vulnerability detection is not inherently misuse, creating a hard classification problem.
• Non-consensual intimate imagery. Digitally altered NCII long predates foundation models — Photoshop, then face-swapping tools; one Telegram bot generated over 100,000 sexualised images of women. Open-source community efforts to curb misuse are unlikely to suffice, though platforms and coordinated takedown efforts such as Stop NCII help. Here the marginal risk is clear: open models can be fine-tuned to produce sexualised images of specific people, are far easier for non-experts than Photoshop, and their guardrails are easily circumvented, with several documented real-world incidents.

Findings from applying the framework

• Across the seven studies assessed (spear-phishing scams, cybersecurity, disinformation, biosecurity, voice-cloning scams, NCII, CSAM), the risk analysis is incomplete for six.
• Incomplete assessment does not mean prior analysis is flawed — only that those studies alone are insufficient evidence to demonstrate increased marginal societal risk from open models.
• The two worked instantiations reach opposite conclusions: for cybersecurity via automated vulnerability detection, current marginal risk is low and several defences exist, including using AI for defence; for digitally altered NCII, marginal risk is considerable at present and plausible defences seem hard.
• The framework explains why marginal risk is low in some cases with evidence from earlier waves of digital technology, and retrospectively explains why the debate has been contentious: past studies implicitly analyse different subsets of the framework under different assumptions.
• It also clarifies points of contention by forcing specificity — researchers often club distinct cybersecurity threats together (malware creation, for instance, requires separate analysis from vulnerability detection) — and by pointing to places in the misuse supply chain where risk is better addressed.
• On biosecurity specifically: open language models can generate accurate information about pandemic-causing pathogens, yet such information is publicly available on the Internet; two studies found language model access does not significantly increase access to information needed for biosecurity attacks relative to Internet access, and information access may not be the binding constraint — stronger interventions may lie downstream.
• The authors call for more empirically grounded work on marginal risk, noting that its nature will evolve as capabilities and societal defences evolve.

Limitations of the framework

• It is not a complete framework for release decisions: it offers no mechanism for trading marginal benefits against marginal risk, and does not address the opportunity cost of not releasing openly.
• It evaluates release risk for known risk categories and does not account for unknown unknowns.
• It can identify coordination opportunities — reducing NCII risk would require open developers to coordinate with social media platforms and downstream platforms such as CivitAI — but cannot bring that coordination about.
• The authors expect other approaches to risk analysis will be needed alongside it.

Recommendations and calls to action

AI developers

• Unlike closed models, which developers and users treat with product safety expectations, open models have unclear safety expectations and no established norms for how responsibility is divided between developer and user.
• Developers of open models should therefore be transparent both about the responsible AI practices they implement and about those they recommend or delegate to downstream developers and deployers.
• Downstream developers procuring foundation models should consider which responsible AI measures are already implemented and their measured efficacy, and implement or bargain for the rest — so that responsible AI practices do not fall through the cracks between upstream and downstream providers.

Researchers investigating AI risks

• Significant uncertainty remains across several misuse vectors because evidence is incomplete or unsatisfactory; new research should clarify marginal risks.
• In light of the survey’s findings, more attention should go to articulating the status quo, constructing realistic threat models (or arguing why speculative ones yield generalizable evidence), and considering the full misuse supply chain.

Policymakers

• Government funding agencies should ensure research on open foundation model risks is sufficiently funded while remaining appropriately independent of foundation model developers’ interests.
• Further policy interventions — e.g. hardening downstream attack surfaces — can be considered once uncertainty around specific misuse vectors is reduced, including via improved tracing of downstream usage, and if marginal risks prove significant.
• Policymakers should proactively assess the impact of proposed regulation on open developers: some proposals impose high compliance burdens and should be pursued only with sufficient justification of the adverse effect on the open ecosystem.
• Obligations making developers responsible for downstream use are intrinsically challenging if not impossible for open developers to meet. Recent liability and watermarking proposals, if interpreted strictly to apply to foundation model developers regardless of downstream adaptation or use, would be difficult for open developers to comply with, since they have little ability to monitor, moderate, or prohibit downstream usage.

Competition regulators

• The theoretical benefits of open models — catalysing innovation, distributing power, fostering competition — are largely undocumented in the absence of large-scale economic analyses or market surveillance.
• Many benefits hinge on open models meaningfully expanding consumer choice and reducing costs; if factors such as model quality are the more direct drivers of adoption, the purported benefits may not manifest.
• Regulators should invest in measuring the benefits of foundation models and the impact of openness on them. The UK Competition and Markets Authority has begun such work, which would be bolstered by parallel efforts in other jurisdictions.

Conclusion

• Open foundation models are controversial because of fundamental philosophical disagreements, fragmented conceptual understanding, and poor empirical evidence.
• The paper aims to rectify the conceptual confusion by defining open foundation models, identifying their distinctive properties, and clarifying benefits and risks.
• Some philosophical tensions are unlikely ever to be resolved, particularly where entangled with the incentives of different actors, so the authors encourage future work to address the empirical deficits.
• They are optimistic that open foundation models can contribute to a vibrant AI ecosystem, but say realising this will require significant action from many stakeholders.