Distributed and Decentralised Training: Technical Governance Challenges in a Shifting AI Landscape

Abstract

Low-communication training algorithms are making it feasible to train large models across geographically dispersed clusters or across community-contributed compute with no central coordinator, which undercuts core assumptions of compute governance — detectability, shutdownability, and FLOP thresholds — while leaving export controls and the general relevance of compute intact.

Abstract (as published)

Advances in low-communication training algorithms are enabling a shift from centralised model training to compute setups that are either distributed across multiple clusters or decentralised via community-driven contributions. This paper distinguishes these two scenarios – distributed and decentralised training – which are little understood and often conflated in policy discourse. We discuss how they could impact technical AI governance through an increased risk of compute structuring, capability proliferation, and the erosion of detectability and shutdownability. While these trends foreshadow a possible new paradigm that could challenge key assumptions of compute governance, we emphasise that certain policy levers, like export controls, remain relevant. We also acknowledge potential benefits of decentralised AI, including privacy-preserving training runs that could unlock access to more data, and mitigating harmful power concentration. Our goal is to support more precise policymaking around compute, capability proliferation, and decentralised AI development.

Framing and contributions

• Historically, frontier training compute has sat in a single large data centre with tightly interconnected GPUs — attractive for governance because it is detectable and concentrated among few entities.
• Recent algorithmic work opens two new paradigms: hyperscalers training across geographically dispersed clusters, and community-driven training over the Internet with no central point of control.
• Frontier models have already been trained across multiple data centres (Gemini 1.5, GPT-4.5). Since early 2024 decentralised-training startups have raised roughly $145 million in venture funding (Nous Research $50M, Gensyn AI $43M, Prime Intellect $20M, Flower Labs $20M, Pluralis $7.6M, Berkeley Compute $5M), with a stated goal of training o3-level models by end-2025.
• Stated contributions: propose the distributed/decentralised terminological split; describe the enabling algorithmic progress; detail governance implications; highlight benefits of decentralisation.

The proposed distinction

• Distributed training — training across multiple physically distant compute pools, with a central entity coordinating. Alternatives: “multi-data centre training”, “geographically distributed training”.
• Decentralised training — training using community-provided compute with no central coordinating entity.
• The original technical sense of “distributed” (training across several GPUs) is obsolete, since every frontier run is now distributed in that sense.
• The paper organises regimes on two axes — number of contributing parties and number of compute locations — with Grok 3 (one party, one location), GPT-4.5 (one party, many locations), INTELLECT-1 (many parties, many locations), and an empty hypothetical quadrant for many parties in one location.

Distributed training

• Motivation is power: the largest runs today draw well above 100 megawatts (roughly 80,000 households) and are projected to reach 5 gigawatts or more by 2030, making single-site builds and permitting the binding constraint.
• Data parallelism trains concurrent model copies on data shards and synchronises parameters; traditionally this needs very high-bandwidth interconnects within one cluster.
• Analysis suggests hyperscalers could achieve sufficient long-distance synchronisation by tapping unused US “dark fibre”.
• Low-communication algorithms reduce synchronisation intensity by as much as a factor of 500, potentially enabling training over networks as slow as typical Internet speeds and lowering the importance of some interconnect types.

Decentralised training

• Prime Intellect and Nous Research have pre-trained 10-billion-parameter models on GPUs placed on different continents communicating at typical Internet speeds.
• These models underperform similarly sized counterparts on benchmarks, but the efforts prioritised demonstrating feasibility over performance; the ultimate competitiveness gap is uncertain. (INTELLECT-2, a decentralised RL post-trained reasoning model, appears to underperform its base model, though the cause is unclear.)
• Stated goals: democratise access to AI, encourage open innovation, counterbalance hyperscaler dominance, and accumulate enough community compute to build collectively owned open-source models rivalling industry products.
• Precedent for pooled compute: Folding@Home peaked at around 280,000 GPUs and 4.8 million CPUs and was the first pool to cross 10¹⁸ FLOPS (GPUs supplied ~94% of it) — though AI workloads differ enough that raw FLOPS should not transfer directly.

Peer-to-peer communication

• Most GPU layouts contain algorithmic centralisation (synchronisation through one server) or operational centralisation (an organisational layer assigning work, monitoring, managing resources).
• True P2P networks have no central choke point: nodes initiate and coordinate all tasks, new nodes contribute trustlessly, work is attested by cryptographic proofs and verified by peers, and synchronisation happens peer-to-peer without a master node.
• Node failure does not halt the run — the network reorganises dynamically and accepts replacements that pull instructions from peers.
• The design resembles blockchains such as Ethereum more than traditional data-parallel training. No protocol is yet fully decentralised, but it is a stated goal.

Governance implications

Capability proliferation

• Decentralised AI joins distillation, algorithmic progress, and hardware efficiency as a driver of capability proliferation — especially significant if traditional open-weight providers shift to closed-weight deployment.
• It offers an alternative path for a broader set of actors, potentially including consumer GPUs, to contribute compute; post-training to unlock stronger capabilities is particularly well-suited to low-communication environments.
• Expectation: access to and ownership of increasingly powerful models becomes more commoditised. The authors echo calls to prioritise societal-resilience research.
• The authors take no stance on whether decentralising AI development is good, and call for continued evaluation via the “marginal risk” framework.

Shutdownability

• Preserving the option to shut down training and inference has been proposed as a control on catastrophic risk.
• Open weights make this harder but are not equivalent to large-scale capability diffusion, because running such models at scale still requires specialised, centralised, expensive compute — a concrete point of intervention.
• A true P2P network could allow training or inference with no obvious point of accountability or intervention; the problem is worse for agentic systems. Decentralised inference work is also underway.
• The “no off-switch” problem is recognised within the decentralised AI community, without a clear solution.

Compute structuring

• FLOP-based compute thresholds (e.g. EU AI Act Article 51) assume compute is a rough proxy for capability and hence a targeting mechanism for interventions.
• An actor could evade thresholds by splitting a run into smaller parallel workloads across different cloud providers — “compute structuring” — which benefits directly from reduced synchronisation frequency.
• Detection might rely on monitoring data volume and IP addresses of a compute pool’s communications, but is hard without information sharing between cloud providers; KYC and workload-monitoring work is not progressing fast enough.
• Masking is plausible: INTELLECT-1 synchronised only once every 38 minutes, and the interval can be deliberately prolonged to obfuscate characteristic patterns.

Relevance of compute

• The analysis does not imply compute quantity and quality are obsolete: low-communication training changes how compute is leveraged, not how much is needed.
• Hyperscalers should continue to lead — they can rapidly adopt any decentralised innovation, and other bottlenecks (e.g. ultra-fast GPU memory, often near half the cost of top chips) preserve their moat.
• Compute dominance at company and country level continues to matter, so export controls retain their relevance.

Benefits of decentralisation

• Democratised access may help avoid economic and political disempowerment of individuals and nations; cheap collectively owned models are one route.
• Decentralisation could unlock privately held data — conversations, car telemetry, CCTV — via privacy-preserving training that never transfers data to a central entity, yielding more personalised and locally adapted models.
• Because data need not cross jurisdictions, this could support international cooperation while maintaining data sovereignty.

Recommended next steps

• Build robust methods for detecting parallel compute structuring, combining KYC schemes, workload monitoring, and on-chip mechanisms.
• Establish whether low-communication training can keep pace with hyperscaler training, including whether DiLoCo-like gains extend to tensor or pipeline parallelism; adapt simulation tools accordingly.
• Produce a precise estimate of the compute poolable from consumer GPUs and small-to-medium clusters, to project the scale of feasible decentralised runs.
• Consider extending export controls to intra-node networking equipment and to GPU memory bandwidth and latency.

Appendix A: low-communication data-parallel training

• Standard data parallelism synchronises after every batch, assuming NVLink/InfiniBand-class interconnects. If a batch takes 4 seconds and synchronisation 1 second, 20% of training time is lost to communication; across data centres this becomes prohibitive.
• Worked example: if synchronisation over the public Internet takes 60 seconds and local training 4 seconds per batch, per-batch synchronisation drops compute utilisation to 6.25%.
• DiLoCo (Distributed Low Communication), DeMo (Decoupled Momentum), and successors either synchronise as rarely as every 500 steps or transfer far less data at the same frequency.
• DiLoCo’s core idea is two optimisers: one (an SGD variant) for local updates, another for global parameters after periodic synchronisation, maintaining stability and convergence despite stale gradients.
• Synchronising every 500 steps (~33 minutes) in the worked example raises utilisation from 6.25% to over 97%.
• Additional properties: robustness to asynchronous averaging (contributors dropping in and out); robustness to data shards from different distributions (key to using private data); and improving effectiveness with model scale — follow-up work found DiLoCo-style methods can beat traditional data parallelism on convergence speed and final loss.
• These contrast with normal data parallelism, which is fragile to node failures, needs homogeneous hardware, and depends on continuous high-speed connectivity.

Appendix B: reasoning models and decentralised compute

• Self-supervised pre-training has a 1:1 ratio of forward to backward passes, so every forward pass requires a communication-heavy weight update.
• RL post-training of reasoning models explores many “thinking trajectories” per gradient update — a forward:backward ratio that can reach 1000:1 — which naturally reduces synchronisation frequency.
• This lowers the bar for contributing compute: generating trajectories needs less compute and memory than backward passes, opening the door to consumer hardware. A single Apple M3 Ultra with up to 512 GB of memory can run inference on DeepSeek-R1 (671B parameters).
• Such nodes need only communicate generated trajectories and rewards, not full gradients, so this part of RL training can be spread across thousands of dispersed contributors on ordinary Internet connections.