Agora is a system for collectively pretraining large language models over the internet. Rather than relying on a single datacenter, Agora can access and train across globally distributed, heterogeneous and unreliable hardware, including local devices connected to home internet, through a combination of data parallelism and pipeline parallelism. Contributors form a swarm in which each device hosts one pipeline stage and several devices replicate each stage, making training robust to individual peers joining or leaving at any time.

Agora cuts communication on both axes of parallelism by combining a series of research advances, allowing training over internet links that are several orders of magnitude slower than datacenter interconnects.

We regularly host training runs in which our global community collaboratively pretrains large language models. Each run brings together consumer GPUs at home, instances from major cloud providers and small-scale operators on GPU marketplaces.

Quick Start · Dashboard · System overview · Leaderboard

Previous runs#

Pluralis-13B
September 2026 · 62.4B tokens
A systems test of Agora's upgraded architecture: a Metadata Store for peer coordination, and peer-to-peer microbatch routing in place of passing activations through centralized trainers.

Pluralis-8B
May 2026 · 531.8B tokens
Focused on training efficiency and stability, introducing asynchronous SPARTA for sparse parameter averaging between same-stage peers.

Node0-7.5B
September 2025 · 36.6B tokens
The predecessor to Agora, which built the foundations for collaborative training of large language models over the internet with Subspace Networks.

Research foundations#

Agora is built from the ground up based on published machine learning and systems work:

Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism
arXiv:2506.01260 · NeurIPS 2025
Introduces Subspace Networks, which constrain forward and backward activations to a low-rank subspace at stage boundaries. In Agora, this compresses inter-stage communication by 100×, enabling training over the internet.

AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism
arXiv:2601.22442 · Preprint
Exchanges per-step exact gradient AllReduce collectives in favour of asynchronous sparse parameter averaging. In Agora, same-stage peers communicate sparse weight partitions, overlapped with training.

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
arXiv:2301.11913 · ICML 2023
Demonstrated that pipeline parallel training can survive unreliable and heterogeneous hardware by introducing stochastic wiring. Agora's peer-to-peer routing builds directly on this idea.

Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
arXiv:2607.13332 · Preprint
Presents the full Agora system, combining activation compression, asynchronous sparse averaging and fault-tolerant routing into our permissionless, internet-scale pretraining stack.