RL Post-Training on Macs
Post-training an 8B MoE with GRPO on 14 consumer Macs and one B200, decoupled over R2; held-out pass@1 on agentic search more than doubled, from 29% to 63%
Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo
Reducing blocking communication in DiLoCo by factoring synchronization into a non-blocking parameter mixing and a minimal blocking gradient mixing
From Base to Reasoning Model: A Full Post-Training Pipeline on a Single Node
We replicated Meta's full post-training pipeline on a single compute node with one researcher, matching Llama-3.2-1B-Instruct quality
Pluralis' Multi-party Training Stack
A deep dive into our library built for fault-tolerant multi-party distributed training
Asynchronous Pipeline Parallelism
Nesterov Method for Asynchronous Pipeline Parallel Optimization
SWARM Parallel with Asynchronous Updates
We significantly improve training reliability, robustness and speed of asynchronous pipeline-parallel training
Beyond Top-K: Pipeline Parallelism Over Slow Networks
A novel method enabling efficient model-parallel training over low-bandwidth networks with 90% compression
Efficient Asynchronous Low-Bandwidth Training on Heterogenous GPUs
A new asynchronous method that surpasses synchronous methods in low-communication training while supporting heterogenous GPUs
A Third Path: Protocol Learning
Developing the true open-source AI
Protocol Learning, Protocol Models and the Great Convergence
Two enormous, previously disparate fields converge and a path towards the largest models to ever be trained is opened
Decentralized Training Looms
Collaborative Training of foundation models is closer to actualization than broadly understood. The popular view that low bandwidth node-to-node connections render this infeasible is incorrect
