Pluralis' Multi-party Training Stack
A deep dive into our library built for fault-tolerant multi-party distributed training
Asynchronous Pipeline Parallelism
Nesterov Method for Asynchronous Pipeline Parallel Optimization
SWARM Parallel with Asynchronous Updates
We significantly improve training reliability, robustness and speed of asynchronous pipeline-parallel training
Beyond Top-K: Pipeline Parallelism Over Slow Networks
A novel method enabling efficient model-parallel training over low-bandwidth networks with 90% compression
Efficient Asynchronous Low-Bandwidth Training on Heterogenous GPUs
A new asynchronous method that surpasses synchronous methods in low-communication training while supporting heterogenous GPUs
A Third Path: Protocol Learning
Developing the true open-source AI
Protocol Learning, Protocol Models and the Great Convergence
Two enormous, previously disparate fields converge and a path towards the largest models to ever be trained is opened
Decentralized Training Looms
Collaborative Training of foundation models is closer to actualization than broadly understood. The popular view that low bandwidth node-to-node connections render this infeasible is incorrect