Pluralis' Multi-party Training Stack
A deep dive into our library built for fault-tolerant multi-party distributed training
Asynchronous Pipeline Parallelism
Nesterov Method for Asynchronous Pipeline Parallel Optimization
Protocol Learning, Protocol Models and the Great Convergence
Two enormous, previously disparate fields converge and a path towards the largest models to ever be trained is opened
Decentralized Training Looms
Collaborative Training of foundation models is closer to actualization than broadly understood. The popular view that low bandwidth node-to-node connections render this infeasible is incorrect