NeurIPS Workshop: Collaborative, Open, and Decentralized Training of Foundation Models
CODEC-FM, a NeurIPS 2026 workshop co-organized by Pluralis, on scaling foundation model training across distributed, heterogeneous compute: from inter-datacenter training to internet-scale collaboration.
Training large-scale foundation models today depends on massive, centralized GPU clusters that are inaccessible to most academic institutions, startups, and industries.
This concentration of compute creates high barriers to entry, centralizes AI innovation, and limits broader progress in developing and studying frontier-scale foundation models. Decentralization and resource pooling provide a way to enable large-scale runs without these barriers, by training models across geographically distributed and heterogeneous devices — from coordinated inter-datacenter settings to consumer devices collaborating over the internet.
This workshop brings together researchers and practitioners from the open-model and decentralized-training communities to address the core technical challenges of this paradigm, extending foundation model training beyond the confines of a single datacenter and supporting more scalable, collaborative open model development.
Topics of interest
Open model development and community infrastructure
Communication-efficient distributed training
Asynchronous and local-update optimization
Inter-datacenter training
Internet-scale volunteer collaboration
Federated learning at scale
Heterogeneous devices and interconnects
Fault tolerance, stragglers, and elasticity
Parallelism strategies over low-bandwidth interconnects
Professor of Computer Science at KAUST, where he leads the Optimization and Machine Learning Lab. One of the original developers of federated learning and an expert in randomized, distributed, and communication-efficient optimization algorithms.
Author of Interconnects, a widely read newsletter on the open-model ecosystem, and an expert in RLHF and open language model post-training. Formerly post-training lead at the Allen Institute for AI (Ai2), where he developed RewardBench and led the Tülu 3 post-training recipe.
Distinguished Research Scientist at Together AI and a pioneer of fault-tolerant decentralized training, including SWARM parallelism, Hivemind, and Petals — frameworks for training and serving large models over unreliable, heterogeneous devices.
Founder of Pluralis Research and a pioneer of Protocol Learning — open-source foundation model development via decentralized compute collaborations, including model-parallel training over the internet. Previously an applied scientist at Amazon.
Leonardo Associate Professor of Machine Learning at Carnegie Mellon University and Sloan Research Fellow; an expert in federated learning and communication-efficient distributed optimization, including the CoCoA framework and foundational work on the challenges and methods of federated learning.
Professor of Computer Science at UC Berkeley, where he leads the Sky Computing Lab. Co-founder of Databricks, Anyscale, and Conviva, and a co-creator of Apache Spark, Ray, and Mesos.
Founding Scientist at Pluralis Research and Visiting Researcher at the Australian National University. Previously a senior ML scientist at Amazon and a research fellow at Oxford and ANU; works on optimization, continual learning, and robust decentralized training algorithms.
Founding Scientist at Pluralis Research and Visiting Research Fellow at the University of Adelaide; PhD from ANU. Previously a research scientist at Amazon; works on implicit neural representations, efficient decentralized training, and optimization on manifolds.
Final-year PhD student at Mila / Université de Montréal advised by Irina Rish and Eugene Belilovsky; FRQNT PhD scholar. Works on meta-learned optimizers, continual pre-training, and optimizers for cross-datacenter and over-the-internet pre-training; previously at Meta FAIR and Capital One.
Assistant Professor of AI and Optimization at the University of Zurich. Previously a postdoctoral researcher at Stanford; PhD from EPFL in the Machine Learning and Optimization lab with Martin Jaggi. Works on optimization for decentralized and collaborative learning and privacy.
PhD student at KAUST advised by Peter Richtárik and a Google PhD Fellow in Algorithms and Optimization. Develops scalable optimization methods with strong theoretical guarantees for distributed, federated, and communication-efficient learning; previously at Warwick and Oxford.
Associate Professor at Concordia University and member of Mila, leading a research group on efficient training for large-scale deep learning. Research spans decentralized learning, federated learning, and model parallelism; co-organized workshops at ICML, ICLR, and NeurIPS.
Adjunct professor at Stanford; Program Chair for MLSys 2026. Formerly a lead researcher at Reflection AI, a Senior Staff Research Scientist at Meta, and a technical lead on PaLM (540B) and lead contributor to Gemini pre-training at Google.
Professor of Machine Learning Systems at the University of Cambridge, where he leads the CaMLSys lab; co-founder and Chief Scientific Officer of Flower Labs, creators of the Flower federated learning framework. Royal Academy of Engineering Chair in Decentralized AI.
Papers
Submissions closed on August 29, 2026 and were reviewed double blind. Six accepted papers are given as contributed talks, and every accepted paper is presented in the poster session. There are awards for the best paper, best student paper, and best tiny paper. The workshop is non-archival, and talks will be made public afterwards.