event

NeurIPS Workshop: Collaborative, Open, and Decentralized Training of Foundation Models

CODEC-FM, a NeurIPS 2026 workshop co-organized by Pluralis, on scaling foundation model training across distributed, heterogeneous compute: from inter-datacenter training to internet-scale collaboration.

  Sydney, Australia

About

Training large-scale foundation models today depends on massive, centralized GPU clusters that are inaccessible to most academic institutions, startups, and industries.

This concentration of compute creates high barriers to entry, centralizes AI innovation, and limits broader progress in developing and studying frontier-scale foundation models. Decentralization and resource pooling provide a way to enable large-scale runs without these barriers, by training models across geographically distributed and heterogeneous devices — from coordinated inter-datacenter settings to consumer devices collaborating over the internet.

This workshop brings together researchers and practitioners from the open-model and decentralized-training communities to address the core technical challenges of this paradigm, extending foundation model training beyond the confines of a single datacenter and supporting more scalable, collaborative open model development.

Topics of interest

  • Open model development and community infrastructure
  • Communication-efficient distributed training
  • Asynchronous and local-update optimization
  • Inter-datacenter training
  • Internet-scale volunteer collaboration
  • Federated learning at scale
  • Heterogeneous devices and interconnects
  • Fault tolerance, stragglers, and elasticity
  • Parallelism strategies over low-bandwidth interconnects
  • Security and robustness in trustless settings
  • Incentives, governance, and economic models
  • Benchmarks and evaluation

Speakers6

Professor of Computer Science at KAUST, where he leads the Optimization and Machine Learning Lab. One of the original developers of federated learning and an expert in randomized, distributed, and communication-efficient optimization algorithms.

Nathan Lambert

Interconnects · formerly Ai2

Author of Interconnects, a widely read newsletter on the open-model ecosystem, and an expert in RLHF and open language model post-training. Formerly post-training lead at the Allen Institute for AI (Ai2), where he developed RewardBench and led the Tülu 3 post-training recipe.

Max Ryabinin

Together AI

Distinguished Research Scientist at Together AI and a pioneer of fault-tolerant decentralized training, including SWARM parallelism, Hivemind, and Petals — frameworks for training and serving large models over unreliable, heterogeneous devices.

Alexander Long

Pluralis Research

Founder of Pluralis Research and a pioneer of Protocol Learning — open-source foundation model development via decentralized compute collaborations, including model-parallel training over the internet. Previously an applied scientist at Amazon.

Virginia Smith

Carnegie Mellon University

Leonardo Associate Professor of Machine Learning at Carnegie Mellon University and Sloan Research Fellow; an expert in federated learning and communication-efficient distributed optimization, including the CoCoA framework and foundational work on the challenges and methods of federated learning.

Ion Stoica

UC Berkeley

Professor of Computer Science at UC Berkeley, where he leads the Sky Computing Lab. Co-founder of Databricks, Anyscale, and Conviva, and a co-creator of Apache Spark, Ray, and Mesos.

Schedule

  1. 8:15 – 8:30Opening remarksOrganizers

Morning

  1. 8:30 – 9:00Keynote 1Nathan Lambert
  2. 9:00 – 9:30Keynote 2Invited speaker
  3. 9:30 – 9:45Coffee break
  4. 9:45 – 10:00Contributed talk: SESAME: Communication-Optimal Secure Aggregation for Sign-Based Federated Learning
  5. 10:00 – 10:15Contributed talk: GridFlow: Decentralized Block-Based Flow Matching
  6. 10:15 – 10:30Contributed talk: TBA
  7. 10:30 – 11:00Keynote 3Invited speaker
  8. 11:00 – 12:30Interactive poster sessionAll poster authors
  9. 12:30 – 1:30Lunch

Afternoon

  1. 1:30 – 2:00Keynote 4Invited speaker
  2. 2:00 – 2:15Contributed talk: SeedFlood: Toward Scalable Decentralized LLM Fine-tuning via Zeroth Order Optimization
  3. 2:15 – 2:30Contributed talk: TMax: A simple recipe for terminal agents
  4. 2:30 – 2:45Contributed talk: TBA
  5. 2:45 – 3:00Coffee break
  6. 3:00 – 3:30Keynote 5Invited speaker
  7. 3:30 – 4:45Panel discussion & Q&A — core research, scalability, and economic sustainability; moderated by the organizersInvited speakers
  8. 4:45 – 5:00Awards & closing remarksOrganizers

Schedule is tentative; keynote speaker assignments will be announced closer to the workshop.

Organizers8

Thalaiyasingam Ajanthan

Pluralis Research · ANU

Founding Scientist at Pluralis Research and Visiting Researcher at the Australian National University. Previously a senior ML scientist at Amazon and a research fellow at Oxford and ANU; works on optimization, continual learning, and robust decentralized training algorithms.

Sameera Ramasinghe

Pluralis Research · U Adelaide

Founding Scientist at Pluralis Research and Visiting Research Fellow at the University of Adelaide; PhD from ANU. Previously a research scientist at Amazon; works on implicit neural representations, efficient decentralized training, and optimization on manifolds.

Benjamin Thérien

Mila · UdeM

Final-year PhD student at Mila / Université de Montréal advised by Irina Rish and Eugene Belilovsky; FRQNT PhD scholar. Works on meta-learned optimizers, continual pre-training, and optimizers for cross-datacenter and over-the-internet pre-training; previously at Meta FAIR and Capital One.

Anastasia Koloskova

University of Zurich

Assistant Professor of AI and Optimization at the University of Zurich. Previously a postdoctoral researcher at Stanford; PhD from EPFL in the Machine Learning and Optimization lab with Martin Jaggi. Works on optimization for decentralized and collaborative learning and privacy.

PhD student at KAUST advised by Peter Richtárik and a Google PhD Fellow in Algorithms and Optimization. Develops scalable optimization methods with strong theoretical guarantees for distributed, federated, and communication-efficient learning; previously at Warwick and Oxford.

Eugene Belilovsky

Concordia · Mila

Associate Professor at Concordia University and member of Mila, leading a research group on efficient training for large-scale deep learning. Research spans decentralized learning, federated learning, and model parallelism; co-organized workshops at ICML, ICLR, and NeurIPS.

Adjunct professor at Stanford; Program Chair for MLSys 2026. Formerly a lead researcher at Reflection AI, a Senior Staff Research Scientist at Meta, and a technical lead on PaLM (540B) and lead contributor to Gemini pre-training at Google.

Nicholas Lane

Cambridge · Flower Labs

Professor of Machine Learning Systems at the University of Cambridge, where he leads the CaMLSys lab; co-founder and Chief Scientific Officer of Flower Labs, creators of the Flower federated learning framework. Royal Academy of Engineering Chair in Decentralized AI.

Papers

Submissions closed on August 29, 2026 and were reviewed double blind. Six accepted papers are given as contributed talks, and every accepted paper is presented in the poster session. There are awards for the best paper, best student paper, and best tiny paper. The workshop is non-archival, and talks will be made public afterwards.

Questions go to the organizers at codec-fm-neurips-2026@googlegroups.com.

All events