For open-source pretraining to match the scale of frontier labs, we need to train efficiently using compute scattered all over the world, despite the slow connections between such compute. In this work we investigate techniques on the data-parallel (DP) axis toward this goal.

Our paper builds upon two existing ideas: DiLoCo^[2], which reduces the frequency of communication during training, and gossip^[7], which reduces how many other workers each worker must communicate with. Rather than simply combining them, we show it pays to factorize the communication of the combination into two parts: parameter communication (Mix1), which is non-blocking, and gradient communication (Mix2), which is blocking.

Overview#

A key part of our method is removing DiLoCo's global synchronization between workers, a restriction that gossip naturally forces us to drop anyway. This does not just cut communication requirements; it also reduces the impact of slow or failed workers on the system. That property matters even in the centralized setting, where hardware failures can cause massive downtime. (Recently, Decoupled DiLoCo from Google DeepMind takes another route to removing global synchronization, aimed at centralized but highly distributed systems; we compare in the takeaways.)

As soon as you remove global synchronization, you can no longer communicate gradients alone; you have to communicate parameters. If workers' parameters have already drifted apart, updating them with a shared, synchronized gradient does nothing to pull them back together.

Rather than communicate the latest parameters and block training, Factored Gossip DiLoCo (FGDiLoCo)^[1] factorizes the latest parameter state into an older parameter state and a delta. The older parameter state can be averaged (potentially globally) without blocking, overlapped with computation, while communication of the delta must block (stall computation), but can be kept light by doing very local communication.

What we find notable is that most methods with computation–communication overlap introduce temporal staleness: workers fold gradients computed at older weights into current weights, or merge older parameters into newer ones. Our framework does not. Instead it introduces gradient noise (a quantity that already exists, from data sampling), and we show how to control it.

Technical details#

Synchronized-Data Parallel, used in standard centralized datacenter training, has each worker $m$ (1) compute local gradients $g_{m,r}$ on its own data $b_{m,r}$, (2) participate in an all-reduce to obtain the global gradient $g_r = \frac{1}{M}\sum_{m=1}^{M} g_{m,r}$, and (3) take an optimization step with it:

That communication (global averaging) is very expensive in decentralized training, driving compute utilization¹ toward zero. We combine two approaches that improve it: DiLoCo and gossip.

DiLoCo has each worker take $H$ inner steps locally ($H$ is usually 20-500), then treats all $H$ steps combined as an outer gradient that is globally averaged for an outeroptimization step. Communication (global averaging) now happens only every $H$ steps, which improves compute utilization. Concretely, from the synchronized outer parameters $x_r$:

The inner optimizer is typically AdamW and the outer optimizer is SGD with Nesterov momentum^[8].

Gossip, on the other hand, uses local averaging: each worker averages only with a few others, which we call mixing. Workers are therefore never globally synchronized. This only works with parameter averaging; gossiping just gradients would diverge. There are two ways to place the mixing in the literature: after the optimization step, or before it.

Gossip (After):

Gossip (Before):

The benefit of mixing before the step is that it can be overlapped with the forward–backward pass, at the cost of a noisier gradient, since the gradient is then computed on the fully local result of the previous step $z_{m,r}$ rather than the more-mixed result $x_{m,r}$.

Factored Gossip DiLoCo uses this idea with DiLoCo. We perform outer parameter mixing (Mix1) during the inner local steps (a large amount of computation to overlap with), and we add an optional outer gradient mixing (Mix2) before the outer optimization step (blocking, no overlap):

Most methods that overlap computation and communication to improve utilization do so by introducing temporal staleness: they apply round-$r$ gradients to round-$(r{+}1)$ parameters, or merge round-$r$ parameters into round-$(r{+}1)$ parameters. Our method overlaps (via Mix1) withoutany temporal staleness. Instead, the overlap introduces outer-gradient noise: the round-$(r{+}1)$ outer gradients are computed from the more-local parameters $z_{m,r}$ rather than the more-global $x_{m,r}$. Mix2 is precisely what controls this noise: it makes $z_{m,r}$ more global, at the cost of more blocking communication.

In the paper we analyze this framework and show that the bound on the consensus error (the amount of disagreement between workers) carries a factor $F$that depends on the types of mixing used for Mix1 and Mix2. This kind of term directly affects the convergence rate.

The dependence is through the contraction factors $\rho_1$and $\rho_2$ of Mix1 and Mix2: how strongly the mixing pulls results toward the global average. Here $\rho = 0$ is global averaging, $\rho = 1$ is no averaging, and local averaging sits in between, $0 < \rho < 1$. In our experiments the local mixing is pairwise averaging², which has $\tfrac{1}{2} < \rho_{\text{pair}} < 1$. The bound is dependent on a factor

This says Mix1 is critical ($F$ diverges as $\rho_1 \to 1$) while Mix2 helps but is less critical ($F$ grows only linearly in $\rho_2$). That leads us to consider the following variants:

• DiLoCo (no Mix1, global Mix2): the maximum amount of blocking communication, but also the minimum (best) factor, $F = 1$. Best when communication is cheap.
• GlobalM1 (global Mix1, no Mix2): no blocking communication, but a large factor, $F = 2$. Best when communication is very costly and optimization is stable enough to converge. (For measuring stability, see the JS-distance metric below and in the paper.)
• GlobalM1LocalM2 (global Mix1, local Mix2): a small amount of blocking communication. Best when communication is costly and optimization is not very stable.
• LocalM1M2 (local Mix1 and local Mix2, similar to prior work): a poor factor, $F > 2$, so not a desirable choice.

Results#

Across variants, performance per step is highly similar to, or slightly worse than, DiLoCo, but performance per unit time is much better than DiLoCo in our target low-bandwidth settings.

Concretely, at 1.5B parameters on FineWeb with $M = 8$ workers and $H = 100$ (validation perplexity at 10B tokens; compute utilization at 100/200 Mbps; max JS distance):

The pattern: GlobalM1 buys the most utilization for a small perplexity penalty (and the most drift); adding Mix2 back restores stability, with GlobalM1LocalM2 landing closest to DiLoCo. Translated to wall-clock, training to 1T tokens at 200 Mbps takes roughly 16 weeks for GlobalM1 versus 29 weeks for DiLoCo.

A good way to see which variant wins at which bandwidth is to look at the compute-utilization curves (we show $H = 100$ and $H = 500$inner steps).

Combining the perplexity gap with the utilization curves gives a rule of thumb for this 1.5B model at $H = 100$, $M = 8$:

Finally, an observation we find genuinely useful: the Jensen–Shannon (JS) distance is a much better disparity measure than L2 distance. It aligns with performance and reflects optimization stability^[6] (loss spikes coincide with JS spikes, not L2). We discuss this in depth in the paper.

Notes and takeaways#

Comparison to Decoupled DiLoCo#

As mentioned above, the benefit of not having a globally synchronized state is that communication can be more lightweight and the impact of stragglers and failures is smaller, and there is a clear trend toward this in centralized distributed systems too. The recently proposed Decoupled DiLoCo^[4] from Google DeepMind decouples workers further: they are not only not globally synchronized, they do not even train with the same inner or outer time steps. Instead each worker always trains locally on its own time steps, and a separate global parameter server (the syncer), hosted over the CPUs of all workers and running on its own independent time step, handles pulling parameters from workers for the outer optimization rounds and pushing the results back. This is a great fit for a centralized distributed system, but it does not work for a decentralized one: the syncer is a centralized component, so its decisions would need expensive global consensus and its result would have to be broadcast back to all workers. Still, many of its ideas carry over to decentralized systems, especially aggregating updates from workers with different step counts and batch sizes, i.e. handling worker heterogeneity.

Comparison to NoLoCo#

NoLoCo^[5] also combines gossip and DiLoCo, and its approach is approximately equivalent to our LocalM1M2 variant. We include further comparisons in the paper, and show that on the DP side alone this type of approach is not efficient, though NoLoCo demonstrates it can be, with pipeline randomization.

Comparison to Streaming DiLoCo#

Streaming DiLoCo^[3] is a good baseline for introducing temporal staleness to achieve computation–communication overlap. In the paper we compare against it in a way that isolates just that staleness, setting aside its other components (parameter partitioning and quantization).

Other axes#

We did not investigate compression (quantization, sparsification, or low-rank) or variance-reduction approaches like pipeline randomization. These are complementary and left to future work, and JS distance could help guide where to compress.

Citation#

For citations, please cite the original paper using the BibTeX citation:

@inproceedings{hewakoneputugodage2026factored,
  title={Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo},
  author={Hewa Koneputugodage, Chamin and Ajanthan, Thalaiyasingam and Ramasinghe, Sameera and Mohaghegh Dolatabadi, Hadi and Siriwardhana, Shamane and Avraham, Gil and Shevchenko, Violetta and Pajak, Karol and Snewin, James and Long, Alexander},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2026}
}

References#

1. Hewa Koneputugodage, C., Ajanthan, T., Ramasinghe, S., Mohaghegh Dolatabadi, H., Siriwardhana, S., Avraham, G., Shevchenko, V., Pajak, K., Snewin, J., & Long, A. (2026). Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo. ICML 2026 (PMLR 306). arXiv:2606.22768.
2. Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., & Shen, J. (2023). DiLoCo: Distributed Low-Communication Training of Language Models. arXiv:2311.08105.
3. Douillard, A., Donchev, Y., Rush, K., Kale, S., Charles, Z., Garrett, Z., et al. (2025). Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch. arXiv:2501.18512.
4. Decoupled DiLoCo Team, Google DeepMind & Google Research (2026). Decoupled DiLoCo for Resilient Distributed Pre-training. arXiv:2604.21428.
5. Kolehmainen, A., Blagoev, J., Donaghy, J., Ersoy, O., & Nies, C. (2025). NoLoCo: No-all-reduce low-communication training method for large models. arXiv:2506.10911.
6. Kong, L., Lin, T., Koloskova, A., Jaggi, M., & Stich, S. (2021). Consensus control for decentralized deep learning. ICML (PMLR volume 139).
7. Boyd, S., Ghosh, A., Prabhakar, B., & Shah, D. (2006). Randomized gossip algorithms. IEEE Transactions on Information Theory.
8. Nesterov, Y. (1983). A method for solving the convex programming problem with convergence rate O(1/k²). Soviet Mathematics Doklady, 269:543–547.