Watch: XM on text diffusion: worse likelihood, better low-step sampling

Does Explorative Modeling help text diffusion models? To find out, we compared two versions of a one-hundred-and-fifteen-million parameter masked diffusion language model. We trained a standard baseline for twenty-thousand steps, and a model using Explorative Modeling for ten-thousand steps. Because Explorative Modeling costs twice as much per step, these two runs used the exact same amount of computing time. When we looked at the results, the two measurements pointed in opposite directions. First, the baseline model won on likelihood, showing a better score on the test set. But when it came to actually generating text, Explorative Modeling won. It produced a fifteen-percent improvement in generative quality when sampling with very few steps. As we gave the model more steps to generate text, this advantage shrank, but the quick-sampling benefit is real and translates to faster deployment. Surprisingly, this improvement did not happen the way we expected. The theory behind Explorative Modeling suggests that different options, or latents, will specialize into different topics or modes. Instead, we found they were almost completely redundant. The model did not split into distinct specialties. This suggests the benefit might actually come from training dynamics, acting like a filter that gives the model easier gradients to learn from. While these early results are promising, some questions remain. We still need to rule out whether the difference is simply due to the shorter training run, and we need to see if these benefits hold up as we scale to larger, more capable models.