Listen: XM on text diffusion: a small real gain, and a bigger effect we were not looking for
An evaluation of Explorative Modeling on a text diffusion model: a small consistent gain at matched steps, and a larger likelihood-versus-sampling tradeoff that turned out to belong to the training schedule.
Transcript
We set out to test if a technique called Explorative Modeling, or XM, improves text diffusion models. We trained a masked diffusion language model on a large text corpus. Because XM takes twice as much compute per step, we compared a baseline model trained for twenty thousand steps against an XM model trained for ten thousand steps, matching their total training time. At first, the results were striking but contradictory. The XM model had worse overall likelihood scores, but it was fifteen percent better at generating text when given a very low sampling budget. Curiously, the alternative paths the model could choose from didn't specialize as expected; they remained completely redundant. To make sure the shorter run length wasn't throwing off the results, we ran a control baseline for those same ten thousand steps. This changed the story. We discovered that most of the generation boost actually came from training for fewer steps, not from XM itself. When compared to the equal-step control, XM still wins, but by a much more modest five percent. The biggest surprise is that more training actually seems to hurt low-budget sampling. In the end, the training schedule did most of the heavy lifting, leaving us with a real but much smaller benefit from Explorative Modeling.