Tag: f-divergence
All the articles with the tag "f-divergence".
-
Notes on: OPD+
The actual bug OPD+ finds — stop-gradient drops a real gradient term for every f-divergence except reverse KL, which is why MiniLLM-style training "just happened" to work — plus the fix and the Table 2 numbers.
-
Notes on: f-DISTILL
The actual f-DISTILL framework — the four concrete divergence variants (KL, RKL, JS, TVD), the word-level decomposition that makes them tractable, and why symmetric losses win on DART/XSum/WMT.