Archives

The actual bug OPD+ finds — stop-gradient drops a real gradient term for every f-divergence except reverse KL, which is why MiniLLM-style training "just happened" to work — plus the fix and the Table 2 numbers.

The actual f-DISTILL framework — the four concrete divergence variants (KL, RKL, JS, TVD), the word-level decomposition that makes them tractable, and why symmetric losses win on DART/XSum/WMT.

The actual mechanism behind when on-policy distillation works — token overlap ratio rising toward 91%, a "stronger teacher can fail while a weaker one succeeds" result, and a reverse-distillation experiment that undercuts the whole premise.

The reverse-KL policy-gradient derivation behind MiniLLM — per-token rewards, teacher-mixed sampling, length normalization — plus the actual GPT-2/OPT/LLaMA numbers.

The actual GKD objective — a λ-weighted mix of fixed-dataset and student-sampled sequences, a tunable JSD(β) divergence family, and where each choice wins across T5-family experiments.

Jiao et al.'s four-loss layer-wise distillation and two-stage framework, with the actual GLUE table — TinyBERT6 essentially matches BERT-base, TinyBERT4 keeps 96.8% of it at 7.5x smaller.

Sanh et al.'s actual architecture and loss recipe — halved layers initialized every-other-layer from BERT, a triple loss, and the specific GLUE/SQuAD numbers behind the "97% at 40% smaller" headline.

Hinton, Vinyals & Dean's 2015 paper — the mechanics of soft-target distillation, the "omit a digit" MNIST experiment, and how it scales an ensemble of specialists back into one model.

The first entry in the new Notes section — shorter, more informal write-ups than regular posts.

Everything you need to know about ThreadPools and how they help high-scale systems handle massive workloads efficiently.