Tag: nlp

All the articles with the tag "nlp".

• Notes on: f-DISTILL

  6 Sep, 2026

  The actual f-DISTILL framework — the four concrete divergence variants (KL, RKL, JS, TVD), the word-level decomposition that makes them tractable, and why symmetric losses win on DART/XSum/WMT.

• Notes on: TinyBERT

  6 Sep, 2026

  Jiao et al.'s four-loss layer-wise distillation and two-stage framework, with the actual GLUE table — TinyBERT6 essentially matches BERT-base, TinyBERT4 keeps 96.8% of it at 7.5x smaller.

• Notes on: DistilBERT

  6 Sep, 2026

  Sanh et al.'s actual architecture and loss recipe — halved layers initialized every-other-layer from BERT, a triple loss, and the specific GLUE/SQuAD numbers behind the "97% at 40% smaller" headline.