Tag: attention-transfer

All the articles with the tag "attention-transfer".

• Notes on: TinyBERT

  6 Sep, 2026

  Jiao et al.'s four-loss layer-wise distillation and two-stage framework, with the actual GLUE table — TinyBERT6 essentially matches BERT-base, TinyBERT4 keeps 96.8% of it at 7.5x smaller.