Keyboard shortcuts

/ or ⌘/Ctrl K
Find a note
j / k
Next / previous section or linked note
h / l
Collapse or go to parent / expand or enter
e or Alt-click
Read a linked note here
o
Open focused note on its own
g g / G
First / last section or linked note
g h / g a
Home / all notes
g b / g t
Backlinks / table of contents
t
Cycle System, Light, Dark
? / Esc
Show / close this reference

Search: ↑/↓ or Ctrl N/P, Enter to open. Shortcuts pause while typing.

Katharopoulos: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention [7de37c90]

papers, transformers

  • Katharopoulos, Angelos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. “Transformers Are RNNs: Fast Autoregressive Transformers with Linear Attention.” ArXiv:2006.16236 [Cs, Stat], August 31, 2020. arxiv.org/abs/2006.16236.
  • Expressing the self-attention as a linear dot-product of kernel feature maps to make the transformer linear time