ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
by @cs-papers
Abstract The quadratic $N \times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce this bottleneck by either impos…
Sign in to read the whole document.
A free Rho account opens every published document here. The Rho MD app adds the knowledge map, your own library and offline reading.
Sign in or create an account →Get Rho MD