Sparse attention
Techniques that let transformer models process long sequences without paying the O(n²) cost of full attention — critical for 100K+ token context windows to be practical.
Standard transformer attention scales quadratically with sequence length: doubling the input length quadruples the compute cost. For sequences of a few hundred tokens this is fine; for 200K-token contexts, it's ruinous. Sparse attention is the family of techniques that break the quadratic assumption.
Common approaches include local attention (each token only attends to nearby tokens, with occasional global tokens), strided attention (attend to every Nth token as well as nearby ones), and various learned sparsity patterns. FlashAttention and related work speed up the underlying computation without changing the pattern; the sparse-pattern family reduces total work.
The practical implication: 100K+ context windows in modern frontier models are only affordable because of some combination of these techniques. Without them, a 200K-token request would cost enormously more than it does.
Sparse attention comes with subtle behavior differences. The "lost in the middle" phenomenon — where models pay less attention to information in the middle of long contexts than at the beginning or end — is partly a consequence of sparse patterns emphasizing edge tokens.