HuggingFace Daily PapersRandom Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it