The risk of KV cache compression

Transformer inference on long sequences remains expensive due to softmax attention reading from large KV cache. While KV cache compression replaces full caches with compact summaries, its design relies heavily on empirical experimentation. Theoretical results indicate that compression can be impossible in worst-case scenarios, though new research characterizes the minimax risk to guide algorithm design. Meanwhile, prompt engineering has evolved into context engineering, focusing on model context, tool usage, and history to ensure stability and speed.