Memory Attention

Memory Attention (MA) proposes replacing dedicated value projections in language models with token-indexed memory combined with contextual keys. This method allows normalization folding and CPU offloading to reduce GPU parameters. Separately, memory in C is divided into the Stack and the Heap. The Stack manages short-lived variables and functions automatically via the compiler, offering fast sequential allocation but limited space. In contrast, the Heap provides a different area for memory handling during C programming.