Abstract representational geometry supports inference in large language models

Research indicates that human intelligence relies on the hippocampus to construct abstract representations for inferring task structures. Scientists are now investigating if LLMs form similar abstract representations or depend on statistical regularities. Simultaneously, studies on GPT-style Transformer models address plasticity loss, which is the diminished ability of neural networks to learn new information after prior training. Evidence of this plasticity loss persists across models ranging from 5M to 314M parameters in multilingual continual learning settings.