Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy

Memory in C++ encompasses various concepts regarding how programs manage data for the CPU, utilizing structures like RAM, the Heap, and the Stack. Simultaneously, research into Vision-Language Models and LLMs identifies memory as a critical unsolved challenge. To address this, ECCBench introduces a new evaluation protocol. This benchmark moves beyond simple accuracy to measure memory through three specific axes: efficiency regarding FLOPs, compression of inputs, and calibration of the system's responses during long-horizon tasks.