STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle

Researchers introduced STOCKTAKE, a 26-week supply-chain replenishment benchmark designed to measure the knowing-doing gap in LLM agents. Current evaluations fail to distinguish whether agents misread the world or fail to act despite correct perception. Built as a factored partially observable Markov decision process with six hidden factor processes, STOCKTAKE allows for a computable fair reference policy. This framework addresses the limitation where existing reference policies either lack entirely or utilize privileged information unavailable to agents.