Token Utility for On-Policy Distillation via Local Trust-Region Optimization

Recent advancements in policy optimization introduce new distillation methods. One approach derives token importance for on-policy distillation using local trust-region optimization, calculating utility via logit gradients and the categorical Fisher matrix to prioritize positions based on loss decrease. Simultaneously, Function-Space Policy Optimization (FSPO) addresses coarse supervision in RLVR by implementing function-space temporal self-distillation. FSPO utilizes sparse historical next-token distributions as fixed temporal targets, providing denser guidance and stability while avoiding the issues of evolving teachers.