Fair ASR: Re-Evaluating Black-Box Jailbreaks under Shared Target-Call Budgets

Fair-ASR introduces a new evaluation protocol for black-box jailbreak attacks on LLM safety. Current studies rely on attack success rate (ASR) without considering attack budgets, leading to unfair comparisons. While previous compute-aware evaluations used FLOPs, which are hard to estimate for black-box models, Fair-ASR utilizes shared target-call budgets B. This method provides a directly observable and method-agnostic comparison axis, allowing researchers to track attacker calls separately to ensure a more reliable and fair assessment.