Three things that surprised me while building this: Re-runs were identical - Same model, same prompt, `temperature: 0`: byte-for-byte the same failures every time. That turned prompt tweaking into real A/B testing. **A regex beat the prompt.** I

Evaluating free AI models requires more than a single test run due to nondeterministic systems. A single run is merely a sample, not a measurement, making A/A tests essential to verify consistency before benchmarking. While some users report that identical prompts with temperature: 0 produce byte-for-byte identical failures, allowing for real A/B testing, others warn that inconsistent sampling between free and paid tiers can invalidate comparisons. Proper evaluation must prioritize repeatability over initial accuracy reports.