A Frozen Eval Set Beats a Moving Scoreboard

If you're tuning a model, an agent, or a prompt stack and the score keeps going up, here is the short answer.

That number only means something if the test stayed still.

A frozen held-out eval set — tasks you don't train on and don't keep editing — is how you tell improvement from a moving goalpost. A scoreboard you rewrite every week is mostly theater.

Why a moving test hides regressions

Picture two weeks of work.

Week one, your system scores 70 on a fixed list of tasks. Week two, you change the model, then you also drop the three tasks it failed, add four new ones it handles well, and tweak the grading rubric so "close enough" counts. The new score is 85.

Did you get better? You have no idea. You changed the ruler.

This happens for friendly reasons. A task felt unfair. A new product surface needed coverage. Someone wanted a metric that "reflected reality." Each edit can be defended in isolation. Together they erase the baseline. You can't compare Tuesday to last Tuesday if last Tuesday's test no longer exists.

Regressions hide in that fog. A model that got worse on the old hard cases can look like a win after those cases leave the set. Nobody planned a cover-up. The process just stopped measuring the same thing.

Golden tasks vs vibe checks

Vibe checks are useful. You try the new build on a few prompts you care about, read the answers, and decide if it feels better. Humans are good at that in small doses.

They're bad as your only scoreboard. Memory is soft. Mood is soft. The prompt you loved on Monday is not sitting in a file with a version number.

Golden tasks are the opposite: a written list of inputs (and, where you can, expected outputs or grading rules) that you run the same way every time. Same tasks. Same scorer. Same definition of pass.

You still need judgment for open-ended work — writing, planning, multi-step agents — because "correct" isn't a single string. Judgment belongs on top of a frozen set, not instead of one. The set tells you what to judge. The human decides whether the answer is good. What you don't do is silently swap the tasks whenever the answers disappoint you.

What "frozen" actually means

Frozen does not mean never grow. It means:

Don't train on it. If the eval tasks leak into fine-tuning, prompt-optimization loops, or few-shot banks, the score stops being a test. It becomes homework the system has seen.

Don't edit it to chase the score. Fixing a typo in a task is fine. Dropping a task because the new model fails it is not. Changing the rubric so today's failure becomes a pass is not.

Version it. When you do change the set — and you will — give the change a name and a date. Keep the old set runnable. "Eval v3 (2026-10-17)" means something. "The current eval" means whatever someone last touched.

The held-out idea is old in machine learning for a reason. A score on data the system optimized against is an easy grade. A score on data it never trained on is closer to the question you actually care about: does it still work on the next real case?

When you may add new tests

You should add tests. Products change. New failure modes show up. A frozen set that never grows becomes a museum.

Add them as additions, not replacements.

Keep the old golden tasks. Report the old score and the new score. If you need a headline number, say which set it came from. Over time you may retire a task that no longer matches the product — but retire it in the open, with a note, not by quietly deleting it the week before a launch review.

A useful pattern: a core frozen set that almost never changes, plus a probe set you update when you're exploring. Decisions about "are we better than last month?" use the core. The probe is for learning, not for the victory slide.

Open-ended tasks still need a spine

Agent work and long-form generation don't grade themselves with exact match. That doesn't excuse a floating bar.

For those tasks, freeze the inputs and the rubric, even if a person assigns the score. Same user request. Same checklist: Did it use the tools it should? Did it invent a fact? Did it take an irreversible action without asking? Did it finish?

Two reviewers will still disagree sometimes. That's fine. What you can't afford is a new rubric every time the model changes, written to flatter the new output. Write the rubric when you're calm, before you see this week's answers, and keep it.

If a task is too vague to grade twice the same way, it isn't an eval task yet. It's a vibe prompt. Keep it for exploration. Don't put it on the scoreboard until you can say what "better" means in words.

A few habits that keep scores honest

1. Separate train from test. Anything you optimize against is contaminated for claiming generalization.
2. Freeze a core set with a version label and a date.
3. Log the eval run the same way you log a deploy: which model, which prompts, which set version, who graded.
4. Add tasks; don't swap them when you want a higher number.
5. Report regressions on the core set even when the probe set looks great.
6. Let a human grade open-ended work against a written rubric, not against "I liked this one more."
7. Revisit the core on purpose — quarterly is a fine cadence — and treat that revisit as a product decision, not a cleanup chore before a demo.

The short version

If the test moves every time the model moves, the score is a story you tell yourself.

Keep a frozen held-out set. Grade the same tasks the same way. Add new tests beside the old ones. Use human judgment where answers aren't unique — but judge against a rubric you wrote before you saw the outputs.

That's how you know you got better, instead of only knowing you got better at this week's scoreboard.

When I'm not arguing with a moving metric, I write The Shadows Series, which starts with Shadows of the Forgotten: https://johndclay.com/books