Test a new AI model on your own work, not the benchmarks.

Every few months a new AI model arrives with a page of charts showing it beating the last one. Within a day, someone on your team forwards a clip of it doing something impressive. Neither tells you whether it will draft your grant reports or sort your intake forms any better than the tool you already use.

The charts measure someone else's work.

Published benchmarks are useful for direction. They tell you a model is broadly stronger at reasoning or coding. They are built from test questions chosen by people who have never seen your inbox, your case notes or your board packet.

Viral demos are worse evidence, because they show the best run out of many and say nothing about the other attempts. A model can be excellent at puzzles and still mangle the house style of a county permit letter. The only way to know how it handles your work is to give it your work and look at what comes back.

This kind of review is worth doing when a major new model lands from a provider you already rely on. Small point updates usually don't justify the afternoon.

Build a test set from last month's work.

Pick the two or three uses that matter most to your team. For a professional firm that might be first drafts of engagement letters and summaries of long client emails. For a school district office it might be parent communications and turning meeting notes into action lists.

For each use, pull a handful of real tasks from the past month, somewhere between ten and twenty. Save the exact input you gave the current tool and the output it produced. That pair is your baseline, and it is the piece most teams skip.

Then ask the person who owns that work to write down what good looks like, in four to six plain criteria. A grants manager might list: follows the funder's structure, cites the right program figures, stays under the word limit, sounds like us, needs no factual correction. Those criteria are worth keeping long after this one test, since they also tell new staff what the job expects.

Grade blind, in pairs.

Run the same tasks through the new model. Then put each new output beside the old one, strip any label saying which is which, and ask the work's owner to pick the better one against the criteria. People are much more consistent at choosing between two drafts than at scoring one draft out of ten.

Hiding the labels matters more than it sounds. Most of us lean toward the shiny option, and a few people lean the other way out of loyalty to the tool they know.

If you can't tell which draft came from the new model, you have learned something useful about whether to switch.

If the use involves an agent that takes steps on its own, grade how it worked before you grade what it produced. Did it finish? How many steps did it take, and did it stay inside the folders and systems it was meant to touch? A good final answer reached by wandering through files it had no reason to open is still a problem.

Run it three times and read the worst one.

The same model, given the same task, will not give the same answer twice. Run each task at least three times on both the old and new model, because the spread between runs can be larger than the gap between the models. One lucky draft proves very little.

Decide what counts as a win before anyone looks at results. Write it down: the new model has to be picked clearly more often than the old one, at a cost you can live with. Setting the bar afterward invites everyone to move it toward the answer they already wanted.

Then look at the worst output of each batch, not the average. A model can be better most of the time and still break a spreadsheet format, ignore a required field or invent a reference once in every few runs. For an automation that files records unattended, one bad run can cost more than all the good ones save. Track what each run costs as well, though the cost side deserves its own careful look once you know the quality holds.

Check what it does that you didn't ask for.

A new model can be more capable and also more eager. Test three behaviors on your own tasks. Does it refuse ordinary work the old model handled, such as summarizing a patient complaint or drafting a collections notice? Does it take actions nobody requested, like sending an email when it was asked only to draft one? And does it obey instructions hidden inside the content it reads?

That last test is simple to run and few teams have tried it. Put a line inside a sample document or email that tells the model to do something it shouldn't, such as forwarding the file to an outside address, and see whether it complies. If your agents read incoming mail or uploaded forms, this is the test that matters most.

A one-afternoon test plan.

Before the new model arrives

1. Choose two or three uses that matter, and name an owner for each.
2. Save ten to twenty real inputs and current outputs per use as the baseline.
3. Have each owner write four to six criteria for a good result.
4. Write down the bar a new model must clear, in quality and in cost.

When it arrives

1. Run every task at least three times on the old and new model.
2. Shuffle the pairs, hide the labels, and have each owner pick the better draft.
3. Read the worst output from every batch and note what broke.
4. Run the three safety checks: refusals, unrequested actions, planted instructions.

Deciding

1. If it clears the bar, switch one use at a time and keep the old setup running for a few cycles.
2. If it nearly clears the bar but failed a safety check, fix the permission or instruction, re-run that test and set a date to decide.
3. If it falls short, record why with the date and the results, and keep the test set for the next release.
4. If it can do something your current tool couldn't, give that one task an owner and a few weeks as a small pilot.