You are probably paying top prices for an AI model that does your job worse than a cheap one would.
Almost nobody checks. Your model got picked because it was in the news, wired into everything, and then left alone while cheaper and better ones came out. We build a test out of your own work. It holds real jobs of yours, with the right answer written down. Every model then runs against it. You choose with your own numbers in front of you.
A test someone else ran tells you nothing about your work.
Public league tables score models on public problems. Your work is not on any of those lists. Nobody has checked how a model handles your customers’ emails, or the one odd case that costs you real money when it goes wrong. The model that tops the charts can do badly on a narrow job like the one you run. A cheap little one can beat it outright on your own jobs. That finding is the part that often pays for your whole test several times over.
So we do the dull thing and measure it. We take real examples straight out of your systems. That can be as few as twenty, or as many as a hundred and fifty. How many you need depends on how much of your work is in scope. We agree with you what a right answer looks like. Then we run every model against that same set. We score three things: how often it gets your answer right, how long you wait for it, and what each job costs you.
How the test works
-
We start with your real work
Your examples come out of your own systems, rather than a sample pack someone downloaded. Where the right answer is a judgment call, we go to the person on your team who makes that call today. We watch how they decide, and we write it down. That way the scoring matches your standard, and not ours.
-
Every model runs against the same set
Your set goes to the big costly models and to the cheaper mid-priced ones. On the wider jobs we also run models you could download and keep on your own machines. They all get the same prompts and the same scoring. So when one of them wins, you can see where it won on your work and by how much.
-
You keep the test, not just the answer
From the Bench tier up, you keep the test itself. That is the folder of your real examples with the right answers marked. When a new model comes out, you can run it against your own work in an afternoon. That beats paying someone for an opinion about it.
-
Easy jobs go to the cheap model
Most of your work splits in two. The simple cases can go to a small fast model that costs you very little. The hard ones get passed up to the costly one. We work out where that line sits in your work, and we agree it with you. We also decide what happens when one of these firms has a bad afternoon and stops answering you.
What it costs
What you pay depends on how many jobs you want tested, and on how many examples the test needs. These are the prices. We do not negotiate them.
Worth saying plainly
If you only run one small job now and then, this will not pay for itself. We will tell you so on the call, rather than after you have paid us. The work earns its price when a model is doing the same thing for you over and over. It also earns it when a wrong answer costs you real money.