← tutorials.pro

How to Evaluate LLM Claims Before You Believe the Benchmark

Evaluation · updated Nov 2026

Every model launch comes with a benchmark table. Most of those numbers measure something adjacent to what you actually need. This tutorial gives you a 2-hour process for turning a vendor claim into a number you can trust — or into a reason to walk away.

Step 1: Rewrite the claim as a testable statement

"State-of-the-art reasoning" is not testable. "Scores 92% on our internal invoice-extraction eval" is — if you can see the eval. "444x cheaper than Claude Opus" is testable if you know the token counts and prices on both sides. Write the claim down in one sentence with numbers in it. If you can't, the claim is marketing.

Step 2: Find the comparison's weakest link

Speed and cost claims almost always compare against a slow, expensive baseline. Ask: what was the baseline's latency under what load? Was the new model's number measured with caching warm? Quantized or full precision? "Tokens per second" without batch size and context length is meaningless — decode speed at batch=1 and 32k context is a different product than batch=64 at 2k.

Step 3: Build a 20-example eval of YOUR task

Don't use their eval set — it was chosen to flatter the model. Take 20 real inputs from your actual workload, write down what a correct answer looks like (even loosely), and run both models on them. Score pass/fail. Twenty examples won't give you statistical confidence, but it will tell you if the model is in the right ballpark or embarrassing itself.

Step 4: Measure the thing that actually costs you

For API models: cost per completed task, not per token. A model that's 30% cheaper per token but needs 3 retries per task is more expensive. For local models: tokens/sec on YOUR hardware at YOUR context length, measured with llama-bench or the server's own timing logs — not the marketing page's GPU.

Step 5: Check the failure modes, not just the average

Run the eval twice: once normally, once with a slightly adversarial input (long context, unusual formatting, a prompt-injection attempt in the data). Models that ace the clean eval and fall apart on the weird input are demo models, not production models.

The 80/20 version

Benchmarks are advertisements. Your eval is the only number that matters.