How to test whether one prompt is really better than another
Forty examples cannot tell an 85% prompt from a 75% one, and forty examples is what almost everybody uses.
The short answer
A prompt comparison needs enough examples that the difference you measure is larger than the noise in the measurement. At forty examples the uncertainty on an 85% score is around plus or minus eleven points, which is wider than most of the differences people act on.
The formula
accuracy = correct / total
standard error = sqrt(accuracy x (1 - accuracy) / n)
95% interval = accuracy +/- 1.96 x standard error
a difference is only meaningful when the intervals barely overlap
This is the same arithmetic used for any proportion. Nothing about language models makes it not apply, and the small-sample intuition people bring from software testing does not transfer.
Worked example
Two prompts scored on the same forty examples:
Prompt A: 34 / 40 = 85.0%
Prompt B: 30 / 40 = 75.0%
SE for A = sqrt(0.85 x 0.15 / 40) = 0.0565
95% interval for A = 85.0% +/- 11.1 → 73.9% to 96.1%
B's interval overlaps A's almost entirely
to detect a 10 point difference reliably you need roughly 300 each
The ten point gap looks decisive and is well inside the noise. Run the same forty examples again with a non-zero temperature and the ranking can reverse.
Build the test set before you write the prompts
Writing examples after you have a prompt in mind produces a test set shaped by the prompt, and it will pass. This is not dishonesty, it is unavoidable unless the order is fixed.
Include the cases you already know are hard, and keep a handful of examples you never look at while iterating. Those are the only ones that tell you whether you improved the prompt or fitted it to the test.
What to record so a result survives a week
The exact prompt, the model version, the temperature, the date, the test set version, and the raw per-example outcomes rather than just the score. Without the model version a result is meaningless within a month, because the model underneath it changes.
Per-example outcomes matter more than the total. Two prompts scoring 80% can be failing on completely different examples, and knowing which failures moved is the only way to make the next change deliberate.
When a small sample is enough
When the difference is large. A prompt going from 40% to 90% is obvious at n=20 and no arithmetic is needed to see it. Small samples find big problems reliably; they cannot rank close alternatives.
That suggests a practical order: use twenty examples to throw out the clearly broken versions quickly, then spend a larger set on the two or three that survive. Running three hundred examples against six candidates is expensive and mostly wasted.
What this leaves out
- Assumes each example is scored right or wrong. Graded quality scores need different arithmetic and usually more examples, not fewer.
- Assumes examples are independent and representative. Forty variations of the same underlying case is effectively a sample of one.
- Excludes latency and cost, which frequently decide between two prompts of similar accuracy.
Common questions
- How many examples do I need to compare two prompts?
- It depends on the size of the difference you care about. Detecting a ten point difference reliably takes a few hundred per prompt; detecting a fifty point difference takes twenty. Deciding which difference matters comes first.
- Why did my prompt get worse without me changing it?
- Model version, temperature, or a test set that changed. All three are common and only the first is outside your control, which is the argument for recording all three every time.
- Is a human review better than an automated score?
- More accurate and much slower, which usually means a smaller sample. A useful compromise is automated scoring on a large set plus human review on the disagreements, where the interesting cases live anyway.
Spreadsheets that do this
The formulas above, already built and checked — so you fill in your numbers rather than the arithmetic.
Related guides
Last reviewed 22 August 2026