Evals are the foundation, not the finishing touch
Without measurement, every prompt change is a coin flip.
LLM systems fail *softly*. A regression does not raise an exception — it produces a slightly worse answer that nobody notices until customers complain. The only defence is a evaluation suite: a set of representative inputs with known-good criteria, run automatically on every prompt, model or pipeline change.
Start small. Twenty to fifty carefully chosen cases that cover the real distribution — including the ugly edges — deliver most of the value. A thousand cases sampled from the happy path measure nothing you care about. Grow the suite from production failures: every incident becomes a permanent test case.
Build evals before you optimise. Optimising without measurement is not engineering; it is superstition.
- LLM regressions are silent — only evals catch them.
- 20–50 well-chosen, edge-heavy cases beat thousands of easy ones.
- Every production failure becomes a regression test.