How to measure an AI system (and why metrics won't tell you what's failing)

"How well does it work?" sounds like an easy question, and with an AI system it almost never is. A demo can impress with ten well-chosen questions and fail on the eleventh, the one a real user asks on a Tuesday morning.

Measuring an AI system properly is what separates a pilot from something your company can use with confidence. And measuring properly has two parts: knowing how much it fails and knowing why.

Start by defining what "working" means

Before choosing metrics, define success in business terms. For an assistant that answers questions about internal policies, working might mean "answers correctly, cites the document where it got the answer, and says 'I don't know' when the information doesn't exist."

That definition determines what you measure. If you don't write it down, you end up measuring what is easy to measure instead of what matters.

Build an evaluation dataset

An evaluation dataset is a list of cases with the expected answer for each one. It is the most important tool for measuring an AI system, and it must be built with the people who know the work.

A good evaluation dataset:

• Comes from real cases: questions people actually ask, documents that actually arrive.
• Includes hard cases: ambiguous questions, poorly scanned documents, missing information, edge cases.
• Has expert-validated answers: someone from the domain confirms what constitutes a correct answer.
• Grows over time: every error that appears in production becomes a new case.

With that dataset, you can compare versions: if you change the model, the instructions, or the way you search for information, you know whether the system improved or got worse.

What to measure according to the type of system

System type	What to measure
Classification (assigning categories, detecting risk)	How often it gets it right, and separately: how many cases it flags by mistake (false positives) and how many slip through (false negatives)
Extraction (reading invoices, forms)	Percentage of fields extracted correctly, by field type
Answers over documents	Whether the answer is correct, whether it is grounded in sources, whether it cites properly, and whether it recognizes when it doesn't know
Agents executing actions	Whether it completes the task, in how many steps, and how often it requires human intervention

Add operational metrics to that, which matter just as much as quality: response time, cost per query, and how many cases end up being resolved by a human.

For free-text answers, other AI models are often used as evaluators, because reviewing thousands of responses by hand doesn't scale. It works, as long as you manually audit a sample to confirm the evaluator scores like an expert would.

What a metric doesn't tell you

Suppose your system answers correctly in 80% of the cases in your evaluation dataset. That number tells you where you are, but not what to do to improve. The remaining 20% can stem from completely different causes:

• The input data: the document was poorly scanned or the information was outdated.
• The search: the answer existed, but the system didn't find the right document.
• The instructions: the system found the information, but didn't follow the expected format or rules.
• The model: it had everything it needed and still reasoned incorrectly.
• The question: it was ambiguous and there was more than one reasonable answer.

Each cause has a different fix. Switching to a more powerful model won't fix a search that pulls the wrong document; rewriting the instructions won't fix outdated data.

Error analysis: reading failures one by one

The way to uncover the causes is unglamorous and highly effective: take the failed cases and review them one by one.

1. Gather the failures from the evaluation dataset and from production.
2. Read each case completely: the input, what the system searched for, what it found, and what it answered.
3. Note the cause of each failure using categories like the ones above.
4. Tally: how many failures each cause accounts for.
5. Fix the most frequent cause first, measure again, and repeat.

After reviewing a few dozen cases, a pattern almost always emerges. And often most errors concentrate in one or two causes, turning a vague problem ("the system sometimes fails") into a concrete task.

Measuring doesn't end at launch

An AI system that works today can degrade without anyone touching the code: documents change, new types of questions appear, or the provider updates the model. That is why evaluation must be repeated with every change and the system must be monitored in production, with a simple way for users to report poor answers.

In summary

To measure an AI system, first define what working means, build an evaluation dataset with real cases, and choose metrics based on the type of system. Then don't stop at the number: read the errors, classify their causes, and fix the most frequent one. Metrics tell you if you are improving; error analysis tells you what to do to improve.