27 Oct 2026 LLM evaluation: how to test AI answers that are never the same twice

In short: LLM evaluation is how you test an application with a language model inside. Its answers change every time, so you don’t check exact text. You build a fixed set of real questions, score each answer on clear criteria, run each question several times and agree a threshold. People keep checking the scores.

If your AI assistant passed a test once, you know very little. Ask the same question tomorrow and it may answer differently, and sometimes wrongly. That’s why an application with a large language model inside needs its own way of testing. A large language model (LLM) is the kind of AI behind tools like ChatGPT: it writes answers word by word. LLM evaluation is the method to test those answers. It’s closer to normal testing than most teams think.

Why doesn’t normal pass/fail testing work for an LLM?

Normal pass/fail testing fails for an LLM because the same input doesn’t give the same output. A classic test compares the actual result with the expected result. If they match, the test passes. A language model words its answer differently each time, even when the content is the same. An exact match would fail on harmless changes in wording. Worse, an answer can read well and still be wrong. It can contain a made-up number, an old policy or a promise the company can’t keep. So you test two things instead. First, does each answer meet the criteria you agreed? Second, how much do the answers vary when you ask the same question again? Software with AI inside needs that different approach, because a simple pass/fail isn’t enough. We know how to deal with that, and the rest of this post shows how.

None of this replaces normal testing. The screens, the login and the links to other systems still need regular tests. LLM evaluation covers the part that writes the answers.

How do you build an evaluation set?

An evaluation set is a fixed list of questions with, for each question, what a good answer must contain and what it must never do. It’s the LLM version of a regression set. The best questions come from real use: support tickets, chat logs from a pilot, and the questions business experts hear every week. Add the hard cases on purpose. Think of vague questions, questions with a typo, questions the AI should refuse, and questions it should pass on to a person.

For each question, write short criteria. The table shows the idea for a webshop assistant.

The OWASP Top 10 for LLM Applications helps you fill the “must never” column. It lists risks such as leaking sensitive data and spreading wrong information. Keep the set under version control, just like test code. And every time someone finds a bad answer, add that question to the set. A few dozen real questions are a better start than a thousand generated ones.

How do you score answers that keep changing?

You score answers per criterion, over several runs, against a threshold you agreed in advance. Per criterion means you don’t give one overall grade. You check separately whether the answer is correct, is based on your own sources, is safe, follows company policy and has the right tone. Several runs means you ask each question more than once, for example five times. Then you look at the worst answer, not only the average. A customer who gets the one bad answer doesn’t care that the other four were fine. A threshold means you decide upfront what is good enough. For example: every “must never” criterion passes in every run, without exception.

Scoring by hand doesn’t scale, so many teams use a second model as the judge. This is called LLM-as-a-judge. It’s fast, but it has blind spots. Research by Zheng and others found that judge models can favour longer answers and the answer they see first. So people stay in the loop. A tester checks a sample of the scores and every case where the judge is unsure. AI helps with the volume, and people keep the judgement.

One trade-off to name: evaluation can’t prove an AI is always right. It shows how often it goes wrong, and where. For a low-risk internal tool, a small set may be enough. For a customer-facing assistant, it isn’t.

When do you run the evaluation again?

Run the evaluation again after every change that can affect the answers, and before every release. That includes a new prompt, a new model version from your supplier, changed settings and new source documents. Suppliers update models more often than most teams notice, so plan for it. Treat the set like a regression suite: it runs automatically, and a drop below the threshold blocks the release.

After go-live, testing moves into production. Real users ask questions nobody thought of. So you keep watching the answers with AI monitoring after go-live, and new questions go into your set. NIST’s Generative AI Profile also treats testing and evaluation as ongoing work, not a one-off check. Evaluation is one half of AI in software testing: testing the AI itself, as opposed to testing with AI tools.

What this means for your team

Building the first evaluation set is the easy part. The hard part is what comes after. The set has to grow with every new question type, run after every change and produce results that people outside the team can read. A product owner wants to know if the new prompt is better. A compliance officer wants to know which failures touch the law. In many teams that work sits with one developer, next to their real job.

Our in-house tool VeritasAI is built for that ongoing work, and we use it to deliver AI testing as a service. It tests your AI application with real questions and keeps doing so. An operational dashboard shows which questions the application fails. For each failure it shows the law, regulation or framework it maps to, with the prompt, the answer and a priority score. A management dashboard gives the overview.

Want to start this week? Collect twenty real questions your users ask, and write down for each one what a good answer must contain. That list is the start of your evaluation set.

Frequently asked questions

What is the difference between LLM evaluation and normal software testing?

Normal testing checks for one expected result. LLM evaluation checks whether varying answers meet agreed criteria, over several runs. The rest of the application still gets normal tests.

How many questions does an evaluation set need?

Start with a few dozen real questions that cover your main tasks and risks. Grow the set with every bad answer you find. Quality and variety matter more than size.

Can you trust an LLM as a judge?

Partly. A judge model is useful for scoring many answers fast. It has known biases, so a person should check a sample and every unclear case.

When should you rerun an LLM evaluation?

After every change to the prompt, model, settings or source data, and before each release. After go-live, add monitoring of real answers.