Testing AI Agents: QA Strategies for Non-Deterministic Systems

You cannot assertEquals your way through AI agent testing. I have watched teams try. They write beautiful test suites full of exact-match assertions, run them against a non-deterministic system, and get a different failure pattern every time.

The fundamental problem: the same input to an AI agent can produce different but equally correct outputs. "Summarize this document" might yield ten valid summaries. "Write a function to sort a list" might produce merge sort, quicksort, or something entirely creative. Traditional testing assumes one right answer. Agent testing cannot.

Rethinking What "Correct" Means

The first shift is moving from equality testing to criteria testing. Instead of "does the output match this expected string," you ask "does the output satisfy these requirements."

For a customer support agent, the criteria might be: response addresses the customer's actual question, tone is professional and empathetic, no factually incorrect statements, suggested actions are actionable, and response length is within acceptable range.

Each criterion becomes a scoring function. Some can be automated. Length checks, format validation, forbidden content detection. Others require an LLM evaluator that reads the output and scores it against the criteria. The best systems use both.

This is fundamentally harder than expect(output).toBe(expected). It is also fundamentally more honest about what you are actually testing.

Building an Evaluation Framework

A proper agent evaluation framework has three layers.

Deterministic checks. These are the easy ones. Does the output parse as valid JSON? Does it include required fields? Is it within length limits? Does it avoid explicitly forbidden content? These should be fast, cheap, and run on every single output.

Heuristic checks. Pattern-based evaluations that catch common failure modes. Does the response contain hallucinated URLs? Does it reference knowledge that should not be in scope? Does it contradict information provided in the prompt? These are more expensive but still automated.

LLM-graded evaluation. Use a separate LLM to evaluate the agent's output against detailed rubrics. "Rate the factual accuracy of this response on a scale of 1-5, citing specific claims and whether they are supported by the provided context." This is the most expensive layer but catches nuanced quality issues that rule-based checks miss.

Run all three layers. The deterministic checks filter out garbage. The heuristic checks catch systematic problems. The LLM grading catches quality issues. Together they give you a reliable quality signal.

The Eval Dataset Is Everything

Your evaluation is only as good as your test cases. And most teams dramatically underinvest here.

A production-quality eval dataset needs several hundred test cases minimum. Each test case includes an input, the relevant context, the expected behavior criteria, and a difficulty rating. Some should be easy cases that any working agent handles. Some should be hard edge cases that stress the system. Some should be adversarial inputs designed to make the agent fail.

Build your eval dataset from real usage. When users report problems, turn those into test cases. When the agent handles something surprisingly well, capture it. When you find a failure mode, create multiple variants of it. Over time, your eval dataset becomes a comprehensive map of your agent's capabilities and limitations.

Version your eval dataset. Track when test cases are added, modified, or removed. When your agent's scores change, you need to know whether the agent changed or the tests changed.

Regression Testing for Non-Deterministic Systems

Traditional regression testing checks that previously working functionality still works after changes. For agents, this is complicated by non-determinism.

The solution is aggregate scoring. Run your full eval suite before and after changes. Individual test case results will vary. That is expected. What should not vary is the aggregate score distribution. If your agent scored 4.2 average on factual accuracy before the change and 3.8 after, you have a regression regardless of whether any individual test case matched exactly.

Set confidence thresholds. A change that drops aggregate quality by more than 0.3 points on any metric should block deployment. A change that improves one metric while dropping another should trigger review. A change that maintains or improves all metrics proceeds.

Run your eval suite multiple times per change. Non-determinism means a single run might be an outlier. Three to five runs give you a reliable signal. Yes, this is expensive. It is cheaper than deploying a regression.

Edge Cases Will Destroy You

Agents handle happy-path inputs well. That is the easy part. What kills you in production are the inputs nobody anticipated.

Ambiguous queries. "Fix the thing from yesterday." What thing? What yesterday? A good agent asks for clarification. A bad agent guesses confidently and acts on the wrong assumption.

Contradictory instructions. "Make it shorter but include all the details." The agent needs to recognize the contradiction and negotiate a resolution rather than producing nonsensical output.

Out-of-scope requests. A coding agent asked to write a poem. A customer support agent asked for medical advice. The agent should recognize its boundaries and decline gracefully rather than attempting tasks it is not designed for.

Adversarial inputs. Prompt injection attempts. Inputs designed to make the agent reveal system instructions. Social engineering attempts to expand the agent's permissions. Your agent needs to be robust against all of these.

Build dedicated test suites for each category. Run them regularly. The adversarial suite in particular should be updated constantly as new attack vectors emerge.

Continuous Evaluation in Production

Testing before deployment is necessary but insufficient. You need to evaluate agent quality continuously in production.

Sample production interactions and run them through your evaluation pipeline. This catches quality issues that your test suite missed because it did not include that scenario. It also catches quality drift, where an agent's performance degrades gradually over time due to changes in user behavior, data distribution, or upstream model updates.

Build feedback loops. When users flag bad responses, those interactions flow automatically into your eval dataset. When users complete tasks successfully, those interactions validate your quality metrics. Over time, your evaluation system learns from real usage in ways that a static test suite never can.

The teams that invest heavily in evaluation infrastructure ship faster and more confidently than teams that rely on manual testing. The initial investment is significant. The compounding returns make it one of the highest-leverage investments in any agent project.