How QA Changes When the System Can Be Wrong Without Breaking

Not long ago, a test usually had a clear outcome: the system worked, or it broke. AI failures are harder to catch because a wrong answer can still look perfectly plausible. And nothing crashes. There may be no obvious sign that anything went wrong. The test may even pass despite producing the wrong answer.

That is the fundamental shift in modern quality assurance. Now QA also has to catch hallucinations and bias, as well as unexpected behavior under unusual prompts.

This makes robust validation of LLM output essential to catch subtle inaccuracies before they reach production. Tools such as DeepEval, RAGAS, and Braintrust emerged to measure what traditional tests were not built to measure: factuality, consistency, and sensitivity to prompt wording. As AI changes software, it is also changing the tools and skills QA teams need.

TL;DR

AI changed what a software failure looks like. A request can complete successfully, return a polished answer, and still produce the wrong outcome. That means QA has to define what acceptable behavior looks like before release and measure the system against that standard. More of the routine maintenance around brittle tests can now be automated. The harder work still belongs to the team: deciding which failures matter and how much uncertainty is acceptable in production.

What Changes When the Same Input Can Produce a Different Answer

For the last thirty years, software testing has relied on one simple assumption: code is deterministic. You give the system the same input, and you get the same output. Traditional test automation assumes repeatable results, an assumption that breaks down with non-deterministic AI systems.

The same request can yield different valid responses because the model’s behavior is nondeterministic.

What an AI Failure Actually Looks Like

Consider an LLM handling customer support, reviewing a credit application, or working with a contract. If it makes a mistake, nothing may appear to go wrong. No crash. No traceback. The answer sounds convincing and internally consistent. It is just wrong. From the system’s perspective, the request completed successfully.

That changes the cost of a missed defect. The bigger risk is that the system keeps running and producing bad results until someone notices. Sometimes customers catch the problem first. Sometimes journalists do.

Why AI Quality Became a Business Risk

The World Quality Report 2025-2026 identifies data privacy and model reliability as major barriers.

Companies face growing pressure to document AI risks and show how they monitor failures.

How NIST Gives Teams a Structure for AI Risk

NIST’s AI Risk Management Framework gives companies a practical basis for testing and managing AI risk. NIST’s framework can fit into existing compliance programs such as ISO 27001 and SOC 2 while providing teams with a structured approach to AI risk.

What Govern, Map, Measure, and Manage Look Like in Practice

Govern defines who owns AI risk and what the organization considers acceptable use. Govern only works if leadership backs it because a QA team cannot handle this on its own. Typical outputs include AI testing policies and risk escalation procedures. Teams may also need dedicated budgets for validation tools.

The map identifies where the model is used and which data it touches. It also shows who could be affected when the system fails. Incomplete system mapping can hide risks behind a false sense of control.

Measure determines whether the system behaves within acceptable limits and whether that behavior changes over time.

Manage covers what teams do with the risks they have identified and how they monitor them in production.

The goal is to prevent failures and ensure the system responds safely when something goes wrong.

How TEVV Changes the Way AI Systems Are Evaluated

TEVV stands for Testing, Evaluation, Verification, and Validation, and the basic approach actually predates generative AI. In 2026, that approach was formalized as TEVV-Athlon. It moves evaluation earlier in the development process, aligning with shift-left testing.

For a customer support chatbot, teams can group automated checks into evaluation blocks based on the risks they want to measure. Instead of a single pass/fail result, the evaluation produces scores across several risk categories.

The same structure works for statistical models, LLMs, and multimodal agents.

Testing is no longer limited to pass or fail. Teams have to evaluate model behavior throughout the system’s life cycle and decide whether the remaining risk is acceptable.

Red teaming marks one of the biggest departures from traditional QA.

AI also blurs the traditional line between functional QA and security testing by introducing threats such as prompt injection. A user might trick a support bot into revealing system instructions or bypassing business rules, so QA teams also need adversarial testing.

How Teams Use One Model to Evaluate Another

The next part is less intuitive: using one LLM to evaluate another.

This approach is known as LLM-as-a-judge evaluation. The setup is simple: a second model evaluates the first model’s output against a scoring rubric. Once a system produces thousands of outputs per day, reviewing every response by hand becomes impractical.

The problem is that the judge can be wrong too. Worse, its errors can be systematic. Without controls, teams may end up trusting biased evaluations.

Three Biases You Have to Account for in an LLM Judge

Position bias can cause an LLM judge to favor whichever answer appears first. Swap the order of answers A and B, and the score may change. It is similar to an examiner consistently favoring the first paper in a stack regardless of its quality.

Self-preference is another problem because a judge may favor outputs from its own model family. Using the same model to judge competing providers can introduce a built-in bias toward the model family it belongs to.

The third is verbosity bias, in which the judge tends to rate longer, more detailed answers as higher quality, even when they contain factual errors or filler. A judge may prefer a longer response even when a shorter one is more precise. The judge can mistake length for quality.

The practical goal is consistency.

Teams pin the judge to a specific model version so platform updates cannot silently change evaluation behavior. If the provider changes the model behind the scenes, the same outputs may suddenly receive different scores, breaking the CI pipeline. Nothing in your application changed, yet the tests start failing. The result looks like a flaky build, except the instability comes from the judge rather than your code.

Teams use tolerance ranges because the judge itself is inherently uncertain, so an exact threshold can create a false sense of precision.

Teams also use a fixed benchmark, often called a golden test set. A golden test set is a fixed collection of examples that teams rerun to establish a stable baseline. Running the same evaluation on every production request is usually impractical because each judge call adds cost and latency. At millions of requests per day, an additional model call for every response quickly becomes expensive and adds latency.

Teams therefore rarely run an LLM judge on every production request. Instead, teams usually run the judge against a fixed data set and a pinned version of the judge before release. Production monitoring is a separate problem and uses a different set of tools.

How the Main AI Testing Tools Fit Into the Workflow

The LLM evaluation market is crowded, but teams rarely need every tool they encounter.

No single tool covers the whole workflow, so teams usually combine tools for different stages:

• Before release (CI/CD): Teams can use DeepEval or Promptfoo to catch low-quality changes before they ship.
• RAG evaluation: RAGAS specializes in systems where models retrieve context from a knowledge base before answering.
• Production & tracing: Braintrust, LangSmith, or Arize Phoenix handle central storage, observability, and tracing multi-step agents on live traffic.

What the QA Role Looks Like Now

Self-healing addresses a different QA problem: keeping automated tests working as the UI changes.

Anyone who has maintained UI automation knows the problem. Modern component frameworks often generate dynamic class names and change the DOM as the interface evolves. A UI test may locate an element using a CSS selector or XPath, which means a small DOM change can break the test even when the feature itself still works. Tools like Applitools can reduce false failures by focusing on meaningful interface changes.

Brittle tests waste engineering time by breaking whenever the UI changes. In this case, AI helps solve a longstanding QA problem rather than creating a new one.

Where Self-Healing Actually Saves the Team Time

The potential savings are significant. According to Forrester, AI-based self-healing can cut test-suite maintenance costs by 40–45% in large enterprises.

A self-healing system can recognize the same UI element even after its class name or DOM position changes. Without self-healing, an engineer must manually update the broken selector. 

That gives QA teams more time for work that requires human judgment.

How Self-Healing Can Work Without an LLM Call

Self-healing can work in several ways, and not all of them require an LLM.

A common approach is to use Playwright with models such as Llama or Mistral to repair broken selectors and automatically rerun failed tests.

At scale, those extra model calls add high cost and latency across a regression suite with thousands of tests.

Many self-healing systems avoid model calls by matching elements using stable semantics rather than brittle CSS selectors.

In one e-commerce test, the same approach worked across desktop browsers and on iPhone without an LLM call.

What the Healer Agent Can Handle on Its Own

Self-healing still does not remove the need for human review.

In one six-month deployment, the agent handled routine test recovery while testers focused on business logic.

That automation has clear limits: the agent can improve how the test locates elements and handles minor environmental failures, but it cannot determine whether the business logic itself is correct. If the test still fails after several recovery attempts, the system stops and flags the failure for review.

You can stub external integrations, but someone who understands the business process still has to decide what to simulate and how those stubs should behave.

Aspect	2020 QA	2026 QA
Main Focus	Bug Hunting / Finding Defects	Quality Engineering / Risk Coverage & Prevention
Code Type	Deterministic Code	AI-driven / Non-deterministic
Tooling	Manual/Standard Automation	AI-Native (DeepEval, RAGAS, self-healing)
Key Metrics	Defect counts	User experience / Fairness / Hallucination rates
Data & Security	Security handled largely outside QA	QA works more closely with security on AI-specific risks
Skills	Manual testing / Scripting	Data science basics / AI Risk Management / Prompting

What Modern QA Is Really Responsible For

QA now has to evaluate systems that can fail without crashing or throwing an error, so the role goes well beyond traditional software testing. QA teams now evaluate model behavior while managing the automated systems used to test it. Good test coverage now depends on automated evaluation backed by human judgment. Modern QA has to answer a broader question than whether the software works as expected. Teams need to prioritize the failures that matter most and decide whether the remaining risk is acceptable for production.