Bringing Agent Evals to WebMCP

For decades, software testing has focused on two perspectives: developers and users. Today, AI agents are becoming a third consumer of our applications. That requires a new layer of testing - one that verifies not the code, but the agent's behaviour.

When we build software, testing comes naturally. We write unit tests to verify individual functions. We write integration tests to ensure components collaborate correctly. We write end-to-end tests to validate complete user journeys. Each layer exists because it verifies the application from a different perspective. But what happens when another consumer appears?

That behavioural layer already has a name in the AI world: Agent Evals. Teams building AI systems use them to check tool selection, arguments, grounded answers, and regressions when prompts or schemas change. WebMCP Evals are Chrome's adaptation of that practice to the tools a webpage exposes to the AI Agents - not a new invention, but the same idea applied where your site meets the agent.

AI agents are becoming application users

With WebMCP, our applications expose capabilities directly to AI agents. These agents don't click buttons. They don't fill out forms. They don't inspect the DOM.

Instead, they discover available tools, reason about which one best matches the user's intent, invoke it with structured arguments and generate an answer based on the returned data. This introduces an entirely new class of failures.

A unit test can verify that searchProducts() returns the correct products. An end-to-end test can verify that a human user successfully finds headphones through the UI.

Neither tells us whether an AI agent will choose searchProducts instead of filterProducts. That's a completely different question.

A new testing layer

As AI agents become another consumer of our applications, we need a new testing layer:

Notice that Agent Evals don't replace any existing layer of tests.

Instead, they cover questions that no existing test can answer.

• Did the model choose the correct tool?
• Were the arguments correct?
• Did the model avoid unnecessary tool calls?
• Is the final answer grounded in tool responses?
• Would changing a tool description introduce a regression?

Those aren't implementation tests. They're behavioural tests.

Everything still works… until the agent changes its mind

Let's come back to the web-mcp-angular demo, which exposes four WebMCP tools on the products surface:

• searchProducts
• filterProducts
• addToCart
• getCartSummary

Imagine a situation where everything works perfectly. All unit, integration, and end-to-end tests pass.

A week later you make what looks like an innocent change: you "improve" the description of one tool. Nothing else changes.

Then a user asks: Find wireless headphones below $200.

Instead of invoking the tool you intended, the model suddenly picks a different one. The answer is incomplete, the user gets worse results, and no existing test fails - nothing in CI tells you that you've just introduced a regression. Your application still works. The AI experience doesn't.

That scenario stopped being theoretical the moment I wired WebMCP Evals into the demo.

What broke in a real eval run

I added three evaluation suites to the repository:

• products - tool selection between searchProducts and filterProducts
• cart - addToCart / getCartSummary
• negative - prompts that must not invoke any WebMCP tool (expectedCall: null)

The negative suite includes prompts such as:

• Who won the World Cup?
• What's Angular?
• Tell me a joke
• Write a poem

With a short, generic tool description: Search the product catalog by name and description. the model treated "World Cup" and "Angular" like search queries and called searchProducts.

Joke and poem prompts usually passed evaluation; lookup-shaped ones failed. Handlers, schemas, and Vitest stayed green. The only red signal came from an eval that expected no tool call and saw searchProducts instead.

The fix was not more unit tests

Chrome's WebMCP guidance is clear: when the agent picks the wrong tool, start with adjusting the tool's description and schema.

So I tightened searchProducts to say what it is and what it is not:

Search this store's product catalog (audio, wearable, home, and office items) by name or description. Use only when the user wants to find products to browse or buy. Do not use for general knowledge, news or other non-catalog questions.

After that change, the negative suite stopped treating every curious question as a product search.

The important production lesson is not "remember to write longer strings".

It is this:

Tool descriptions are part of your public API for AI agents. Changing them is a behavioural change. Without evals, that change is invisible.

In a real product you will rewrite descriptions constantly - for clarity, localization, marketing tone, or model upgrades. Agent Evals are how you keep those rewrites honest.

Tool selection is a first-class production risk

The products suite makes the same point from the opposite direction.

searchProducts matches free text in names and descriptions. filterProducts applies structured constraints (category, maxPrice). On the /products UI, a human can search by keyword or apply filters and still find what they need. An agent that picks the wrong tool silently degrades quality:

• searching for "under $100" as text instead of filtering by price,
• filtering by category when the user asked for a specific product name,
• or worse - calling addToCart when the user only asked to search.

That last case is covered explicitly: the eval for “Search for headphones” expects searchProducts, and fails if the model calls addToCart. Picking the wrong tool is a silent trust failure - and traditional tests will never catch it.

Evals also keep your expectations honest

Not every failure means the model is wrong.

One cart case originally expected:

1. addToCart
2. then getCartSummary

for the prompt: Add one pair of headphones … and then tell me what is in my cart.

It kept failing. Why? Because addToCart in this demo already returns the cart snapshot (items, itemCount, total). A competent agent answers after the first call. Requiring a second tool was an unrealistic expectation.

Evals forced the contract to match real agent behaviour. That matters in production too. If your tools return rich structured payloads, agents will skip redundant follow-up calls. If your evals demand a brittle multi-step ritual the model no longer needs, you will "fix" good behaviour and ship worse tool design to satisfy the suite.

Good Agent Evals protect users from bad agent behaviour and protect teams from testing the wrong thing.

Treat tool contracts like API contracts

Once AI agents are a real traffic source - browser assistants, shopping agents etc. - behavioural regressions become customer-facing regressions.

Production teams already know this story for APIs: contracts drift, clients break, and only contract tests catch it early.

WebMCP tools are the same kind of contract, except the client is probabilistic. So production-ready agent surfaces need:

1. Deterministic tests for handlers, schemas, and lifetimes (we already have those).
2. Agent Evals for selection, arguments, refusal to call tools, and grounded answers.
3. CI that treats description/schema changes as risky, because they change agent behaviour even when TypeScript still compiles.

In the demo this is as simple as:

npm run evals

npm run evals

which runs the official webmcp-evals CLI against committed JSON suites under evals/suites/. The mechanism is deliberately boring. The failure modes it catches are not.

Failures become first-class citizens

One of my favourite aspects of WebMCP Evals is that they make behavioural regressions visible. Someone introduces another tool with a very similar description. Or slightly rewrites the existing descriptions. Or upgrades the underlying model. The application still behaves correctly. Yet the agent suddenly starts choosing the wrong capability - or calling a store search for "Who won the World Cup?". Traditional tests won't notice. Agent Evals will.

That's exactly the kind of regression that becomes increasingly important as AI agents become real users of our applications.

Bringing Agent Evals into CI

Once evaluations exist, there's little reason to run them only on a laptop. Just like unit tests or end-to-end tests, they belong next to your merge gate - with the usual caveats for probabilistic systems (retries, soft budgets, quarantines for flaky prompts).

Every meaningful change to tools can verify that:

• existing prompts still succeed,
• tool selection hasn't drifted,
• arguments remain correct,
• off-topic prompts still avoid tools,
• the overall agent behaviour hasn't regressed.

This transforms AI behaviour into something we can continuously verify.

Final thoughts

For decades we've tested software from the perspective of developers. Later we learned to test it from the perspective of users. Today we're entering another transition. AI agents are becoming consumers of our applications too.

If we want to build truly agent-ready - and production-ready - applications, verifying the correctness of our code is no longer enough. We also need to verify that AI agents continue to understand how to use it.

I believe WebMCP Evals are how that layer becomes practical for agent-ready web apps. Not because they replace unit or end-to-end tests. But because they answer a question those tests never could:

Does the AI agent still know how to use my application?

And after watching a one-line description change decide whether "What's Angular?" becomes a product search, I no longer treat that question as optional.

Sources

• WebMCP Evals - Chrome documentation for evaluating agent behaviour against WebMCP tools
• webmcp-evals CLI - official evaluation runner used in the demo
• web-mcp-angular - source code of demo app and committed eval suites (products, cart, negative)
• Demo app - live Angular WebMCP surface
• WebMCP - overview of the WebMCP proposal