Goal-Oriented Evals for WebMCP: Did the Agent Achieve the Goal?

Imagine asking an AI assistant to find a direct flight from Warsaw to London on 12 April, arriving before noon, for under €200 including a cabin bag.

Suppose the website exposes three WebMCP tools:

You write an eval expecting searchFlights followed by getFareDetails. The agent should find a suitable flight and check its fare conditions, without calling bookFlight.

But in one run, the search result already includes everything needed: a direct flight arriving at 10:30, with a total fare of €190 and a cabin bag explicitly included. The agent presents that option and lets you decide whether to book. It skips getFareDetails because there is nothing left to clarify.

Making getFareDetails optional would fix this particular test. But as the conversation grows, we have more decisions to account for: which fares to compare, when to ask a question, and when the available information is enough. Could we evaluate the outcome without describing every acceptable path?

This is the idea behind goal-oriented simulations, which I recently contributed to WebMCP Evals. The PR has now been merged.

Define what success looks like

In Bringing Agent Evals to WebMCP, I explored why tool selection and arguments deserve their own tests. Those checks remain valuable.

Goal-oriented evaluation gives us another way to write that test. We specify the conditions that must be satisfied when the agent finishes. The evaluation then checks the result against those conditions, without requiring a particular sequence of tool calls.

The agent still receives the original user message asking it to find a flight. Separately, we give the evaluation its success criteria:

The agent presents a direct Warsaw–London flight on 12 April, arriving before noon local time. The total fare is under €200 including a cabin bag. No booking was made.

Both paths from our example can satisfy these criteria, provided the agent obtains all the information needed to support its recommendation.

One message or a conversation

Simple case: one user message. We write the user’s message directly into the test. The agent receives it and uses the website’s tools to complete the task. No additional model is needed to play the user.

Complex case: a conversation. We write a scenario for a model playing the user, describing what they want to achieve and any constraints, such as their budget or baggage requirements. The simulated user answers the agent’s questions as the agent works with the website’s tools. It sees the conversation, but not the tool traffic or success criteria. This lets us test how the agent handles missing details, such as asking which London airports the user is willing to fly to.

Check the result

After the conversation, there are two ways to evaluate the outcome:

• A judge model reviews the conversation and tool results against success criteria written in plain language.
• Deterministic checks verify conditions we define against the final page state (HTML/DOM), such as the arrival time or total fare shown for a selected flight.

A case can use either approach or combine both. We still need to choose checks that cover the full request. A €180 fare with a required €40 baggage charge exceeds our €200 budget, however convincing the agent's answer sounds.

Success can also mean stopping

We can also write a separate case where every direct flight arriving before noon costs more than €200 with a cabin bag. Its success criterion is different: the agent should explain that no matching option was found, supported by the search and fare results. Presenting a cheaper flight with a stopover as a match would fail this test.

Even when a suitable flight exists, the request is only to find it. Calling bookFlight would go beyond what the user asked for. The evaluation needs to capture that boundary too.

Goal-oriented evals make those boundaries explicit. They complement unit tests, smoke tests, and checks of expected tool calls. They also remain probabilistic: conversations vary, extra model calls cost time and money, and a model evaluating the result can be wrong. One passing run is not a reliability guarantee.

Start with one user journey. Define what a successful outcome looks like, then check whether the agent can reach it reliably.

I'll follow up soon with a deep dive into writing and running these simulations. In the meantime, take a look at the Pizza Maker simulation examples and the documentation on GitHub.