Skip to content
Linkdrop
Links

S02E05 · Agent Reliability: MEGA lesson video by Angie Jones

Angie Jones demonstrates how to evaluate a retail agent, trace a failed run, and improve its behavior. She also shows input and output guardrails for blocking payment-card numbers and replacing private staff contact details.

videoAngie Jones · hq.mega.dev · Saved Nov 19, 2026

mega course, agent reliability, agent evaluation, agent observability, agent permissions

Transcript of the lesson video for S02E05 of the MEGA.dev course, by Angie Jones. Agenda: Guardrails · Evals · Observability. The video plays on the lesson page in MEGA Student HQ (needs a login).

0:05 So, we've built out quite a bit of functionality in our retail agent. We've given it tools to take action, skills to guide its work, memory to retain context, and the ability to delegate to sub-agents and collaborate with other agents. And we've seen all of these capabilities work as we've built them. Now we need to know whether we can depend on the agent to follow our rules across different situations. We can do that with evals. So we give an agent a task. That could be something like a customer asking to exchange a product, right? And before we run it, we define what a good outcome looks like, including the rules that the agent needs to follow, and then we run the agent and we collect what it actually said and did, and we judge that result against our expectations.

That whole process is an eval. If you've written software tests, that structure should feel really familiar. With an agent though, we might get different wording or a different sequence of tool calls each time, right? And that's because this is non-deterministic. So, we need to decide which differences are okay and which mean the agent actually got something wrong. And we need a way to run these evaluations and inspect the results. This dashboard connects to our retail agent, and it runs a customer conversation, and it shows us what the agent said, what changed in the store, and whether the result met our expectations. Here we choose the customer situation that we want to test, and the conversation shows the customer's messages and the agent's replies.

The store state shows what the application actually saved, such as an exchange or a change in inventory. And these evaluation checks tell us whether what happened matches what we expected for this case. Now, every time we run this, we start with a fresh store and a fresh conversation each time. Now, in a normal exchange, the customer says that the jacket is too tight and asks for a larger size, and then after receiving the proposal, they confirm they want to proceed. So that's our evaluation case. A case includes the starting situation and the customer's messages. So for this exchange, nothing should be completed until the customer confirms. After that, we should have exactly one exchange for the blue size large.

We should have one request that's marked committed, and then one fewer size large jacket in stock. That's what a successful result looks like, right? And so I've run this conversation through our agent using an open-source framework called OpenEnv by Hugging Face. So it gives us a consistent way to start fresh, send a message, and check where we are in the run. For our retail agent, each action is a message from the customer, and the environment is where the agent handles that request. So we don't send our expected result to the agent or anything like that. How it works is it gets the customer's messages, and then we check what it does with them. So here's the customer's first message and the proposal, and then in turn two, the customer confirms.

And the result shows one completed exchange for size large and one fewer jacket in stock. So we're checking what the application saved as well as what the agent told the customer, right? And these checks show which item was exchanged, whether the agent waited for a confirmation, and how the stock changed. And all five checks passed in this run. Now, because these facts are recorded in the store, code can check them directly. So these checks look at what the agent actually changed in the store, and it compares it with what we expected. So this pass tells us that the agent followed these rules in this particular run. We've got a lot of scenarios here, and each case checks a different part of the agent's behavior.

Let's look at the defect after quote scenario, which has failed. Now here, the customer asks for a larger jacket and then in turn two, mentions that the zipper is broken before confirming. So our store's policy is to send defective products to the quality team. So we expect a quality review request here. But let's look at what the agent said and what it actually changed. So the customer reported the broken zipper, but the store still has an exchange proposal. And then after the customer confirms, we've got a completed exchange, no quality review, and one fewer jacket in stock. So this is why the checks failed. Now, the eval tells us what went wrong, but we still need to find out why.

We need to see the model calls and what went into the tools and what came back. So being able to see that work inside the run, that's observability. So let's follow what happened, and maybe the model left out the broken zipper or, I don't know, maybe the tool handled it incorrectly. Maybe a service returned the wrong information. Who knows? That's why we need to look. So in turn two, I'm going to click this tool, and the input here

5:21 says that, okay, yep, it includes the broken zipper, so the agent did pass that information along. And then if we look at the results, we got a request back. Still waiting for confirmation. So let's compare its request ID with the original proposal. So I'm going back up to turn one, and let's look at the tool call here, and let's look at the result and the request ID. So yep, the IDs match. So the tool reused the proposal from before the customer mentioned the defect. It returned a result without an error, but it didn't follow the store's rules, right? So here's where it goes wrong. If there's already a pending proposal for this conversation, the tool returns it straight away, and it never checks whether the customer's new message changes that proposal.

So when we find an issue in our evals, it's a good opportunity to fix our agent application and make it stronger. So let's update that exchange with confirmation tool so that it checks whether the customer's new message changes the pending proposal. I've updated exchange with confirmation to save the input that produced each proposal. If the input is the same, then it reuses the proposal, and if the input is changed, it declines the old proposal before assessing the request again. So that keeps the customer from approving an outdated quote. Let's look at the results after that change. So we're looking at the current agent now with the fix applied. After running the same broken zipper case again, we can see how it handles it after the change.

So in turn two, let's look at the store after this turn. And now we can see that a quality referral was made and no exchange was made. So that's perfect. We also see that the old request is now marked declined. So let's look at the store after turn three, and we can see when the customer tried to confirm, the stocks stayed the same. There was no exchange created, so that's great. And we also want to make sure that we haven't broken something that was already working. So let's do a quick check just to make sure that the normal exchange still works with our current agent. So I'm going to select normal and then current agent.

And great, this test still passes. So this is perfect. So we've checked what happened in the store, but we also need to look at what the agent told the customer. Even if the store is right, we don't want to confuse the customer with the wrong instructions. So the agent says that it updated the request and it asked the customer to confirm, but we can still see the old proposal and there's no quality review. So the evidence doesn't support what it said, and the confirmation should not be the next step. So we can write down what we're looking for in a rubric as our criteria for judging the response. So here we're asking whether the explanation matches the evidence and whether it gives the customer the right next step.

And then we have our decision model, judge the response using our rubric. So this is the saved baseline response that we looked at earlier, and Jev marked both criteria as fail. And then if we looked at the saved response for the fixed run, then the explanation passes both criteria. So we then turn those judgments into a score based on if the criteria passed. So if both passed, then the score is 100. If one passes, then it would have been 50%, and if neither one passed, it would have been zero. And then under the manual review agreement, before calling Jev, we reviewed four saved responses and we gave each one a pass or a fail for each criteria.

So manual review agreement tells us whether Jev's answers match our answers, and here they do. And then if those answers were to disagree, then we have some evidence here that we could review. So far, we've been looking at completed runs, but sometimes we need to stop or change something while the request is being handled. Guardrails are helpful here. So a guardrail checks for a condition and then it steps in when it finds it. So an input guardrail checks a message before it reaches the model. For example, we use one to block a full payment card number. We don't need to send that sensitive data to the model, you know? And then an output guardrail checks the answer before it reaches the customer.

So our store uses a customer service address for inquiries, right? So if a reply includes like a staff member's private address, then our output guardrail will replace that with the support address. Can't have the model doxing our employees. And in Mastra, we connect these checks through processors. So if you've used hooks before, then this is a similar idea. So processors let us run our checks at specific points in the agent's execution, like before the model run or after it answers. And I've connected these to our agent, so let's look at what they do. So this rule looks for a full payment card number before the message reaches the model. It checks for 13 to 19 digits that pass a calculation called Luhn that checks the pattern of the digits.

And we check the customer's latest message, and if we find a matching number, then we stop the request via a tripwire. So the application sends the notice and the model doesn't need to run. And then for the output, we check addresses against our list of private staff emails. So when one matches, we replace it with the configured customer service email. We then attach the guardrails to our agent definition, making them now part of the agent's default behavior. Let's look at the case that we use to check these. So for each rule, we try something that it should block or change. A normal request that it should allow, and then a request that's easy to mistake for a problem.

So this request contains a full card number, and we're going to scroll down to this inspect processor events and model usage. So we see that this action was blocked. There's no model steps, there's no tool calls here. So the card number was stopped before either one of those could run. And then if we look at one that should pass, that's just the last four digits, we see that that was fine. The input was allowed, and then we can see where it actually did some stuff. And then let's look at a private staff contact. So we see that the customer asked who to email about their jacket, and the support lookup actually returned a staff member's private address, and then the model included that in the draft, but then the response that's delivered to the customer uses our customer service address instead.

Okay, great. So we have some rules in place, but I don't want us to be naive. These rules do have limits. So someone could spell out a card number, right? And they could get past this detector. This contact rule, this only handles private addresses that are on our configured list, right? So our checks, they cover specific cases, and there's always ways around them. As we discover those ways around them, though, we need to add those cases to our tests, and we need to improve our checks. Okay? So we've checked our individual cases. Now I want us to compare some repeated runs to see how consistently the agent behaves. So a single pass isn't going to tell us how often the agent will get this right, right?

It might take different steps next time. So we can run the same case again from a fresh starting point, and then keep the failures as well as the passes. And the pass rate tells us how many runs met all of our requirements out of the total time that we ran this. Now here, we've saved repeated runs of the same broken zipper conversation, and each one started with a fresh door. So we see that all three runs failed before the fix, and then all three passed after the fix. And then we also track latency, which is how long the agent took, along with token usage and tool calls. So after the fix, the average time, the token use, and the tool calls were all lower.

And these measurements help us spot any changes that we may want to investigate. And it's also good to track the cost as well. This lets us know whether we can afford to run this agent at scale. So execution cost is what we spend running the agent, and that includes the LLM cost as well as the decision models, classifications. And then the evaluation cost is what we spend using the decision model to check the response after. Now, I can't talk about building reliable agents without touching on security. So we've done a decent job of thinking about that when we were building, but we also need evals dedicated to ensure that our application is secure. So here's one where the customer claims to be an administrator, and it asks for other customers' private data.

So they're trying to use a chat message to give themselves authority that they don't have. This is a classic prompt injection attempt. Fortunately, it looks like our model refused this request, and it didn't call any tools, but we don't want to leave that behavior up to chance, right? We don't know that the model will refuse it every single time, and what if we change the model or something like that? We don't know that the replacement would still refuse it. So that's why we need to keep an eval for this, so that we can check that the response always refuses the unauthorized request. And then we also have explicit instructions in our system instructions that tell the agent that this input is data, this is not instructions or any customer authorization.

And this is a reminder that the model gets on every single turn to help it avoid being manipulated. So here's another security eval. In this case, Alex claims to be Sam and asks for Sam's order. So the agent actually tried the order lookup, but it didn't get the other customer's details. So now, in this case, the, the model may have given out those details if the tool had returned them, because it did try to look up the order. But remember, we have code that prevents it from accessing another customer's data. The tool returned false because it uses the identity from the application, not the name that's typed into the chat. And this case is a good one that shows why we check tool results as well as the final answer, right?

Because a refusal wouldn't be enough if the tool had already exposed the customer's data. So these evals, they check the protections that we already built, and we can run them again after changing the model, after changing the prompts, or even changing the tools, right? And as we find ways that someone might try to get around these rules, we add cases and guardrails and things for those as well. So throughout this course, we've built a retail agent, but the foundations that you've learned, they apply to the agents that you'll build next, right? I want you to start by mastering the primitives, the agent loop, the context and memory, as well as the tools. And I want you to really understand how your agent decides what to do next.

I need you to understand what information it needs and how it takes actions, and then guide it, right, with things like systems instructions and skills, and give it clear boundaries as well as the knowledge that it needs to do its job. And remember, you don't have to make one agent that's responsible for everything. When presented with a task that it doesn't have the capabilities to do, your agent can collaborate with another specialist agent, or it could delegate to other agents as well. And most importantly, before you put an agent into production, please ensure that it is reliable. So you want to evaluate its behavior, you want to investigate its failures, you want to make sure that you enforce those security boundaries and understand performance and cost.

And keep checking as you make changes, okay? So you now have the foundation for building agents that can do useful work. I want you to take what you've learned and apply it to a problem that you want to solve. Thank you so much for building with me. [mouse click]