S02E02 · Building an Agent: MEGA lesson video by Angie Jones
Angie Jones demonstrates how to build an exchange service agent with policy checks enforced in code, inventory tools, and JEFF decision models. She covers validating model responses, routing uncertain or defective cases, and evaluating decision thresholds.
videoAngie Jones · hq.mega.dev · Saved Nov 19, 2026
mega course, decision models, agent evaluation, agent permissions, tool calling
Transcript of the lesson video for S02E02 of the MEGA.dev course, by Angie Jones. Agenda: Framework · Models · Tool Design · Context and Memory · Workflows and Controls. The video plays on the lesson page in MEGA Student HQ (needs a login).
0:05 We're going to enhance our customer service agent by giving it an exchange capability. Let's start with the policy. So our store allows exchanges within 30 calendar days of delivery and excludes final sale items. We also need to ensure that the requested quantity hasn't already been exchanged. We'll enforce these rules using the data that's stored in our database. These guardrails make our exchange service reject requests that violate the policy, even if the model asks to proceed. First, we retrieve the order the customer is asking about and verify that it's actually their order. Our service receives the customer identity from the session, so the model doesn't choose whose order it can access. And remember, get_order uses both the customer ID and the order ID in its database query to return only an order that belongs to the customer.
These checks enforce the policy against the purchase item. Even if the customer persuades the model to exchange a final sale item, the service is going to reject that request. The same checks apply regardless of how the customer phrases the request or which tools the model called earlier. Then the check eligibility function returns whether the request passed those checks, and if it didn't, it provides the reason. So that gives the agent a result that it can explain to the customer, right? Passing these checks doesn't authorize an exchange, it only establishes eligibility. The eligibility rules are fixed, so we enforce those in code. But choosing a replacement involves a conversation. For example, the customer might want a larger size, another color, or maybe some help with exploring the options.
So we need our agent to be able to interpret that request and query our inventory with the relevant filters. We've registered the inventory tool on the agent, which gives the model access to the tool so it can look up replacement options. Here, size 9 has three pairs available, while size 10 is out of stock. Both sizes exist in the catalog, but only size 9 is available for an exchange. The inventory tool gives our agent that information so that it can offer an available replacement. The tool accepts size and color filters, which keeps the model's request close to what the customer describes. It also returns the product IDs from the database, which gives the agent the exact values to use when requesting an exchange.
We don't want it trying to make one up, right? As for the quote for the exchange, we calculate it using the amount paid for the original item, the current replacement price, and then the configured exchange fee. And this is handled in code because, well, frankly, it can be, right? Basically, anything that you can do in code, you should. This is a deterministic calculation that does not change given the same input, so there's no need for the model to handle this. The agent's job is to explain the return quote using the amounts that the service provides, and the quote is just a proposal. The customer still needs to accept those exact details before the service changes anything in the store.
Now, consider the customer's reason for requesting an exchange. "These shoes squeeze my toes" is different from "The sole separated on the first day of wearing them." Both customers might be requesting another size, but the second customer has reported a product defect that our policy says must be sent to a person for review. To choose the right handling path, we need to classify the customer's description as a fit or preference issue, a product defect, or just something that needs clarification. Our LLM could classify the issue, but since we have a fixed set of options here, it's actually a good task to consider a decision model for. A decision model evaluates information against criteria, and it returns a structured answer.
For our exchange agent, we'll use the JEFF decision model to identify the kind of issue that the customer reported, which then gives our code info it can use to choose the next step. We have a function to classify issue, which calls JEFF through System 1, which is the type of decision model that JEFFs belongs to. State contains the customer's message, which is what we want JEFF to evaluate, and choice is for multiple-choice questions. We use it here because we want JEFF to pick one of our issue categories, and we give the question a name, issue, so we can find its answer in the response. JEFF also supports other types of questions beyond multiple choice.
You can use a score question to have JEFF rate something along levels that we define, such as how frustrated a customer sounds. And null answers a yes or no question, such as whether the customer is asking for a refund, and it returns the probability that the answer is yes. Let's see how those answers change with the customer's message. State contains the customer's message, and questions contains the question and the descriptions of the choices. These are the same descriptions we use in our code. So let's start with the customer whose shoes squeeze their toes. We're going to run the request, and we see that JEFF selected fit preference, even though the customer never used the word fit.
It connected squeeze my toes with the description of a fit issue. That lets our agent handle different ways that customers describe the same problem. These percentages show how likely JEFF thinks each option is. Here, the probability of fit preference being correct is 100%, while the others are 0%. The probabilities of all of the options add up to 100%, and they can be split across several options. JEFF selects the option with the highest probability as its answers, and it gives you a confidence indicator. So confidence summarizes how strongly the probabilities favor one option. It's high here because all of the probability is on fit preference. So that describes the model's prediction, but it doesn't guarantee that the answer is correct.
Let's try another one. Now, for this one, JEFF selected a product problem, which is the correct answer because the customer described a specific failure. Based on our store's policy, if an item is damaged, then we send the exchange request to a human for review. Using this answer from JEFF, we can execute a different branch of our code. Let's try a vague one. Here, the customer only says something is wrong with their shoes. JEFF selected unclear, which is an option that we provided under our questions. Giving it that option lets the model return a useful answer when the message doesn't contain enough detail. If we only offered fit and product problem, then we'd be asking it to pick between explanations that the customer hasn't necessarily given us.
So given this, we can have our agent ask, "What's wrong with the shoes?" to gather more information so that the model can make a better decision. Also, notice here, this is the first time that we've seen the answer that wasn't 100% probability or 100% confidence. So you can see how the probabilities are distributed across the options, but they add up to be 100. Okay, let's try one that's a bit ambiguous. Okay, for this one, the customer mentions both tight shoes and a separated sole. So JEFF selected product problem, even though the customer also described the fit issue. If we look at our questions, the fit description excludes a reported defect, and that distinction helps separate the choices when a message mentions both.
So this shows that JEFF actually follows that distinction. Each case checks a different behavior, understanding everyday wording, recognizing a reported problem, asking for missing detail, and handling overlapping descriptions. When you use a decision model for your agents, build your own test set around the distinctions that your agent needs to make. If an answer is wrong, check whether the message has enough information and whether your choices describe the distinction that you need. And remember, a confident answer can still be wrong, so compare the choices with your expected answers before deciding how much confidence is enough that your agent should trust it. The choice question type fits our exchange question because we need a category. But for something like deciding which support requests need attention first, right, we might want a rating instead.
Here, we're asking how frustrated the customer sounds, and we describe a calm request, a frustrated but civil request, and then a very angry request. And these levels give the model something specific to look for. Their positions in this array define the scales, starting at zero. So with three levels, the score runs from zero to two, and this is a rating, not a percentage. Let's set the state. The customer politely asks for help with a larger size. Let's run this request. The score is zero, our calm level, and the customer has a problem with the shoes, but they, they're not expressing frustration, right? This question measures their tone separately from the issue that needs resolving.
Let's try one where the customer is angry. This customer says they've contacted the store three times, they're furious, they're never shopping here again. So this score is a two, which is our highest level that describes strong anger or a threat to stop shopping with the store. Um, and this message contains both. So within our code, we can guide our agent on what to do next based on this. If a customer is this frustrated, then it's probably best to bring in a human specialist who might be able to save the day or something like that. Okay, now let's try a question that needs a yes or no judgment by using the null question type. So in this question, we ask JEFF if the customer is asking for a refund, and then the true and false descriptions explain what counts as a refund request.
Now let's add a customer state. So this customer explicitly asks for a refund. Let's run the request, and JEFF says yes, there is a 99% probability that this customer is asking for a refund. Now, this could be a Boolean yes or no value, but it's really good that it's not, because if our agent needs to choose a branch, our code must decide what probability is enough and what number would be considered uncertain, right? Okay, let's try one more. So this customer is asking to exchange their shoes, no refund. And when we run this one, the probability of yes drops to 2%. I just wanted to give you an overview of the different types of questions that you can ask with the decision model.
For our exchange service, we're going to stick with choice. Um, and let me show you how we'll use Jeff's response to guide the agent. This call stores Jeff's response and result. Let's look at what our agent receives. The response is for the shoes that squeeze the customer's toes. Under answers.issues, choice is fit preference. In the API, confidence is from 0 to 1, where 1 corresponds to 100%. The percentages for the choices are probabilities, so the API uses 0 to 1 here as well. So a 0.6 would be 60%. Now, before using the response, we want to validate that each response has the format and the values that our code can use. Let's look at this validate decision function.
We check that the answer contains one of our expected choices and valid numbers. If it doesn't, we stop so that our agent doesn't act on an invalid answer. Once the answer passes those checks, then our code can decide what to do with it. This issue branch function decides what our agent does with Jeff's answer. So we've set the minimum confidence to 0.8. A lower value sends the request for clarification. So does an unclear answer, even with high confidence. A product problem needs human review, and a fit or preference issue can move forward with the exchange proposal. The 0.8 threshold is arbitrary here. For your agents, you'll actually want to run some evals. I know I say that a lot, but that's the answer.
Um, and the evals will help you establish the best cutoff for your use case. Okay, now that we have the branch that Jeff decided on, prepare exchange flow uses that branch to perform the next action. So let's try this out in a chat session. Okay, Squeeze My Toes is asking for an exchange, and Jeff identified that it's a fit issue, so our code allowed the request to reach the proposal step. And our agent presents the quote and is now waiting for confirmation before completing the exchange. Now let's choose the same order but report a product problem, and let's look at the tool. This time the tool returns a referral to the quality team. Even though the customer requested an exchange, Jeff's judgment led our code down a different path.
We've been running our agent in Studio, which is our developer playground, but your company is likely going to run the agent in a real interface, like maybe on the store's website. And that website could show the exchange details with buttons to confirm or cancel, and then it could display a receipt when the exchange is complete. To use our agent in such an interface, we needed to return the agent's output in a structured format. We defined that format in a schema, and this is an ordinary TypeScript file where we describe the fields that we want in the answer, their types, the values that they're allowed to contain, and we're using Zod to express those rules.
We keep this with our exchange code and export the schema so that the code calling our agent can import it. So in your project, you could keep it beside the agent or in a shared schemas folder if several parts of your code need it. To decide what goes in your schema, start with what the interface needs to display or what it needs to do next. Our website needs to know whether to ask for a confirmation or if to show a receipt. It also needs the customer's message and the amounts to display. That's why we have status and message. We define the allowed statuses so that the website can handle each outcome, and then the message is the text that we show the customer.
For something like a scheduling agent, those fields might be appointment time or a booking ID, right? It just depends on what you're building. So choose fields for the job that your agent does and the interface that will use its answers. The descriptions tell the model what belongs in each field. So here for the exchange ID, we're telling it where to get the exchange ID from and when to return null. So you want to write these descriptions to explain the meaning of your fields, not just their types. Here's the call the website server would make. So we pass the customer's message to agent.generate and our imported schema as structuredoutput.schema, and that asks for an answer in the format that we defined.
The structured answer comes back in an object. The server can return this data to the website. And here's an example of what our agent would return. So a nice structured object that the external application can use, and that status tells the interface that we're waiting for the customer's confirmation. But let's look at how we can make our agent actually wait before processing the exchange. The chat already waits for the customer to send another message, but our agent can call several tools before it ever replies. So we need the exchange process to stop at the proposal and then require the customer's confirmation- ... before it can continue. And for that, we need code that saves the proposal and prevents the exchange from proceeding without confirmation.
For this, we'll use a workflow. A workflow defines the steps of a task and the rules for moving between them. The workflow checks eligibility, it uses JEV to identify the issue, and it prepares a proposal if the request can proceed. For our exchange, we assess the request and then prepare a proposal. Then we wait for the customer. We can save the workflow's progress in a database and then link it to the conversation. And then when the customer replies, we load that saved run and continue from the confirmation step. If the customer confirms, then we proceed with the exchange, and if they decline, we stop. Let's look at how we've defined those steps for our agent.
This line defines the order of the steps. If assess produces a proposal, confirm waits for the customer's decision. In Maestro, we define each step with create_step, and then we connect those steps into a workflow. So the first step calls prepare_exchange_flow to check eligibility, ask JEV about the issue, and then prepare a proposal when the request qualifies. If there is no proposal, then we return the result for our agent to explain. If there is a proposal, then suspend pauses the workflow and saves its progress. Our agent can show the proposal and finish its turn while we wait for the customer. And then when the customer replies, this step runs again with their decision in resume_data.
This condition only pauses the workflow when there's no decision yet. Now that we have one, the step continues. Decline stops the exchange, confirm lets the service attempt the exchange, and the workflow is registered with Maestro here alongside our agent. Our workflow is designed to wait for a customer decision. Let's see how a reply in chat gets it moving again. We'll ask for an exchange on the customer's behalf. The confirm_exchange tool result shows that the workflow resumed, an exchange was approved, and a receipt was returned. Our exchange completed, and the agent received the receipt. But a model might call a tool again if it misunderstands the result or it interprets another message as a request to repeat the action, for example.
Preventing duplicate exchanges is a service responsibility, with or without an agent, right? But with an agent, we also need to make sure that a repeated tool call doesn't cause unintended consequences. This check prevents the service from processing the same exchange twice. If the exchange request is already complete, then it returns the saved receipt. Making the same request safe to repeat is called idempotency, and this is classic software engineering, and it still applies when building agents. So the request ID is what connects. So the request ID is what connects those attempts. If every retry got a new ID, then the service couldn't recognize it as the same request, right? So that's why our code preserves the request instead of relying on the model to remember what it already did.
And repeated calls also cost time and money, right? So even when they don't change the data, we still need to be mindful. And while our agent may only call a tool once, that tool may have its own retry logic that we need to account for. For JEV, we allow two retries, giving us at most three attempts with a 10-second timeout for each attempt. So if the call still fails after that, then the tool will report an error. And these retries, they happen inside the tool, so they don't count toward our agent's five-step limit. We need limits on both the agent's tool calls [chuckles], we need limits on both the agent's tool calls and the work that each tool does.
Okay, so our agent has come pretty far, right? It can now handle an exchange from understanding the customer's request to completing the approved action. And we were careful to use multiple types of models for what they were best at, like interpreting customer requests and then choosing tools to call, while also using deterministic code to enforce the store's policies and the customer's approval. We've made excellent progress building a performant agent, and in the next chapter, we're going to move some of these service connections behind MCP, the Model Context Protocol, so that our agent can access them through a standard interface. [music]