Mastra Observability and Evals Masterclass with Alex Booker and Yujohn Nattrass
Alex Booker and Yujohn Nattrass demonstrate how to use Mastra traces, datasets, scorers, and experiments to investigate agent failures and compare changes. They cover error analysis, judge calibration, multi-turn evaluation, and catching regressions in CI.
videoMastra · 53 min · Nov 5, 2026 · youtube.com · Saved Nov 9, 2026
tracing, agent evaluation, evaluation design
Description
This workshop covers how to build an eval loop in Mastra using traces, datasets, scorers, and experiments to investigate failures and compare agent changes. Using the Mastra docs agent, we show how production mistakes become reusable test cases that help you measure improvements and catch regressions.
Hosted by:
Alex Booker, Product Lead, Observability and Evals @ Mastra
Yujohn Nattrass, Engineer, Observability and Evals @ Mastra
CHAPTERS
00:00 Welcome and introduction
02:29 Understanding traces, spans, and agent metrics
08:08 Eval fundamentals: datasets, ground truth, and scorers
18:39 Investigating agent runs in Mastra Studio
23:18 Running experiments and comparing models
27:55 Iterating on agents and calibrating LLM judges
31:19 Building datasets and evaluating multi-turn conversations
35:01 Error analysis: turning production failures into test cases
49:16 Catching regressions with evals in CI/CD
50:08 Q&A: platform integration, tool mocks, and prompt versions
WHAT IS MASTRA
Mastra is an open-source TypeScript framework designed for building and shipping AI-powered applications and agents with minimal friction. It supports the full lifecycle of agent development, from prototype to production.
📚 MASTRA RESOURCES
Website: https://mastra.ai
X: https://x.com/mastra_ai
Discord: https://mastra.ai/community/discord
GitHub: https://github.com/mastra-ai
Learn Mastra: https://mastra.ai/learn
Principles of Building AI Agents (Book): https://mastra.ai/books/principles-of...
Patterns for Building AI Agents (Book): https://mastra.ai/books/patterns-of-b...
Transcript
Welcome and introduction
0:00 Thank you to Card Rabbit for hosting us today. We're in like a new environment. Just getting the AV stuff set up. We're here for the Mastra offsite basically, hanging out with the whole team, working on some exciting stuff. >> Yeah, a lot a lot of cool stuff will be shipping and in the few near coming future. >> So, let me get my screen shared here real quick. Mm, screen.
0:23 What are you excited to ship this week? >> Uh well, I've been working with Daniel on the alerting stuff. So, right now in observability platform, it's pretty soon you'll be able to uh go ahead and create alerts on specific queries or metrics. Um something really important for production agents. >> Yeah, that's right. >> Yeah. >> Well, if you're watching and you've had any of these questions before about your agents, you are in the right place.
0:53 Welcome to our observability and evals masterclass. My name is Alex and I'm the product lead on the observability and evals team. >> My name is Eugene and I'm an engineer on the observability team. >> We both work at Mastra and we're building tools and a product to help you build agents. One part of that is the open-source framework, which enables you to define agents, give them tools, give them lots of primitives and controls.
1:18 Increasingly, we understand that people need help operating those agents. And so, if part of Mastra as well is observability and evals. Here's a quick preview at what the dashboard looks like. We wanted to host this workshop to teach you a little bit about it. We've been doing a lot of improvements lately and pretty much everything we're building, we're not just building it for the sake of it. We're thinking through what are the workflows that people are really struggling with. How do you build this product in such a way that folks like yourself can truly get value out of it and that it feels intuitive to an extent. Of course, today we're going to break it all down. We'll talk about how you can use observability to understand every agent run. We'll talk a little bit about how to monitor performance, cost, and correctness. We're going to build an
2:01 eval loop. >> Yeah, pretty exciting. >> Yeah. And we're going to spend a little bit of time at the end as well diagnosing failures with a technique called error analysis, which is kind of a nerdy idea from the machine learning world, but it's something we borrowed at the application layer. It's a mix of lots of lots of traces and turn them into something valuable.
2:19 One thing that I think is really underrated about evals potentially is using them to catch regressions in CI. We'll get into all of that. >> Yeah.
Understanding traces, spans, and agent metrics
2:29 >> Okay. So, how do how does, rather, observability help you understand every agent run? I think it makes sense to start with some definitions just to make sure we're all on the same page. And by the way, at this juncture, let me just say that if you have any questions or any thoughts, drop them in the chats. Maybe you, John, you can monitor the chat on YouTube on your screen.
2:47 >> Um yeah, about that. My actually internet is not working. So, this is more for decoration purposes here. [laughter] >> We're having all kinds of technical difficulties today. Uh but we'll keep an eye on the chat in a moment then. Okay. So, essentially the essential building unit of observability is this idea of a span. Maybe back in the web application world, it was quite common to use logs, but a span essentially captures something that happened within the agent run.
3:17 And then we assemble them into a trace, which you can think of as a receipt for everything that happened under the hood for an agent. Maybe it called a tool, maybe it called multiple sub-agents and different models. And then anything can happen at any one of these spans, and so that granularity helps you dig into what matters.
3:33 Each of these spans has some metadata encoded in it. For example, it's got an ID, sometimes it has a parent ID. It also has things like the input and output, which you can use to understand exactly what happened. Traces can look a little bit gnarly, I think, especially big ones. Um but there's nothing too crazy about them. Like, I mean, this looks more gnarly, arguably, but what I want to illustrate is that it's fundamentally just a JSON object under the hood, and we render it in a tree to make it a little bit easier to read.
4:03 So, to get those traces in your uh observability, you need to enable instrumentation. Which is where you record what happened during the agent run. The really great thing about using Mastra is where all of the instrumentation is native. So, you can enable a couple of lines of code, you will get deep rich instrumentation with no details left behind. There are two options in Mastra. One is to just do the open source route as you see on the left. In this case, you can use everything open source, self-host it.
4:33 Um most of the time you're going to use a columnar database like ClickHouse for this type of system. Although it does have some limitations. Equally, you can use the platform, which is a little bit easier to set up, designed to scale, and because it's online, you can share things like traces with your team. And we'll show you more of that in a second.
4:50 What are we looking at here, you done? >> Yeah. Actually, before we begin, I just want got my internet working here and just want to make sure that this is correct. Should they be seeing us or no? >> The >> You're just showing the full >> Oh, I'm just zoomed in on the screen right now. Just so the details are a little bit bigger.
5:10 But, rest assured. >> Oh, okay. >> We are here. >> Yeah, okay. That sounds good. Okay, I was just making sure. Um but yeah, let's go ahead and take a look at this. So, yeah, right here uh so right here we're looking at a um sorry, I'm not uh we're looking at a thread, actually. So, a thread is a So, when you talk to your agent um and you have multiple messages, they'll be stored in the thread.
5:38 And a lot of the times when you're doing um when you're looking through the traces, you don't really get the full context of the entire conversation. So, we added a new view called the thread view. And this will allow you to easily go and see um go ahead and observe not just one trace at a time, but observe the whole um the conversation um more easily. So, this will allow you to go ahead and like dive deep into, you know, um into maybe uh customer interactions that were flagged incorrect or um and kind of understand what went wrong there.
6:18 >> Mhm. Yes. Those traces can be a little bit tricky to read, but seeing it as a conversation view, like she pretty much just experienced like a user did, and you'll get a really good idea of like what happened, maybe what went wrong. So, when you send traces to the platform, you get two things. The first is a way to like search them, render them in different ways.
6:38 You can also do calculations on multiple traces, maybe like adding up the cost over time, and that's what we call a metric. And so, you can create a metrics dashboard where you track things like your spend as well as latency. Traces are very helpful because they let you inspect what happened in a specific run. They enable you to track cost and latency and things like that over time in the metrics dashboard.
7:00 You can also, you know, sometimes you have a very specific uh thing you're trying to look into, maybe like a user submitted a complaint or something. But often we're just trying to build the best agents we can, we don't really know all the ways in which people are using them or where they're failing. And so, because you have the traces, you can actually start to look at them, analyze them, group the You can start to group them by failure mode and see which ones perhaps are not performing and not working as you intended. That'll identify root cause in your agent that you can go on and address, and then ideally test.
7:34 But traces alone aren't enough to do that. You also need evals. And so, evals also let you do things like grade live answers with scorers. So, whereas a trace might tell you that this took really long, or you can look at it and know that something doesn't look right, an online scorer can automatically look at the trace and highlight things that need your attention.
7:54 A trace doesn't really tell you if the answer was correct, an eval can help you do that. If you do find a failure mode, it's all very good and well, but how do you verify it's good? it doesn't happen again. That's where evals come into the mix.
Eval fundamentals: datasets, ground truth, and scorers
8:08 So, an eval is essentially a test case which you then pass to a target such as an agent or a workflow, maybe a tool call. That agent run, typically it's an agent run, will produce a trace, and then you can run a scorer on that trace to produce a number. Here's an example of uh few test cases.
8:29 We group test cases into a data set, and so here we have our inputs. We're building a little Mastra Docs agent right here. We have some inputs asking typical questions about Mastra. And then our data set also has a ground truth. The ground truth is basically what you believe to be a good answer. It's not going to be an exact match due to the stochastic nature of models. We're not doing like a string comparison, but we're capturing something about what we think the correct answer looks like.
8:55 Sometimes that is just an example of a correct answer, as we see here. Other times it's like a rubric that says it must include these things, must not include these things. And then it's typical to use an LLM as judge or classifier as judge to then compare the output with the ground truth to determine if it gets a good score or bad score.
9:13 A scorer is very simply a function that takes a trace, you know, we saw the JSON earlier, and it produces a numeric score. You can use built-in scorers. We have a few that we offer to help you get started, but it's also very common to build them out manually, in which case you can do them using deterministic code where, for example, you look at the JSON and check if a certain tool was called. You can also use a classifier like JAB to determine if the given answer fits within the rubric. You can also use LLM as judge.
9:45 It's also kind of okay to just like review it manually sometimes. Like you don't always need a scorer when you're doing evals. It's actually optional. And that can help you with some nuance, for example, or in some cases where you're like calibrating an LLM as a judge scorer. >> Yeah, actually, one one thing about that is I think that the manual way is underrated. You know, a lot of people want to dive in and go ahead and create your own scores, but you can get a lot of value of just viewing traces by yourself and um um and sort of like figuring out your own problems, especially the ones that are uh quite critical to the product. Like, for example, I think one of the failure modes we found was uh step count not working, right? So, you can easily fix that by increasing the step count or
10:27 other ways, and um you can already you don't need a score to find uh to kind of fix that. So, you can go ahead and just fix it without any save the headache of creating a score. >> Yeah, I think there's a temptation sometimes to always reach for a score, but not always necessary. Let's look at a few examples of scores.
10:45 Maybe you, John, you can run us through these. Here's a code-based scorer that uses the It's got like a utility function there. I love asking you, John, about this cuz you're the one who built this in the master app. >> Okay. Yeah, yeah. Um so, to create a score, first of all, you go ahead and import it from our uh evals or the core evals package. And when you create a score, there're going to be um four different steps you can define. Um three of them are optional, which is the pre-process step. This is for like um sort of like formatting or preparing the data before you go ahead and run your analysis on it. The second step is analyzing the the data.
11:25 So, you know, go ahead and um seeing what is wrong there, and then through that analysis, you can go ahead and score your your um uh your trace there. And uh Sorry. Uh may I have you see that example once more once you're ready? Um And the final is reason. So, uh this is this is the reason that you generate for or the This is the reason for the score that you generate. So, in this example here, uh we only have the um the generate score step, and I highly recommend just try and keep it simple first um without going ahead and try and fill out like the four the three other steps you can do. But once you kind of get your score working properly, then it may be beneficial to go ahead and break things down in terms of like uh you can make it more cost-efficient
12:18 as well. Um especially when it comes to elements judge. >> That is a fantastic answer. Dude, I'm just going to take 2 seconds to fix the screen sharing cuz this is kind of lucky to be honest. Yeah. Okay, so there we are. Let me share my screen real quick. Sorry about this, folks. Uh doing a new studio but also like a new streaming software, so just figuring that out. I feel like a picture picture type thing would be Okay, this is perfect. We'll just leave it on this for the time being. Okay, so that's like a code-based deterministic scorer. Let's look Let's look at an example of a built-in scorer.
12:51 >> Yeah, so this scorer here um so built-in scorers there we have several different type or a couple a few different types. So here we have our uh elements judge scorer where you go ahead and uh pass in the model and you sort in it import this from our eval's package. So this scorer here is just checking if the output of the agent is correct. The um Yeah, anyways, that's that's one way to build a score. Um do you have a check Do you have example of checks as well or >> I don't have checks right now.
13:26 >> Oh, okay. >> Yeah. >> So yeah, before um we go into that, checks are just quick deterministic um scores. I kind of prefer those. They're very cheap to run. They're very uh you know, like I said, deterministic and they're very um what's the word? Your uh practical. That's That's the word I'm looking for. Yeah, they're very practical.
13:51 >> Recently we added support for Java as well to create classifier, so you can use Java for your scorers as well. Okay, so here's an example of the data sets, right? This is the inputs and the ground truths that we wrote. Then what you'll do is sort of run each item through the agent run, right? Uh or perhaps a different type of task, and then you get out the other side the output, which is then ultimately scored.
14:17 And you can see here that the ground truth for the question about does Mastra support Java? Yes, we do support Java. We just saw that. When we asked the agent, it said, "I couldn't find Java in the Mastra docs, so I can't confirm support." And then we know in that data set that this is something that needs attention because it scored badly.
14:33 One thing you can do when you run an experiment is you can compare experiment results. We call an experiment the process of running one or more test cases through the agent. And you will often form a hypothesis. For example, you might be building an agent with a particular model, and your hypothesis might be that if you use a cheaper model, you can save money and get the same quality results. You can go prove or disprove that by running an experiment.
14:58 And then suppose that you run multiple experiments, you can then compare them to see how things differed. For example, here I have one experiment using 5 mini, and another one using 5.6 Astra. You can see the 5.6 Astra is actually faster, but it's also more expensive by quite a margin. You can also see the score results for each of them.
15:19 >> Yeah, actually one thing I want to kind of mention there is although we're running experiments on these agents, you can also run experiments when you create your own email elements judge. And that's pretty important when it comes to like production. So, like uh what I've typically seen is like people you go ahead and want to use the smartest, you know, the the strongest models because, you know, you you sort of trust their judgment more.
15:40 But once you start running that in production, you you either have to sample your traces to make it more affordable, or you can go ahead and run experiments with two different models of your own judge and see which one you can the cheapest model you can get away with. >> Mhm. That's right. [clears throat] Well, I wanted to start with like basically a bunch of uh examples and just to get the the whole high-level picture. We spoke about traces and how they're made up of spans. We defined the scorer. We looked at data sets. I think it's helpful. However, there's nothing better than seeing an example.
16:12 What we're going to do is go to the studio now, and we'll uh run through some of this stuff together. I should take a quick moment to look at the chat. Do you know someone called Taj Ast? They said, "How are you John?" >> Hello. >> [laughter] >> Nice nice to see you. >> I think the the the the sort of layout was a bit wacky for a moment, but I think it's back now, which is good.
16:57 Okay, there was a quick question here. If you want us to catch regressions, how can those be included in the delivery pipeline where it doesn't cost a fortune? Is there a best practice on when and where the different types of evals run? >> Um yeah, actually that kind of goes back first to kind of the yeah. So, there's two different evals.
17:15 You have LMS judge and then you have code-based or deterministic scores. And when you're building your own scores, like the complexity the complexity of creating an LMS judge is pretty high and as well expensive cuz you need a model to run. So, I you know, if you're running these in production and you you need a score that's running on every single trace, try to, you know, if possible, keep it simple, meaning make sure you're only checking one thing, not multiple things within that score. And two, make it deterministic. That way it'd be fast and cheap. But kind of going back to the experiment comment, one way to also make scores more affordable is by using a cheaper cheaper model. But you have to make sure that the score quality doesn't degrade. So, first you're going to have to build the score the LMS judge up to your
18:08 standards, and then you start running and testing different models and seeing how low, you know, how much more cost-effective you can be. >> What do you reckon? Should we just like share the screen and do some do some demos, basically. >> Yeah, I think. >> Sounds good. >> Good place to start. >> Yeah, so let me switch to this view.
18:30 >> [clears throat] >> Can we full screen this? Mhm, not quite.
Investigating agent runs in Mastra Studio
18:40 Yeah, so if you don't know, the Master Docs actually has a chatbot where you can ask questions about Master app. And of course you want to monitor this and make sure that we're providing good quick answers. And so if you had to the Master dashboard if you're self-hosting you'll use a local studio, but when you're uh using Master observability you can head to the platform and load up studio there where you can access everything.
19:05 >> [clears throat] >> Let's head over to traces. And we can see a list of all of the uh traces for this agent. And so we can see the inputs and the outputs, the status, when they started. You can move columns around. We can kind of see here there were some errors. That could be something to dig into.
19:25 Uh so if there was an error for example, it wasn't uh we ran out of tokens. That's something that a monitor could catch potentially. And then sometimes you want to approach things from the spans point of view. So whereas a trace represents typically an agent run, a span represents everything inside of that agent run. It's just about the lens by which you approach things. So for example, if you wanted to filter spans and find it where the primitive type is maybe like a tool for example, and you want to monitor uh your tools or maybe add a filter for a particular uh like a particular tool to like make sure that it's performing how you expect, you can do that.
20:05 If you go to the threads view, you can also see things as they happened in threads, which is sometimes like in traces you can see the chat bubbles, right, and see what happened. Oh yeah, that's the message I just sent on the Docs agent. When you go to the threads view, you can see things you can see like the So each turn is a spa is a trace. And so, if for example um the conversation happened over multiple turns, that might be something that you want to see.
20:29 Usually, if I'm in traces and I'm looking at a particular conversation it might not always like make sense in isolation. That's usually a sign to me that I can open the full thread and like see how things played out. This is all studio, right? So, if we want to use the dog stage in our cell, we can come here and start talking to them.
20:51 Um but it can be a little bit inefficient to test your agent just by going back and forth with it inside of studio. So, what we can do is create data sets with some test cases. We already have a golden data sets here. Maybe you can talk a little bit about what a golden data set is. >> Yeah, so golden data set is um Well, first there's different kind of data sets. Maybe that'd be a better better question to answer. Yeah, when when creating these data sets um how they're initially formed is you go and go ahead, generate a lot of traces, either you or you and an agent or even an agent will go through your traces and just sort of try to find different types of questions uh or question or different kinds of messages that you would like um to ensure that
21:42 that uh you want to make sure that those questions are being answered properly. So, what are that maybe there's some critical information like for example a lawyer agent and you want to make sure that it's always citing the correct documents. Like this is a kind of a product lead like, "Okay, you can go ahead and create those kinds of questions."
22:03 But there's when you have an agent in production you uh we sort of touched this a little bit in the previous workshop about error analysis is when you manually go ahead and look through your traces and you sort of notice like little kind of product taste like something kind of subjective. And then you go and add those to another data set as well. And that will help you capture like how like find failure modes within your agent and how seeing how prevalent they are. So, when it comes to a golden data set, these are the the checks that like, you know, you want to make sure that they pass. Though, I would say when it comes to running these golden data sets, like one thing one thing you have to keep in mind is that um agents are non-deterministic. So, one of the main differences between a golden
22:49 data set and a test suite is that test suites you generally want 100% pass rate, right? But, when it comes to agents, um you sort of want to find that threshold. So, maybe, you know, 90, 80% is good enough. Yeah. >> Well, yeah, because you measure baseline and then a deviation from So, if your baseline is 90% and it goes down, then you know it's got worse. It doesn't necessarily have to be 100%.
23:14 >> Exactly. >> As you say, uh everything's still non-deterministic.
Running experiments and comparing models
23:18 So, we have this data set of uh like what I would describe as very common questions and answers. We can now run an experiment to see how the agent does. Instead of like manually So, I think the thing about evals that people sometimes don't see is that it could be easy to equate evals with scores. But, actually evals is more of a systematic process of improving your agent.
23:40 First of all, instead of just going back and forth with the agent, vibing and testing stuff, we've taken a moment to define a data set that we can reuse over time. So, if we wanted to by the way, I'm going to yeah, I'll I'll I was going to It takes like a minute to run all of these, so I was thinking about trimming it, but let's just let's just roll with it. So, you click new experiment, you pick the target. Remember that slide I showed you earlier? We'll pick the Master Dogs agent. We pick the data set, which is the golden data set. And we can pick a scorer as well. So, yeah, we could judge this the tone, but what I want to use this time is a custom LM as judge I made called a reference answer quality scorer. And what that's going to do is
24:17 look at the output for each test case and compare it to the ground truth to see what the similarity is. And using 5 nano, I think it might not do so good cuz it's quite an old model. So, let's just choose something more powerful like Astra. And one really nice thing about using the experiment dialogue here is you can just like quickly test your agent with different models. This doesn't change the model your code is using or what your agent uses by default. Just like test and see what happens. So, yeah, you can hit run experiment and kick things off.
24:45 Rather than waiting for this to elapse because it is obviously doing a lot of agent runs. I'm sure I can find something in here where Yeah, for example, before I I've got an experiment here with the same Oh, I need to filter on the golden data set. Let me see. Oh, here's a great example. Um Here I'm using Qwen, which is a open weight model that perfectly good, but probably not as powerful as Astra.
25:10 And the order matters here. Like the contender is compared with the baseline. The baseline is what you're using right now. So, like just imagine I'm using GPT-6 Astra in my project. And I have a question or hypothesis. I'm like, can I get away with using Qwen? Is that going to be better? Well, I run the experiment. I can check manually, but the best thing to do is compare and compare. And when you compare, you can get a good sense pretty quickly of how they're different. For example, Astra costs five bucks.
25:37 And Qwen costs 60 cents. So, it's almost 100% cheaper. It's slower, which I was surprised by. Actually, I thought Qwen would be faster for some reason. It uses far fewer tokens. You can also just kind of like compare the outputs and see, you know, without with the same prompt, the same context, how how are they different? You can get like a feel for the differences in the models and their personality, basically. Um I would say looking at this is that Astra is more concise. I'm not sure if that's a big differentiator, but I did notice that it didn't make any assumptions. Whereas with Qwen, it's just kind of like tried to It doesn't read very well, you know?
26:12 And like this might deter me from using a particular model. >> Actually, for this experiment, do you have scores on there or this was run without scores? >> This one, let me check. Um so, I don't think this one has a score on it, but that's a great question because Oh, yeah. So, this one just completed. And so, this is the one I just kicked off, but I thought it would take too long.
26:32 Didn't take that long actually using Nano. And so, you can see that this was the input, "How much does Master cost?" The output says I couldn't verify the current dollar amounts. And the reference answer quality score gave it a 50% score, basically 0.5. And every LLM is judged, at least the good ones, return a reason. And the reason why it returns a reason is because now you know why it didn't score 100% and then maybe you can do something with that.
26:56 Um I don't This is correct. [clears throat] Um The answer correctly distinguishes the self-hosted framework from the hosted platform and notes infrastructure and model costs. However, it never states that the open-source framework is free and omits the available hosted pricing. And this is what I would expect really because uh this particular agent is just looking at the docs. All of Master's pricing is like on their website, on the pricing page, for example. I almost feel like a five a 0.5 is a bit generous there. It might might get a zero.
27:26 Um And then when you click into it, you can also see the the >> Yeah, nice. Cuz like another thing about when you want to compare experiments like you kind of said, you want to you want to quickly um test another model. So, if you already have those scores already built and you can run them in then you don't have to, you know, manually look at the outputs. You could just like, "Oh, this is a better score." Or it maintains the same score. Or even maybe it goes down a little bit, but you're saving way more money, so.
Iterating on agents and calibrating LLM judges
27:55 >> Yeah. >> Well, well, here's the kind of interesting thing here. Um This is where the loop part comes in. Uh we systematically created a data set. We've run the experiment and now we know we need to improve things because we're not happy with the score of 0.5. And so, what I might do is go into my agent code, make an update, rerun this experiment, check if this number is now better, check if the score is better. But like you say, you don't always need a score. I could probably just look at it and tell you if it's not right. Um and more to the point that you didn't break other stuff in the process.
28:27 And to rerun the experiment, you just press one button like this and we run it. And you can do that a bit like TDD, honestly, where like you create the data set, you run it, and then you just keep improving it over time. And that's the loop, essentially. That's what people mean when they say a loop. >> Mhm.
28:43 Well, do you have a demo for that? Or >> I would, but I don't think uh Like you You mean like update the code and run the scorer? >> Oh, yeah. No, I was just wondering if we can demonstrate the loop, but maybe maybe maybe next workshop. >> Yeah. I mean, it would just be a case of updating the code and then rerunning it.
29:02 You believe me, I could fix it. It would get that score. >> Actually, one thing that's also kind of interesting about this eval loop is like going back to scores, you could do the same thing with LMS judge. So, like if you have if you have a way to make sure that let's say your data set is making sure whether or not you and your judge is aligned, and you have some way to measure that, um you can go ahead and Google this, look for uh true pass rate and true negative rate with evals. Those are two ways you can measure how well aligned your eval is to your own judgment uh when it comes to LMS judge. But once you have that, uh you know, that formula to make sure that your your judge is aligned, you could do the same process where you run an
29:48 experiment, you you see the the results of those two measures, whether they're aligned with you or not. If not, go ahead and automatically update your prompt for your LMS judge, and continue to you can say like hill climb your own judge just to uh make it better. >> I'm really tempted to like do the demo just because um I want to show one thing, which is that when you think about the eval loop, um speed is quite important. And right now, this is all on platform, but what you're typically doing when you're building a mushroom agent is like developing on localhost.
30:21 So, for example, I have studio running on localhost, and if you go to data sets, um Oh, that's not very encouraging. I'm not sure why my data sets aren't loading, unless I'm on a different port, maybe. Uh but then, what I do often, maybe this [clears throat] will work. No. I'm not sure what's going on there, but often I will like uh connect my local project to the cloud database. So, as I create those data sets inside of platform, I can also like access them locally and then run the experiment again locally um in the same environment where I'm updating the code, so you get like that really fast iteration.
31:02 Yeah. Okay, let's do a quick look at the chat and then we'll look a little bit of error analysis. Oh, this question been on the stream the whole time. Um
Building datasets and evaluating multi-turn conversations
31:19 Um, yeah, that's a good question about how to generate the Oh, sure. Yeah, how to generate the data set. Um, so in my case I just like had a good intuition about what the data set should be cuz I know what kind of questions people ask about Mastra. Um, sometimes you'll like create a data set from production traces.
31:40 So, for example if you go to uh plat- if you go to studio and you look at the traces you might just see a question that you think like oh yeah, that's something that Sorry. I'm on the local studio. You might just see a trace where you're like, oh, that should be part of my golden data set. For example, yeah, it's a great question. What are the features that can help me reduce token usage?
32:05 So, you click add to data set and then you can pick the golden data set and add it from production traces. Um, what do you think about synthetically generating data sets for NLMs? >> Yeah, I mean, NLMs are getting more stronger. I do think there is a proper technique to creating um a data set using our synthetic data set. And you can find examples of these online or past workshops that Alex and I have done.
32:30 But you pretty much want to go ahead and find different dimensions of a like for this docs agent for example. You want to find different dimensions of what a user would ask like are they a beginner? You know, are they looking for some specific uh primitive? And so, once you find these dimensions like uh I guess one would be user experience. So, you have beginner, intermediate, and expert. And then let's say you have primitives which is like memory uh memory uh I don't know what's not to do. Agent uh scores. Um you go ahead and you find all the different values of dimensions and you sort of create um each possible combination.
33:13 And then you have your element is or you have your, you know, own codex or cloud code or LM, whatever, and go ahead and generate the the user questions based on these dimensions. And what that will do is it will make sure that your the LM is not just generating the same kinds of questions just in different ways. This would be a warm a more broader um broader data set.
33:44 >> Mm, that's a good question here about evaluating multi-turn conversations, tracking each turn, tool call, and agent behavior rather than just input and output. >> Mhm. So, that's a that's kind of like Yeah, that's a that's a very good question and you can also even bring this down to like more like a trajectory score, right? Um So, when it comes to evaluating those, it is a it's a bit difficult and especially when you're using elements judge, I always say try to make it as bit, you know, as simple and basic as possible. So, if you're able to capture that failure within a single score or a single user message in a data set, that'll be great.
34:25 But, we also do provide ways of having uh you know, of having multiple messages within the data set as well. So, if you put like where's the weather in France? How about Toronto? So, um Um, could do this programmatically with Maestra. And uh we have ways to also um create a score that will score whether or not you you you have different gates between each step that you can help track and you also have a way to evaluate the entire conversation as well.
34:56 >> It's a great answer. >> Yeah. >> Okay.
Error analysis: turning production failures into test cases
35:01 Let's look a little bit at this idea of uh error analysis. So, sometimes you don't know what you don't know about your agent, right? Like, is it doing a good job? Where is it tripping up? Um, the only way to really get to the bottom of that is to just look at your traces, right? As you look at them, you make a judgment of if the agent did the right thing.
35:25 And as you make that judgment, you record it in the form of annotation. And then, you know, you have a couple of options. You could fix each thing individually, but to be honest, like that is a lot and might be a bit too granular because many issues will have the same root cause. And if you fix the root cause, you can focus on the high leverage thing and it's more likely that your agent is going to feel cohesive instead of having a bunch of one-off conditions and checks and things like that.
35:54 And so, that's one option to just do it fix it as you see it. But the other option is to label a bunch of data and then group the common ones into a particular category, put it in a data set, and that represents the failure mode. The cool thing about having the data set in the systematic way of building agents is that now you can use the data set to run experiments and check if the changes you made actually fixed the agents.
36:21 The other cool thing about building these data sets is they can all be compared like a regression test suite over time as well. This is a really lengthy process, isn't it? Like, what what what's your kind of take? Like, how many sort of traces do you think you need to look at to make this error analysis actually worthwhile?
36:37 >> Yeah. So, maybe I'll take a little bit of a step back um on what the kind of the problem that error analysis kind of solve? Um, Anyone who's creating a score for the first time will probably come, you know, usually find out like, how do I how do I know what scores to create? Um so there's two ways to create a score. One is the top-down, so you're it's like a product-led approach. Um example I always use is like a Twitter bot. You don't want to make sure that's under this number uh characters um in the tweet.
37:15 Um so that's a very well-defined um score that you can build. However, um there's different areas of where your agent can fail. And for example, let's say the tweet it made always starts with you know, here's the thing or some other generic AI language that may be more subjective to your taste. Then you don't realize this at first when you're just talking with an agent one by one, but perhaps over time they have a like 100 or 200 traces and you start reviewing these, you're like, this is actually a bigger problem than I thought. So then you go ahead and create a score with for that. And you do that through error analysis, and this is a more data-driven approach.
37:57 So back to your question about, you know, how many traces you should view, you know, usually I've been hearing about like people recommending about one 100 to 200 traces just to get started. And also that through this process is like you said, it's a way to create your like another data set for your agents as well. So you're getting a lot of value just going through this process.
38:21 >> Should we look at a few together just to show the example? We probably won't be doing 200 today, but we can >> Yeah, we could do >> [laughter] >> Yeah, that's true. We could do two. >> Let's do a couple. Yes, if you go to the Maestro observability dashboard and you go to traces, we can just start looking at things.
38:38 Um maybe I'll skip the German one. >> See, that's a pretty good reason to manually view traces. We have um people with from all different kinds of, you know, all different countries asking Masha Docs Agent. Um like you said you saw a German. I believe I saw Korean and uh we also saw uh maybe French and that could be Marvin, but yeah.
39:08 >> [laughter] >> Uh so I'm on platform. I'm not seeing the annotation button. Um pretty much the type is agent. Click on it. See scores. >> Interesting. >> The the feedback. >> Yeah. >> Yeah, it was >> Is that like Is that a photo? >> Um yeah, let me >> Just give me a hard reload. Mm. >> We do have a backup example, right? Or >> Sure. Yeah. Well, there's basically two ways of doing this error analysis process. One is to use it in the product. The other is to like export the traces into a spreadsheet, at which point you can uh start to label it manually. I'll give you like an an example.
40:04 This is how I how I did it the other day. Yeah, so here I like export this one of the question Not the prettiest rendering, but we can fix it up a little bit. So yeah, the question was what >> You may want to bump up the the size on that thing. Yep. >> Yeah. So the question is what data stores does this new knowledge feature support? It's talking about master knowledge.
40:33 Um here's the answer. And then we can come here and in the see in the feedback column we can leave some feedback. I was actually playing around with using um the master CLI and scale. So give my agent access to these traces and having it do the review, but actually it just kind of sucks to be honest. Like the agent doesn't know what a good product looks like. It doesn't know like when the answer is wrong sometimes.
40:56 You can get a little bit clever and have it like do verification so you can look up, you know, different like if it's answering a question about Mastra, it can do its own look up and just try and disprove it or something. Uh but that's not really the point of this. The point with uh it's called open coding where you look at these traces and you just write a few words about what you think went wrong. Um and honestly, the feedback I would have here personally is that like it's just too verbose.
41:19 I don't think we need like to see every possible database. Probably just like the the ones that matter. Um Here's another trace. This time the question asks it says my model context window is 1 million. And it says, "Okay, over 1 million token window." Again, it just feels like a bit um verbose maybe. So >> Yeah, actually I don't even know what that question was about. Was that even a question or >> Yeah, like it should have and it >> Or actually, this is a >> Sorry to cut you off, but this is a good reason why we created a thread view.
41:50 Maybe this was a part of another question, right? Um >> Yeah. >> So that's another reason why you probably want to do this inside platform. Uh it would platform was recently shipped uh the feedbacks for it or ways to add comments, but we've always had this in studio as well. >> That's right. >> Yeah. >> So suppose you go through a bunch of traces and you write a few words about like what you think went wrong and what you think the failure could be. What's the next step?
42:16 >> Yeah, so once you you go ahead and review, you know, the 200 traces or whatever. Um the next step is you can use an agent to go ahead and sort of cluster those um that feedback into common known failures. So and through these clusters allows you to identify, okay, what failures are happening with your agent and how often is this happening. This will help you This will give you some direction on what scores to build first and prioritize.
42:46 >> So Mastra has a a dashboard that's completely headless, so anything you can do in the UI you can do with the CLI, and therefore, you can use the scale, and you can ask your agents to do it. And so, I don't recommend using an agent for the open coding part, but it's very good at categorizing and grouping them.
43:02 And what you do with that group, just skipping ahead a little bit as not to do 200 test cases, is you basically put it into a data set, because a data set is a grouping of test cases. And so, when I did this exercise the other day, I came across like two concrete failure modes. So, the first one was that it would just miss documentation information. So, here it said, uh you know, does Mashrur support Java?
43:25 The answer is yes, but the agent would say no. It would ask, you know, these are my ask, what is MCP tasks? And the agent would say, like, we don't know, it doesn't exist in Mashrur, but it does exist, it's an experimental feature. Something quite curious, by the way, is that in in this question about, I think it's Spanish, uh someone asked about like the skip status. Um when the agent responded in Spanish, it was like, yo, I don't know what skip is. If you ask in English, it does know. I'm like, I had no idea. So, I started doing this error analysis.
43:52 And so, the idea is that if you come here and run the experiment, let's pick the Mashrur docs, and the data set with this uh series of failure modes. Uh let's see what happens when we run it using 5 nano, which is the default model. Um we should basically see that it's failing in some cases, and we're not using a score in this case, we're just going to evaluate it manually. And then it's the it's the same loop, right?
44:16 Like, we make it fail, and then we're going to like iterate on And again, as we create these data sets, they do essentially become durable in that you can use them for regression testing in the future, making sure that something you added doesn't inadvertently break this again if you add more language support or something like that. >> Mhm.
44:32 So, yeah, in this in in this experiment, what you could do after is, before you make any changes, just go ahead and create a score, so you know that you're actually making progress, and you have a way to measure that, uh instead of just going off vibes. >> You would make a score in this case, that's interesting.
44:47 >> Uh yeah, well, at least uh so, once you run the experiment, and you kind of find out what's wrong, cuz I think when you go into 200 traces, right? You got We're going to have a lot more than just three. So, and we don't want to, you know, >> Yeah. >> analyze that. And um but actually, yeah, anyways, so we'll see once this is finished, we go ahead and, you know, sort of manually score these.
45:12 >> Yeah, so it asked does Master support Java? It said no. Uh we know it does, right? Uh you ask what was the empty task. It says there isn't a distinct MCP feature. Um yeah, I wish I I should have used the reference score here. That would have been helpful, especially with more test cases. Um and if I Google translate this, it basically says there is no skipped uh setting.
45:33 So, the funny thing is here, like the reason they can't, you know, the next question is like why did it fail? And so, you can figure that out by looking at the trace and seeing what happened. And what happened, I think, is that it called search docs. It searched probably for um Let's see. I did two tool calls.
45:57 Uh search doc. Oh, yeah, first search doc is here. Uh so, it searched for Java, but we don't actually have like a top-level page called Java. We call it classifiers, and then inside of that, there's Java. So, I know that like in order for this to work, I'd have to add more of like a richer search to my agent. Um it tried to recover because it looked for like broader options, but it started looking for providers. It kind of knew that Java was a provider, it was going to find providers. Again, we call it classifiers.
46:24 So, I think the way I would fix this probably is by adding a better search tool, or maybe like adding a bit of guidance to the prompt that says, "Yes, Java is a provider, but it's technically classified as a uh a classifier." Um so, we can fix that, rerun the experiment. MCP tasks and uh I'm not sure. I don't know why that one why that one didn't match. We'll have to look into it. The the the language one is interesting because this is probably more so a limitation of the model than it is anything else. And so, probably the solution there would be to use a more powerful model.
46:55 >> Yeah, I actually Can you just what does the trace say there? Just the the tool call? I'm kind of curious there. >> This Yeah. >> Or there was no tool call there? >> I think there was. Yeah, I called search dogs. Um but yeah, there's such an Oh, it did translate to English, so it knew my tools, but only used English.
47:09 >> see how the English one worked. >> Yeah. No, but this is actually I think the input to the agent here is Spanish. >> Yeah. >> But it translated to English for the tool call because the knowledge base is in English. So, not as smart, but not that smart. And like the cool thing here is you can just click rerun. I mean, just It's always fun, isn't it, if you can just like throw a money at it and use a more powerful model? Of course, you're going to go into your agent and refine things.
47:33 Um it'll just take a while to go back and forth, and I'll probably end up cheating by like updating the system prompt or something uh when you want to address it more systematically. And by the way, sometimes you can ask your agent to look at the experiment results. Um you'll have no problem finding it. Look at the trace, figure out what went wrong, and try to fix it.
47:51 I've had like mixed success with that. Sometimes it it works, but sometimes you have to use your own brain, unfortunately. Uh but all the context is there if you want to. >> Did you have a score when you did that, or how did you How did you uh Or you just had a few examples to look at and make it better?
48:06 >> Um yeah, I just like narrowed in on the one that I knew was wrong, like the Java one, for example. I was like, why is it not finding Java? And I would just interrogate with the agent about it. Um >> I'm kind of curious if like if we had the score cuz then the agent could see, oh, this isn't the score is not changing. Go ahead and try different things.
48:24 >> And like L time one I basically >> Yeah, yeah, but uh maybe next time. We'll try that. >> [laughter] >> Next workshop. >> So, this time it did find skipped, I think. Um yeah. >> Oh, nice. >> And it did, I think, find MCP tasks. Uh but still, this failure mode of like Java not being found is the same issue.
48:43 Like that would require You can't just throw a better model at that one, unfortunately. I'm going to show one last thing, and then we'll look at some questions and call it a day, I think. Uh Oh, I had more slides I didn't show. >> Um slides. Yeah, so I guess this is an example of like an illustration of eval we show you.
Catching regressions with evals in CI/CD
49:16 Look at the our analysis. I just want to show you last of all that even though we mostly run the experiments inside of um the studio interface, you can also run evals using the run evals function. And this will return essentially uh a pass or not pass result and you can use that to like maybe in CI throw an error and that will stop the build and stop stop the deployment rather and then you won't deploy something to prod if the um evals are failing essentially. Yeah.
49:45 >> And there's that threshold uh option on there um which was set to one. That means they all the all the evals must score uh must pass in order for that test to pass. But we want in in reality you probably want set that threshold down to like 80 or 70% or whatever it's like you can get get away with.
Q&A: platform integration, tool mocks, and prompt versions
50:08 >> We've got a few minutes for questions. So um What does Maestro have that platforms like Brain Trust don't? Well, if you're using the Maestro framework, the Maestro platform fits like a glove. Like everything is designed to work well together. So the instrumentation happens automatically. When you saw us kind of pick models and compare them, that's something that we can do in Maestro because we also have access to the framework. And so we're just trying to build the most like perfect fit for Maestro right now. Whereas other platforms tend to be a bit more general purpose.
50:37 Um >> Yeah, and there's also some other features like tool marks for example. Um Like we make it very easy to add a tool mark where you can just go ahead and do a trace and create tool mark from a trace. Um Um yeah, actually I haven't really talked about tool marks at all. So just just to kind of explain uh really quickly. A tool mark is uh a way is a mark mark for a tool uh that you probably want for tools that cause you know side effects like calling a database or or whatnot. Um but yeah, so back to the question.
51:14 Um yeah, there's other features like that that we kind of support with Monster just out of the box. >> It's a good question here about running experiments on different prompt versions. Or rather um this question. Uh we do support something called editor which as prompt versions. There's not a great link right now between that and evals but there there should be.
51:35 The nature of Monster is that you define all of your prompts and so forth in TypeScript. So you can have versions in the sense that you can make different uh prompts to version control and do it that way. >> Yeah, so I recommend just testing out the the Monster editor uh package. Um where if you want like sort of like have a more frictionless way of running experiments with different prompts.
52:00 And maybe even tools I think. I need to double check that. Ask Daniel. So >> Sounds great. Anything else you want to share before we wrap up? >> Uh no. Um I think we're pretty much you know Oh, we're right at 10. So >> Perfect timing. Thank you so much everybody for tuning in. We hope you enjoyed this workshop.
52:20 We host these workshops every Thursday to help you build better agents. And I'm sure we'll do another evals one again at some point as we keep improving the platform. >> Yeah, and now that you evals maybe observability If anyone hasn't uh been here for the last month, there's been a lot of change. Haha. I can say that personally for myself. I wasn't I was away for a bit last month but uh yeah, there's been a lot of improvements. So go ahead and uh make sure you update your Monster packages uh to benefit from those.
52:49 >> Thank you so much everyone.