Skip to content
Linkdrop
Links

Build, Deploy, and Observe TypeScript Agents with Mastra

Mastra’s Alex Booker leads a hands-on workshop building and deploying a TypeScript AI teammate, then connecting it to Slack and Notion. The workshop covers observational memory, tracing agent runs, and using evaluation datasets to compare models.

videoMastra · 59 min · Oct 29, 2026 · youtube.com · Saved Nov 9, 2026

agent infrastructure, production deployment, tracing, agent evaluation, agent memory

Description

This workshop covers building a TypeScript AI teammate with Mastra, deploying it, and connecting it to Slack and a Notion knowledge base. Learn how tools and observational memory help it complete tasks, then use traces and evals to debug failures and compare model quality, speed, and cost.

Hosted by:
Alex Booker, Developer Experience Engineer & Educator @ Mastra

CHAPTERS
00:00 Introduction to Mastra and what you can build
07:10 Creating an agent with tools, schedules, and a workspace
18:41 Deploying your agent to Mastra’s platform
21:55 Managing context with observational memory
30:27 Connecting your agent to Slack and Notion
38:30 Q&A: memory, workflows, guardrails, and authentication
49:06 Debugging agent runs with traces
50:51 Creating eval datasets and comparing models
54:29 Q&A: connectors, self-hosting, and subagents

WHAT IS MASTRA
Mastra is an open-source TypeScript framework designed for building and shipping AI-powered applications and agents with minimal friction. It supports the full lifecycle of agent development, from prototype to production.

📚 MASTRA RESOURCES
Website: https://mastra.ai
X: https://x.com/mastra_ai
Discord: https://mastra.ai/community/discord
GitHub: https://github.com/mastra-ai
Learn Mastra: https://mastra.ai/learn
Principles of Building AI Agents (Book): https://mastra.ai/books/principles-of...
Patterns for Building AI Agents (Book): https://mastra.ai/books/patterns-of-b...

Transcript

Introduction to Mastra and what you can build

0:00 Hello and welcome everyone to this weekly Mastra workshop. This is your Mastra 101 workshop in which you will learn how to build, deploy, and observe agents you build in TypeScript. My name is Alex Booker. I'm going to be your host today. I'm a developer experience engineer and educator at Mastra, working across a variety of products including observability and evals.

0:29 You know, I host these workshops every single week and ship before it was about time that we did a beginner introduction to those of you who might not yet have encountered Mastra but are interested in building agents or maybe you already know a little bit about Mastra and you want to learn more. My name's Alex. It's a pleasure to meet you. What's your name? Please do take a moment to let me know in the chat what your name is, where you're tuning in from.

0:58 I would also really like to see a thumbs up if you already know about Mastra and are here to learn more and a thumbs down if you're brand new to Mastra and are just interested in what we can offer. Okay, so I'm seeing a ton of thumbs up in the chat. That's nice to see. Hey Shuman, nice to see you.

1:18 Yeah, lots of thumbs up. What's up guys? Okay, so we're going to talk today about how to build, deploy, and observe TypeScript agents on Mastra. We're not going to do too many slides or anything. We're going to keep this very hands-on and practical including a bunch of demos. I think it's a lot more fun that way and probably you do too.

1:39 I'm going to keep an eye on the chats throughout just to field your questions. I would, if I can, love to help you if any questions you might have about building agents in general. This is something that I've been going deep into for the last couple of years. Would love to share my experience with you as you approach building agents in general or on Mastra.

1:58 In case you're new to Mastra, this is a TypeScript framework for building TypeScript agents. It originally started as an open source project a few years ago. It has 30 K stars on GitHub and just an absolute ton of amazing contributors in the community. The core idea is that you install the Mastra package and then you define your agents, your workflows, your tools, pretty much anything you're going to need when building an agent, be that sub agents or skills, giving it access to channels like Slack. We'll We'll do that together a bit later, or giving it a primitive like a browser or a sandbox so it can run code.

2:34 Generally, when you start with the Mastra framework, you're going to find that as you need something, Mastra likely already has it built into the same place, so it's very easy for you or your coding agent to reach for. We'll get into the code, but just to give you a very basic idea of what a Mastra agent looks like, you instantiate the agent class. You can give it a model, in this case 6.1 Sol.

2:56 The fun fact, by the way, Mastra supports pretty much every provider under the sun. We have a bunch of like built-in providers for the OpenAI's and Fropics, Gemini's, all those big boys, as well as compatibility with routers like Open Router. Mastra even has its own router called Gateway, and you can use local models and different inference providers like Cerebras, etcetera, as well. No worries there.

3:20 And, you know, it's interesting because when Mastra started a couple of years ago, there was a big focus on helping TypeScript developers build agents, controlling the agent loop, interfacing with the providers, providing a standard interface, all that kind of good stuff. But increasingly as people go deeper into agents, we start to hear questions around guardrails, human in the loop, evals, the more productiony bits that you will inevitably need, and Mastra has all those things built into the framework too.

3:50 The other thing that emerged in building an open source framework is that people needed help operating those agents once they get into production. And so about a year and a half ago, we started to invest a lot of effort into building out the platform side of Mastra, which is basically a place where you can deploy your agents with one command. We're going to do that today and I'll show you how to get them into Slack as well. We also offer observability features so that you can trace your runs in production and understand how things are actually going, as well as of course things like um evals and data sets so you can continuously improve your agents.

4:28 One question I get a lot when I show people Mastra and maybe you have the same question as well is, you know, what can I build with it? What can I do with Mastra? Well, let me start with a couple of fun bits. We actually built our own coding agent similar to Claude Code or Codex using Mastra and we built this kind of nerdy CLI which enables you to connect your preferred subscription, be that Codex or Anthropic, and have complete control over your coding harness. This is an open source project.

4:56 We built it to dog food our product and in the process hardened our agent loop and built out advanced features like observational memory. Memory is such an important feature when building agents. I'll show you how all that works in a moment. We also realized that increasingly people want software factories and not just to run a harness locally.

5:17 So we built an entire software factory using Mastra. This is not a product that we sell. It's an open source project just to give you an idea of what's possible. And of course this all uses Mastra primitives and runs on Mastra platform as well. In terms of the more businessy use cases which is so important but maybe not quite as fun as building a coding harness, people are using Mastra to build things like sales agents, marketing agents. Company Brain is a huge one lately, enabling your whole team to share contacts from your knowledge bases and different environments, be that team communication or where you record your meetings, healthcare, customer-facing agents. A lot of the time one of the most telltale use cases for Mastra is when a product realizes their users are copying and pasting stuff from their product into ChatGPT, that's usually a good sign you

6:02 should build your own in-app assistance and many people use Mastra to do that. We provide a number of ways to interface with your agent, but we'll get into that in just a second. I'm going to take a quick pause to look at the chat. Running solo here today, so give me a moment and then we'll jump into some examples. This is meant to be hands-on, so I'm going to show you from scratch. I promise no pre-made It's kind of risky, right? But no pre-made code.

6:26 We're going to spin up a project, see it together, and deploy it. I hope you'll come along for the ride because I'm bound to make a wrong turn here or there. I am not an agent. I am unfortunately human, but we can figure it out together and let's see what we can learn. For people asking about the recording, most certainly anybody registered on Luma will get a recording.

6:47 That seems to be all the questions for now, but I'm sure you'll come up with some more because if you have anything at all that you're curious about, please use this opportunity to ask me your questions and hang out. That's what we're here for. That's why this is live. If there's not many questions in the chat, that means you're almost certainly going to get an answer to yours. So please come and do so.

7:07 Okay, I'm just going to make sure my windows are arranged reasonably. Yeah,

Creating an agent with tools, schedules, and a workspace

7:10 so how do you get started with Mastra? Well, the best thing to do is go to the homepage and follow the quick starts. You can either run this command or give it to your agents. What do you guys think? Let me know here in the chat. Shall we do the command CLI route, which I've done a lot and know inside and out?

7:30 Or should we try giving this to our agent to see how far it can take us together? People are saying CLI, CLI, do it manually. Okay, yeah, we can do that. So let's In fact, we can done it so many times that I can do it by memory. So I'm inside my code directory right here. I'm going to type PN PN PM create Mastra.

7:54 I'm going to use the latest version. And we're here today to build an AI teammate, someone we can add to Slack and ask to do work on our behalf. And so we'll call this AI teammate. And that's going to kick off the initializer. You can pick any model like I mentioned. I'll skip answering the key for the time being. And just like that, it's going to clone the agent harness templates, which we see as a good starting point for most agents these days. It has some things built in like a sandbox as well as some connectors for Slack. It also has memory. Let's check it out together and maybe see how we might want to augment it. So I'm going to CD, which is change directory, into the AI teammate folder.

8:34 And I'm just going to copy over some environment variables that I prepared earlier. And we're going to run the development server with the PND command. That's just an abbreviation that I have locally for PNPM Dev. And if you're using NPM, that's all going to work great as well. And very quickly, we're going to get to see this all in action.

8:53 So the way Maestro works fundamentally is that it starts up a local web server, which is an API, meaning you can interface with your agents from anywhere that supports an API. So it could be a web app, could be a mobile app, microservice, whatever. We also offer a development environment called Studio. Think about Studio like where you go to test your agents, where you go to debug them, where you go to use them and test them and check them quickly and maybe even share early access with your team.

9:23 Here's the first agent that's predefined. You can see that it's using 6.1 Sol, but if we want to, we could change this to Astra and use the latest version. When you want to know like what what capabilities does this agent have? Well, we know it has a model. We can also see here in the top right it has some tools.

9:43 For example, to manage its own task list or to ask the user for clarification by providing them a multiple choice question. It can also start and stop schedules to run things on a cron. It can also fetch things from the web and even search, which means if I ask it about the weather in Austin this weekend, it's going to use the search web tool to do so and let us know what's up.

10:05 I could also ask it to find the stock X um stock X, the SpaceX price and remind me every 60 seconds. And the reason why schedules are great is because you might want to do a recurring task, for example, a roll-up every morning or periodically checking your emails or something like that. In this case, it uses the web search tool to find the stock price.

10:27 It's going to hopefully tell us that our stock is doing great if you are an investor, and then it will continue to run that same command on a schedule every minute or so. We can also see that this agent has something called a workspace. Think about a workspace similar to a sandbox. This enables the agent to drop files, for example, if it generates a report or if it needs to drop an artifact like generating a video or something. It can also run shell commands as well. And by the way, there's that timer running uh in the background there. We got a notification from our signal system, and then it tells us the SpaceX stock price.

11:01 Personally, I might go crazy if I'm checking you know, my crypto stocks retirement fund stats every single minute. That's going to drive me nuts, so let me stop that. And additionally, workspaces enable you to give your agent a way to run shell commands. This is really useful because sometimes an agent doesn't strictly have a tool necessary to complete its task, but it could generate a short script to do so as long as they can do it safely, right? And a sandbox gives your agent a contained environment where it can run code but not access the external system where the agent is hosted and maybe read environment variables or do something else naughty.

11:38 You can also give your agent skills, scorers which are part of Evals, as well as memory. And we've we've got a great memory demo coming up that I'm excited to show you. We'll save that for a bit later. And yeah, here's the system prompt which enables you to give your agent an instruction about how to utilize that model and tools to accomplish something.

11:57 For example, just for fun, I could ask it to generate me a website. So let's copy the event page for today, and I'll go back to Studio, and I'll tell it to create an event website for this event in the style of the mastra.ai brand. And this I think is really a good example of how powerful models are these days.

12:22 It's going to use that web fetch tool. I mean we can't be sure what the outcome is just yet, but I expect it's going to use the web fetch tool to fetch the event page. It's going to give the details. Then it will hopefully fetch mastra.ai. It can look at the CSS and take screenshots and then it won't take a screenshot in this case because it's just using a web fetch, but if we gave the agent a browser, it could do a screenshot.

12:43 And then it will find the style in terms of the fonts, the colors, the vibe, and hopefully recreate something that looks pretty good. We'll we'll come back to it in a second. I just want to observe that this looks a lot like a coding harness like Codex or Cursor that you've seen before. Also looks a bit like Grok bots or Hermes or Open Claw. If you've seen any of those agents, you can create that same power in mastra, which means more or less any agent you or your business needs, there's going to be tools to make that possible.

13:11 I love this task list by the way that helps the agent stay on track when it has a task over a long horizon. I hope you don't mind me jumping around. This will just take a minute. That's the nature of a model. But in the meantime, I'm going to head over to the mastra dashboard and just show you that in addition to the open source framework, there's also a hosted platform where you can deploy your agents, where you can do things like um observability to see metrics about your agents. It also makes it very easy to connect your agent to other services like Slack and Notion.

13:45 In a moment, we're going to build this out into an AI teammate. And two hallmarks of that I think would be putting the teammate in the place where the work happens, which is going to be Slack in my case, as well as giving you access to a knowledge base of information about your projects and company and so forth.

14:00 So if you haven't already, you can head to projects.mastra.ai or just the homepage and create an account. We're going to mostly interface with the platform via the local environment. And one thing I just want to make really clear off the bat is that mastra works amazing with coding agents. You know, if you look at the code here, and hopefully this is big enough, and hopefully you don't mind my using light mode, but here we go.

14:24 The agent is defined similar to how we saw in the docs at the beginning, right? It's the new agent instance with a particular model. And you can change the model here as well. So I could update this to you know, Astro for future reference. And and you see the autocomplete there because being a TypeScript framework, of course, we care a lot about providing comprehensive types so you get good type safety and autocomplete or that kind of stuff.

14:48 And here are the tools that we define in the code. Like Maestro agents are fundamentally defined in code. And for the first year, year and a half or so, uh most of us were building our agents by handwriting the code. I completely accept, and maybe you do too, that developers are writing code less and less. And equally, you might be watching this and you're not necessarily a professional developer.

15:08 The really cool thing is that we give you a skill that you can give to your harness. Could be open code if you're a nerd like me, but could also be Codex or Claude from Anthropic. Could be Dots. It could be Grok Bots. You give this scale to your harness, and then it will have pretty much expert knowledge of Maestro in order to make changes in your project and navigate tasks like deployments or recovering from an error.

15:34 And then we give you a local command line interface that interacts with the platform. So even though we're going to be spending some time here in the platform, we're mostly going to ask our agent to help us. We're mostly going to use the CLI to deploy things. What are we going to deploy? But hopefully our agents. Oh wow, this is going agents have a mind of their own. This is remarkable.

15:59 It's using Playwright. It's figured out how to use Playwright because I have it installed locally. Um that is that is actually just a nuts example. I was not expecting that. But where there's a will, there's a way with an agent. And let's take a look at what it built. Here's our workshop page for today. Build, deploy, and observe. That was created using Astra. It's okay. I think the uh the the the green is a bit too neon-y for our brand, but the syntax highlighting looks similar. It's got some good copywriting too, that kind of bold pen. Don't you love this little You see this a lot in slides and websites, this very like tell-tale sign it was generated by an agent. But But just to be clear, like there's nothing really kind of coding specific happening in this agent. It's utilizing these general tools to do so, and of course the power

16:46 of the model. And you might have heard lately this idea of a harness. Well, you can think about this like a harness. It has things like memory, it has things like a sandbox. The simplest definition of a harness is that it's everything that goes around the model to make it more capable. And the model on its own did the heavy lifting, but it would not have been able to, for example, write files and um remember turns over a long period if not for the harness.

17:14 So how do we deploy this agent, right? How do we put it in hands in the hands of users? How do we get it onto the platform? Well, I'm going to take a quick break to look at the chats and then we're going to do so together. Just let me get a thumbs up in the chat if you can hear me. I hear there's I see there's like a few little audio issues here and there. Okay, that looks good.

17:51 Um good question here from data metal machine. You have a couple of dozen skills in the skills registry. I guess you're talking about skills.sh, but those all master specific or would they be usable outside of a master deployments? Essentially, we have one skill. I'm not sure how it shows up in skills.sh, but let's look it up. Yeah, this is our skill, the most popular one, and it's specific to Astra.

18:17 When you're building AI skills, you can basically create an point, which is your agents MD, and then within it, you can create basically sub-skills and references and scripts and all that kind of stuff. So we give you one entry point and then it will use more specific sub-skills depending on the task. For example, as it relates to using factory or something like that.

18:40 Okay, let's move on and we will recover

Deploying your agent to Mastra’s platform

18:45 my tab. I think it was this one and deploy this. So we're going to go to the CLI. We're going to open a new tab here. Let me just close the coding agent for the time being. And the first one we're going to do is run PNX master auth login. PNX enables you to run a package such as the master CLI without necessarily installing it globally. I prefer this, but you can also install it globally.

19:12 And then we're going to authenticate on the platform. This should pretty much just show me I'm already logged in, but if I log out for illustrative purposes, you're going to see that when we log in, it goes to the browser and I'm already authenticated. You can close the tab, come back. It's really that straightforward. And then what we're going to do is run PNX master deploy.

19:35 There's multiple ways of deploying your projects to master platform. One is with the CLI. You can also make master platform look at your GitHub repo and then it will do continuous deployment. When we look at the existing projects, there's not a good fit. We want to create a new one and it's called AI teammates. Although I did just realize I so deploying the nature of doing a build, the nature of deploying it does take a few minutes. Rather than you watching me do that multiple times as we progress, I had a brainwave earlier that I should make a version with master connect setup already. And then a bit later in the workshop, I'll show you how that all comes together.

20:16 I just added like three lines of code here. It's nothing it's nothing too much, but rather than deploy multiple times, we'll just do it once. So we'll do PNX master deploy. Once again, we'll create a new project. This one's just called teammates. I think that's coming from the package .json file. So let's take a mental note. This is just called teammates.

20:35 And in the future rename it. And then we'll create the project and deploy. This is going to take a couple of moments, but first we have to pick a region. I'm in London myself, but we'll pick US. And one interesting part about master as a framework is that it manages the database for you. When you use open and I'll just quickly put this into put this into motion as it might take a minute while I explain that you can use pretty much any database with master and master will manage it. Could be a lib sequel database, could be using Postgres or something else entirely.

21:07 The the nature of working locally is that lib sequel is perfect. Lib sequel is so fast and easy to get up and running with. But once you deploy to master platform, which is a serverless platform, you can't really have local databases running on the file system like that. Ideally you're going to connect to some kind of hosted database, which you can do yourself or you can use a provider like Neon in the case of Postgres, Turso in the case of lib sequel, many many others of course. And one thing we're aiming for with master and and the platform specifically is to try and offer you everything you need to ship your agent in one place. So if you need a database, we can actually provision that for you. So in the CLI just there it noticed that I don't have a hosted database and offered to make me one. So

21:48 why not? But just to show you it could be Neon rather as Postgres and soon we'll likely support MongoDB as well.

Managing context with observational memory

21:55 This is now doing a build. Now one thing you may have heard about building agents and working with models is that models have a context window size. A context window describes everything the model can see at a given moment. It can't see everything that you could possibly wanted to know because that would just be so enormous, right? It would just take too much processing power and the model would not work. So the name of the game becomes giving the model just enough context in that window for it can do its job without necessarily overwhelming it with too many details.

22:29 Most models can support about 1 million tokens in the context window, but the you before they throw errors. But the usable amount of contacts is usually far less. And if you have any experience using a coding harness like uh open code or a codex or something, what you'll often notice is that they end up compacting the contacts at some point, freezing for many minutes while they do so. And when it returns, oftentimes it feels like it's lost important information because the compaction process essentially is lossy compression. It essentially compresses the context and loses some details in the process, which is no bueno.

23:08 So compaction is a problem. The second problem is if you're building an agent that you want to call a teammate, if you're building an agent that you want to put in front of users, memory is what distinguishes a model from a teammate because a memory is things like personality, it's things about building a relationship with the model. Like it knows who's who, it has coherence as it can reference things you spoke about previously.

23:31 It's also a very efficient tool for giving the agent a way to stay on track as it completes tasks over a longer horizon. Sometimes tasks start with like known tokens in the context window, but they take many hours to run. Things like deep research tasks or coding tasks. And a memory system enables the agent to keep track of what it's done so it doesn't do it again, remember the important details, all these kind of things. And you might have seen earlier as we were running our agent that this system kicked in. It's called observational memory.

24:03 And while that deploys, and it may have already, looks like it's still chugging along here, I just wanted to talk briefly about observational memory and what it enables in agents. And so there are different ways of approaching memory. Observational memory is a novel approach that we invented at Mastr. I've created a little demo to illustrate how all that works.

24:25 And so what we're looking at is a contrived travel agents, and I might ask it to help me plan a trip to Tokyo. It jumps in and it's like, yeah, of course I'd like to help, sounds great. And now I'm going to say, hey, that's a lot. Can you keep your responses short and to the point? So just like a user might, I'm telling it a preference. I don't want it to forget that even a month later because my preference won't have changed.

24:49 It wants to know where we're going and where or like when I'm going and my budget. Next week, five days, 3K budget, flying from San Francisco. And as we go back and forth with the agent, you can see the context window filling. Context window sounds kind of fancy. It's basically a JSON blob of like everything that's happened in the chat, every input, every response from the agent, from the model more specifically.

25:12 I've got like a pretty formatter here that shows you the idea. And what's sort of interesting here is that when we call the get weather tool, it returns a blob of JSON that's also represented. Let's now search for flights and hotels. And now it calls two tools, search flights and search hotels. And quite quickly here, you can see this blue bar, which is the number of tokens in the context window so far. It's just ballooned to 2.4K.

25:38 2.4K is not a lot in the scheme of things, but the point I want to make here is that when you search a flight, when you search a hotel, when you ask the weather, the tool returns an enormous amount of context with it when only a small detail is going to be relevant. Like once I pick my hotel, once the weather is confirmed, once I've chosen my flight, only those like three tiny details matter.

26:00 But the hundreds of options for flights or hotels might still live in the context window in the form of the message history that is always going back and forth with the model. That's a problem. So as I keep chatting, we very quickly cross what is known in the observational memory system as the observation threshold. The observation threshold describes how many tokens do you want to fill before the observer kicks in.

26:25 What is the observer? Well, earlier when I spoke about compaction, I described the kind of lossy compression. What happens is the harness says to the model, hey, here's the whole context. I want you to like compress it for me. And in that compression process, it loses details. That's a synchronous process. With observational memory, when the observer kicks in, let's run it manually, although this does happen automatically.

26:51 It essentially carefully rewrites the message history into the most compact possible form without losing any details. And so here it condensed those 3K or so tokens into just 480. That's a massive reduction. It includes things like the preferences, the user's name, what their task is, but also details along the way. You can see that every single memory has a timestamp associated with it, as well as a importance indicator.

27:22 Eventually, yeah, it's true, you will still run out of context, albeit much much later. At which point the observational memory system can look at the temporal indicators, which are the timestamps, and the importance indicators, which are the emojis. And it's By the way, it's actually represented as emojis in the context window because it just takes a couple of small Unicode characters to do that, and it's cute.

27:43 And then it can gradually forget things that aren't relevant anymore, things like tool call results that haven't been relevant for a long time. Therefore, they are, you know, in this case, you know, low priority. They're just info. Things like cherry blossom season starting, there's no need to hold on to that, although it is handy in the short term.

28:01 And so as we continue to chat, you know, another another thing, by the way, sometimes tool calls just like fail. And even though they failed, you still get a bunch of their response in the context window. Anyway, we get to the limit again, we run the observer again. This process happens again and again until the observation's threshold is met. And what's particularly unique about it is that it happens in the background asynchronously as opposed to blocking.

28:24 So even though I'm doing it manually, a user would never experience it. In terms of how this works in code, you know, we'll just take a quick look here. We We our agents, and you might be like either surprised uh pleasantly, maybe you want more nitty gritty, but you might be surprised at how simple it is. It really is just a flag, right? And you set the model for the observer, in this case GPT-6 Luna. I'm using Luna because it's a slightly faster, lighter, cheaper model. Observational memory works particularly well with Gemini Flash and I'm sure it would work great with cheaper models in the GLM and DeepSeek families as well.

29:00 The whole point I'm getting at is that there's not a lot to it when you can define these things. You can also see by the way something that I skimmed over earlier, which is the ability to add human approval to tool calls. Anyway, let's go back to the deploy command and we should see this pushed. Uh-huh, perfect.

29:19 And then rather than click the links here directly, I'm going to head over to platform. Let's see. Get kind of lost in these tabs sometimes. There's the scale we opened. There's the advance. Maybe I closed the tab in my excitement. Let's have a look. Here's the teammate agent we just deployed 8 minutes ago. We started the deployment.

29:52 Two things have happened. First of all, you know that studio interface that I was showing you earlier? Well, you can now access it on a hosted URL. Therefore, anybody in your team can access it on a hosted URL. And it's a good way to like quickly share access with your team if they want to test it. Maybe they're a developer, maybe they're a PM or data science person or your manager or something.

30:13 And then we also give you the server which you can add authentication to and then connect to from your front end clients. And equally in a moment when we add uh Slack integration, Slack will need a online server to talk to, which is now already available.

Connecting your agent to Slack and Notion

30:28 Here's that database as well. Okay. Let's now add our teammate to Slack. I have a Slack org set up called Acme Alex, just a fictional one. And I'd like to add the teammate that we showed you earlier into Slack. And once we do that, we're going to give it some knowledge as well. Mastra has a feature called Connect.

30:53 So the nature of giving your agent access to external tools is that you often have to like fiddle with environment variables. And if they don't have environment variables, then maybe you have to auth with their MCP server. Now you're managing credentials and authentication. And then on top of that, there's also like a compliance side to things. You probably don't want to be like encoding your personal account into the agent.

31:16 You want to have it maybe authenticate on behalf of users. We launched Connect a few weeks ago, and it provides a very powerful way to give your agent that your team share access to the same environments like Slack and tools. For example, Notion or Confluence might be your knowledge bases, Granola, you know, Grain, Fire uh Fireflies, like these meeting recorders. When you add a connection, you can essentially set up and authenticate access to those environments once. And then your agent can go through Connect in order to access them without every user having to authenticate.

31:52 We're going to start by enabling the Slack channel. This is a little bit I'm going to I think possibly cuz I ran this earlier, it got a bit confused because I did deploy this earlier, but then removed it from Slack. So let's just do create a new Slack channels accounts. The way this works basically is you go to your Slack admin dashboard.

32:17 You want to create a token, pick a workspace, in this case Acme Alex, generate, copy the refresh token, go back to platform and paste it in. That should be valid and the credentials should take. At which point now you can install the Connect package and you can set the uh you can give your Mastra instance access to any channels defined in connect. This is that one like little tweak that I wanted to uh so if we go to CD AI teammate with connect and then open it, you can see that this is basically the change I made. Like nothing too special, but I didn't want to like do that and redeploy and it take a few minutes.

33:01 Additionally, you know, channels represent a way to add your agent to channels such as Slack, Telegram, uh Microsoft Teams, of course. Tools provide a way to give your agent access to knowledge bases. And you can see here the inside of the default agents, if I go to tools, we have imported master connect. We call connect tools in addition to any tools that we might enable via platform, and I'll show you that after. We also pass those tools that we saw earlier manually.

33:36 And so because this is already been all done, you know, we'll click done. And then when we go into Studio, we'll use the hosted version. And we click into agents. Hmm, interesting. I was expecting to see the Slack connector pop up, but I didn't. I do believe this will be because I think possibly the issue is that I previously connected Slack with this.

34:14 Let's try something else just to verify this and we'll see if we can fix it. What I want to do is go to connect and I want to add a connection. Yeah, and just to check I'm in the teammate project. That makes sense. Let's see if we can give our AI teammate access to Notion so it can access our company knowledge base.

34:34 To do that, we'll click add connection. We're going to search for Notion. Yeah, once again it says one available to link. Let's if this should work though. So we'll click it. What that means fundamentally is that earlier I pre-authenticated notion and the authentication belongs to my, I believe account on Master platform and notion. So even though I'm in a new project, I think it still remembers the authentication which I set up an hour ago when I was testing. So there's no problem there. It just means I don't have to do the notion OAuth flow. And when I go back to Studio and reload, I'm hoping to see and indeed we see all these notion tools added.

35:13 We can now ask it a question like use notion to look up Mastra's company address. And perhaps rather curiously, I can now see the Slack channel. So maybe that was just like a little bit scrolled off screen before. Maybe it just showed up when I reloaded. Let's just give this a second to run. The nature of giving your agent access to So there's an interesting challenge here.

35:39 First of all, notion has put a bunch of tools. You can see them in the top right on the agent. And so the model has to kind of pick a tool. And sometimes it can pick the not best tool because it gets a bit overwhelmed. It might do a general search instead of searching for a specific bit of text or prose perhaps.

35:55 Um the other thing that can happen with notion is that it's not really optimized for agents to search it. It's not like a vector store. I think because a lot of people contribute to notion, it's not necessarily naturally organized, right? And so it sometimes can be a little bit inefficient to grab the data. But here it used the notion search tool.

36:14 It put in the query Mastra. Yeah, that's a bit broad. It narrowed down the pages and then it went to get the page with this ID. It returns this content and then it grabs us the address like so. Let's see if we can connect this to Slack real quick. Let's hit connect. Aha, okay. And now because we set up the Slack connector in platform, it becomes very easy to just allow and give access.

36:42 And so we're installing the Slack app right now. Because Master Platform has that refresh token, it can authenticate and stay authenticated such that we can add or remove connections. Of course, we can add other channels as well. And now if we hop over to Slack, you can see there's a new guy here called Agent. By the way, if you go back to the admin dashboard on Slack now, there's two agents, right? We'll reload.

37:04 We're going to see agent pop up here. And you can basically go in here and edit all the details like the avatar, the name, the online indicator, the typing indicator, all that kind of stuff. What you don't have to configure is all of the webhook stuff because when Master Platform via Connect creates the app, it takes care of wiring up the webhook callback and give it and then funneling any requests that comes from Slack into the server that we saw running all earlier.

37:30 Feels pretty seamless, right? And so now we can go into Slack and just, you know, ask it a question again, what is Mastra's company address? I'm going to be prescriptive and tell it to use the Notion tool because otherwise it might search the web. That would be reasonable because I haven't defined it in the system instruction. And just give this a second while the agent wakes up and kicks in. Now it's working. And by the way, fair play to Slack, they've added a lot of much nicer user experience features to help agents be great in Slack, things like the streaming of the responses, things like the tool call rendering. It all it all looks pretty native and nice, right? And so let's just give this a second to to prove it out and then I'll take a look at the chat and answer any final

38:10 questions you might have. It's interesting. It's going through that same process, I'm sure. There we go, searching, getting pages marked down. And there we go. If you want to send us some fan mail, there's there's the address for you.

Q&A: memory, workflows, guardrails, and authentication

38:30 Okay, let me field some questions and then we'll wrap up. Okay. Uh does Mastra supply pre-made agents? That's a great question. We actually have a templates database. We actually build one a week at least, and so there's always new and interesting templates coming out. Just flicking through them so you can see. We also partner with different companies on building agents to show you how Master can integrate in their ecosystems.

39:02 And by the way, if you go to the docs and the integrations tab, we have even more. Like we really pride ourselves at Master about being very open. Like and just to be clear, even though I have deployed Master on platform, it's fundamentally a Node.js server, so you can deploy it anywhere really. Anywhere that supports Node.js. Uh coming out Superbase to that list soon please on platform. That's a good feature request. Thank you.

39:30 Are reflections clipped if they get too large? Eventually yes, even though reflections are highly efficient, you will eventually run out of contacts at which point the observational memory system gradually drops details. However, there is a feature within observational memory that can extract important details into longer-term knowledge. So that even though they get removed from the context window, they can be retrieved later if required.

39:59 A few short messages instead of one long wall of text is a cool idea. Can Master send a few consequent messages in a row? The nature of an a model interaction is that you send a prompt, right? And you get back a response. You're kind of taking turns back and forth. The good thing about signals, which is the system that powers that schedule I showed you earlier, is that the agent can then call the model independently of user inputs, meaning that you don't have to wait for the user to input necessarily. And so in that case it could do two consecutive calls to the model. The model is fundamentally always return one response.

40:37 Does observational memory just make contacts when they go further or does it make us of a chat thread can keep going on forever? Essentially the same thing, right? Because a thread is so when you interact with a model, what typically happens is you send your inputs, you get back an output. I say hello, the agent says hi back.

40:59 When I next send an interaction to the model, what you do is you include the previous message history. So it says hi, hello, the agent said the model said hi back, I say something else. And this happens and happens and happens while you call the model because the model has no real state, it has no memory. So you're always having to remind it what happened recently.

41:17 In the case of observational memory, we send the last few turns uh for short-term coherence as well as the longer-term context. And that is done in such a way that it's highly prompt cacheable, which can save you money in the long term as well. What if you wanted to build your own way to manage context and memory? Is that possible?

41:36 Yes, absolutely. And by the way, one thing that's really worth shouting about, I think, is that the observational memory system is built on master primitives. So the core construct here is this idea of a processor. Think about a processor like in a like a hook. Uh it lets you do something in the agent loop um at a particular moment, like just before calling the model or just after calling the model. And we leverage these for all kinds of things in the master framework. So for example, guard rails use an input processor in order to verify that the user isn't trying to do prompt injection. Equally, guard rails can use an output processor to redact personal personally identifiable information. And we have some built-in processors to that effect.

42:18 Memory leverages the same thing, right? Like after the model returns, it uses the output processor to do what it needs to do to retain the right context at the right time. And in fact, many users do build and augment their own sort of context management systems using processors. They're honestly one of the most powerful features in Mastra combined with signals. But more on those another day.

42:42 Uh it's observational observational model something of a meta harness. Um I wouldn't a meta [clears throat] harness to me is a harness that calls of a harnesses. It's funny you should mention that because Mastra does actually support via our connections uh 82A ACP and SDK agents. So sometimes what we'll see is uh customer or an open source user has a very highly tuned like cursor setup or highly tuned Claude setup that the whole team is on board with, but they still want to build an agent within their enterprise. So rather than converting everybody to Mastra, they build the agent they want to build and then it uses SDK agents to call into for example cursor or Claude Claude. And because you're using matter because you're using Mastra to build a harness and Claude cursor and open AI agents SDK harnesses you could call that uh a meta

43:32 harness, but I know that doesn't quite track with your questions. So let me know if there's something else I can share there. Um is there any amount of crossover with these concepts and Mastra workflows? So to be honest, when we first released Mastra a few years ago, workflows were something I talked about a lot um because they enable you to break tasks down into a series of steps.

43:55 Each step might do some deterministic code, things like validation don't need a model. They might also call a model and do something agentic. And the idea is it gives you more control over how your system uses models, right? So you tend to delegate smaller tasks in each step, have a bit more tighter control. And this was hugely hugely advantageous before models got so good at reasoning.

44:17 And now they're so good at reasoning, workflows are a little bit less popular than they were, but they still certainly have a place. It just so happens that in many cases what you want to do is give your agent a way to dynamically create its own workflow. And that's what dynamic workflows enable you to do. They basically represent a workflow in JSON such that an agent can write JSON, an agent can create a workflow and create like a reusable What doesn't have to be reusable necessarily, although it often is like a procedure that it can execute.

44:44 And so in terms of overlap, there is certainly a little bit, right? It's all Mastra at the end of the day. I just thought it would be interesting to mention that what we've seen is people using workflows a a less as they come to trust and lean on the model's reasoning capabilities more. What about compaction? Does this prevent prompt caching? That's a great question.

45:05 No, it is not. Prompt caching looks at the first block of the context window in order to identify a prefix, search for it can reuse um computation it's done in the past when generating a response. And the way that observational memory works is that it doesn't really mess with the prompt prefix. You saw a bit earlier that every time, you know, this literally represents um what the model what the provider, the inference provider would see. And so cycle one, once it's done, that lives in the prefix. Never changes. When observation two happens, it came after the first cycle. When observation three happens, it comes after the next cycle.

45:47 So because we're always appending to the prefix, we don't break the prefix and we get the most out of the prompt caching, which means you get to leverage those prompt caching tokens and save on input tokens. There's generally a lot of things we try and do in Maestro to help you utilize the prompt cache. It didn't used to be so important, honestly. Output tokens are more expensive, right? But increasingly the amount of input tokens we send to agents has ballooned. And then even though input tokens are less per token, we end up using more volume and therefore prompt caching, which can in some cases be 80% cheaper, becomes very important. So that's a that's a great question.

46:25 Um are there any baked in guard rails on the third party auth component or is it entirely up to the users to apply least privilege? Yeah, great question. Maestro supports a lot. So, you know, when you're building a system, you probably have your own like users table, you probably have your own way of thinking about auth already. Maybe you're rolling it manually, in which case we give you the right white spots to fill in with a function that can interface with your database, do the auth, return a token, all that good stuff.

46:54 We also work with a bunch of different providers, right? So by the way, that's JSON web tokens. We also work with a bunch of different providers including Clerk, Auth0, WorkOS are awesome. And so you can just drop those in and it's probably the fastest way to add auth to your server or your studio instance. You know, authentication as you probably know, but let me just say it anyway, is about verifying the user is who they say they are. It doesn't control what that user can do. That's what authorization is. And in Mastra we also have a feature that there are different ways of doing authorization, like role-based authorization for example is fairly trivial to implement. But we also support more advanced type of authorization called FGA. This is particularly useful in multi-tenant apps and more complex apps as well. It's not necessarily better than role-based by

47:38 the way. It's just an alternative way that is sometimes more powerful depending on the circumstances. We have time for like maybe a couple more questions, so if you want to pop them in about Mastra or in general, hit me up. Coming to the end of my list here and then I'll bid you adieu. Uh so there's a few messages here from Esteban.

47:57 While experimenting with OM on small reflection observation tokens, it will attempt increasing compression levels to fit the reflections. I know the baked in OM structures can be appended to, but is there any plan to make it not so insanely long and allow for fine grain modifications? Also, please y'all stop honking. Stop honking out your window. Oh drivers have no chill. Yeah well that's uh living in London for you. These are great questions, Esteban. I have to confess I I don't know. I can definitely talk to Tyler who is the lead engineer in observational memory and see what we can learn about that. For the time being I'll consider it a feature request.

48:33 And you know and that's what we're we're kind of about at Mastra. You know we we have this open source framework. It's entirely free. It's used by the community. We built the platform to help you operate the agents. We're very active in the community. That's why we do these weekly shows as well to help teach people how to build better agents.

48:51 Next week I'm actually doing a observation uh observational an observability and Evals master class. I'll end quickly with a quick demo of Evals. And so, when you're building

Debugging agent runs with traces

49:06 Yeah, let's just do a quick five-minute demo of Evals. So, essentially when you're building an agent, very different from how we used to build traditional software, where people would give like very specific inputs, right, a form for example, and requests would take a very very pre-known path through the system. It's not how agents work. Users can type anything in that text box. Like we saw at the beginning of this show, it can do surprising things. Sometimes good, like in my case I was like, wow, they figured out how to use Playwright. But sometimes bad, right? Like it might do something wrong, it might get stuck in a loop. A lot of things can go wrong.

49:40 And so it's very important that when you ship an agent into production, you also enable traces. Think about a trace like a receipt for every agent run. It kind of breaks down exactly what happened. Yeah, okay, the core part is calling the LLM, here we called Astra. But did you see that as the agent loop ran, it also called a bunch of tools, it ran a bunch of processors, which might be for memory, might be for guardrails. These are all opportunities for something to go wrong. And so we collect this receipt of everything that happened. And then if something did go wrong, for example, curiously, here it tried to edit a file, that tool call failed.

50:13 Didn't really bother me, it recovered. But if I wanted to dig in and see what happened, I can go here, I can click in and start to have a look at the specific error and maybe sort of identify like a recurring failure mode. And so observability is really important for this reason. It's also perhaps a prerequisite for building a systematic way of improving your agents with Evals. An Eval is where you take a test case, could be something you know to work or something you know to fail, and you run that test case through your agent and verify its output.

50:43 A little bit like a test, although quite different still because the system is so much more non-deterministic due to the stochastic

Creating eval datasets and comparing models

50:51 nature of models. This will make a lot more sense when I when I show you. So for example, here's how I built an agent, our our personal assistant agent. I'm I'm create a new dataset. In fact, let me just rename it to golden data set. Golden data set, they often say is a table of inputs and outputs that you know work and you know what good looks like. And then as you continue to improve your agent, you can run this eval over and over again to make sure that your system is improving.

51:20 I'm just going to give it a question like what is the weather in London? Optionally, you can supply a ground truth and then you can use something called an LLM as judge to compare the agent's output for the test case with the ground truth. We're not going to do that right now. We're not going to do tool marks. We're just going to add a test case. Let's add another one. Um, you know, what is the SpaceX stock?

51:43 Two things we did before, but instead of like doing it manually and just kind of going based on vibes, when you create a test case, you can run an experiment. And so let me click new experiments. We're going to pick the agent target. We're going to use the 6.1 SOL model in the golden data set. And what this is going to do basically is take every test case, you know, those two inputs I gave it, and it's going to run it through the model and show us the outputs.

52:06 I've only got a few minutes. We're kind of blitzing through this. There's lots of more advanced things we can do here that you're going to learn about next week when you tune into the workshop. You can find a link to that in your email after this event. But check it out. We kind of see the cost for running the experiments. We see how long it took. We can also just kind of look at the output and see what happened.

52:25 That that alone is like pretty valuable. The one thing that happens a lot when you're building an agent, and tell me in the chat if you can resonate with this, is that you want to try different models because maybe your model's kind of expensive and you want to know can I use a cheaper model and get the same quality results?

52:42 Have you seen how expensive Astra is? Check this out. If I click new experiment and I pick an agent, and let me just I just want to know how much more expensive is using Astra versus SOL. So I'm going to change to Astra. By the way, a pro tip here is if you use a gateway like OpenRouter or Maestro gateway, you can just like pick any model and just explore to your curiosity's content.

53:03 And so we're running that same test case, but this time we're using Astra. And and all I'm trying to do right now is figure out, what do I like more, Astra or Saul? I'm going to look at the cost, the speed, but also the vibe. Like I'm curious how it changes the output of the agent. The next step here is to add things called scorers, which basically look at the output from the agent and score it based on some criteria defined in code or by an LLM as judge. We're not we're not going to do that right now.

53:30 So yeah, I can kind of poke around and see what they did, but it's kind of hard to compare, right? Except I can go back to experiments, pick my default case, my baseline. So let's just pretend I'm using Saul and that's my baseline. Baseline is like what you're currently doing and what you're kind of comparing against. And then I'm going to choose six Astra as the contender and click compare.

53:52 And now the reason the reason why this interface is so powerful is because I can kind of look at the outputs and see how they're different. Not much difference, right? You could maybe say that the Astra output is a little bit more concise in the bottom right, but I wouldn't pick a model based on that. The really interesting thing is that Astra, in this case, was 15% faster.

54:10 Also 400% more expensive. Astra is so freaking expensive. And so these are the kind of comparisons you can do when you build a uh when you build build an observability loop.

Q&A: connectors, self-hosting, and subagents

54:29 [snorts] Yeah, I can talk a little bit about signals. Let me just field some questions real quick. What features can you expect in the near future? I'm super stoked for Astra knowledge. I actually pulled up this tab earlier. Memory is just one half of context. The other half is knowledge and knowledge is power. This enables you to build a company brain. Like all those problems with notion, like it's a good easy thing to do, but it's very inefficient, right? Unless you give it very specific system instructions or skills, which can be done.

54:59 Knowledge enables you to bring this into your own hands. So very excited to share knowledge with you soon. We're also working on more observability features, including custom dashboards and some fun stuff like that. Um, Is it possible in terms of for each user to authenticate themselves using their own individual accounts, in this case confluence, or you always required to use a shared collective account?

55:21 Brilliant question. I wondered something similar when I first started toying around with Connect. Essentially with Connect today, you're basically sharing an account between the whole team. We are working on support to enable the agent to initiate an OAuth flow and then the user, the end user can authenticate with their own accounts. For the Slack and Notion Connect as I just demoed, can we run those fully local self-hosted on our own node server?

55:52 So Mastra Connect requires using Mastra platform. That is true and it's a very valid thing to call out. I should have done that. It is however completely possible to build these tools in such a way that they run on your server. Uh, the challenge there is you'd have to maintain the authentication, right? But it's something that people have been doing with Mastra for two years before we built Connect. So it's certainly possible. You can also if you like use alternative services like composio.

56:19 I think Arcade is still doing tools versus Connect. It's in a similar kind of category. So you have some options there as well. Not too familiar with the medallion architecture, but it does sound interesting. Uh, in the trace tree can an agent route lightweight processors to a fast model while keeping the core agent on four zero sonnets? Well, don't use four zero. That's a very outdated model.

56:45 Um, but essentially yes, absolutely. This is the concept of having sub agents, right? And there are multiple ways to architect sub agents in Mastra. We support something called the supervisor pattern, which is where you define your main agent, your supervisor if you like, and it can delegate to sub agents you define on the same agent instance. You can also create tools that call into other agents. You can also use workflows and you can mix and match different models. And even in some cases, I don't think it's that really worth it, to be honest, from what I've seen. But sometimes people implement their own type of model router where they classify the complexity of the task using something like Jeff or any kind of decision model.

57:26 And then they will route it to the appropriate model. Uh for teams getting this week before the Evals masterclass, which starter template in the docs do you recommend clone cloning to test observational memory plus Slack first? I mean, in terms of the bit of a disconnect here because the Evals masterclass that is a bit separate from observational memory. I actually myself wrote a guide over here on building an Eval loop that talks about what I just shared in a lot more depth, and you can follow it along. Just Google building an Eval loop in Maestro, you'll find it.

58:00 In terms of observational memory, I recommend you go to the docs and read more about it. We also on the Maestro YouTube channel did an observational memory workshop. So if you just Google that, you'll likely find it with Tyler and I. And in fact, Maestro's co-founders, Shane and Abby, gave a talk about this recently as well. So we've got a fair bit of content here if you want to dig a bit deeper into observational memory.

58:26 And frankly, I'm excited to do more observational memory content. I think it's such a a fun subject. Okay folks, few minutes over, but I was having so much fun. I hope you had fun as well. Please do check out Maestro if you haven't already. I've been Alex Booker. You can follow me on the link in the chat, and please do follow Maestro as well. We are releasing new features every single day. Truly, like when you think about all the new and exciting things happening in the world of agents, famously Jeff a few weeks ago, we learn about it, we encode it in the framework so you can start benefiting from it and build capable agents. We've just been doing this on a loop for the last two years, and we're showing no signs of slowing down. So please come along for the ride.

59:08 And if you have any questions about building agents with Maestro, hit me up on X, join our Discord, or get in touch with our sales and support team if you need help integrating Mastra into uh product that you and your team are working on. Until next time, I've been Alex Booker. Good