Skip to content
Linkdrop
Links

MEGA.dev live Q&A with Angie Jones

Angie Jones answers audience questions about building reliable agentic systems and introducing AI tools across engineering teams.

videoAngie Jones · hq.mega.dev · Saved Nov 19, 2026

mega course, agent orchestration, agent evaluation, engineering teams, agent infrastructure

Transcript of the recorded MEGA.dev live Q&A with Angie Jones (Wednesday, Nov 4 · 91 minutes), Angie Jones. The recording plays in MEGA Student HQ (needs a login).

0:00 Speaker 0: So hey, everyone. I'll, I'll just

0:03 Speaker 0: kinda briefly, uh, set, set the stage for, um, what week two was about. Um, so I think, uh, last week Theo did an amazing job of going through, like, how to master using agents. And so my week was a different approach, um, in, okay, I think all of us are learning how to use agents effectively. What I think is the next phase, and, um, I kinda live, like, like someone joked with me last week, like, "Oh, this is Angie. She lives in the future." Um, and so, like, part of my job is to think about, like, what's coming next and help prepare, uh, developers for that. And so what I think, uh,

0:54 Speaker 0: is next up for us is we will be expected to know how to build agentic systems. So I think companies are going to want, you know, um, to have a commercial agent or some agent that's built into their products. Uh, I think customers are gonna begin to expect this from a product. Like, "Oh, I gotta go to your website and find stuff on my own? Like, what?" Um, you know, when I go to Amazon now, I don't know how many of you have started doing this, but they have, like, a built-in agent there. And so now I'm asking questions, uh, like, "Hey, uh, I'm looking for a product that has X, Y, and C."

Like, this would be too much to put in, like, a search, uh, bar on Amazon, but now I can have a conversation and it gives me several products, and then I can ask follow-up questions and we narrow it down, right? And so I think that that's gonna become the norm where no matter what type, not just, like, retail, uh, sites, but all types of products. When I, you know, worked at Block,

2:02 Speaker 0: this was-- We, we built agents for, um, financial services. So, like, Cash App is a peer-to-peer, uh, money thing, right? Uh, money exchange. So that one has an agent that we built to help you with your finances. Um, for Square, this is for, like, shop owners. There, we built agents that can act as, like, their, um, CFO and, like, help them understand their business better, right? So I, I think that that's gonna be an expectation that a lot of companies will have, is that I'm looking for developers who know how to build agentic systems, not just use a coding agent. So that's what chapter two is all about, looking under the hood of an agent.

How do you build one? What do we need to think about as engineers? And for those of you who have already started the lessons this week, um, hopefully you've seen that there's still a lot of work for us. Um, I kind of smile a bit when I hear developers say, like, "Oh my God,

3:09 Speaker 0: they've taken all the coding away from us. What's next for us? Are we gonna still have a role?" And I think in this lesson you see just how much engineering work and consideration is still left for us to do. So I still feel like I'm in a flow state. Um, I still make a lot of engineering, uh, decisions. And so, like, the coding part, I don't necessarily miss, and I actually loved to code. So, uh, it's, it's funny. But anyway, I will get into the questions. Um, feel free, as I'm answering the questions, if you have, like, a follow-up or a point to make, um, definitely put it in the chat. Uh, I think that all of us are learning this brand new world together, so you might even have a perspective that I don't.

Feel free to share that with the class. All right. So I did answer quite a bit of these already. I'll go to some of the ones... Actually, I, I answered this one, but I wanted to show this one, um, and talk about it a little bit, because I think, one, it was, like, the most upvoted one, uh, but also it's a really great question. So someone is asking, like, and I think this person kinda lives in the future. Uh, they are working as a consultant and they have to go and, like, build agents at companies. And so they run into a lot of the issues that most people run into who are at this stage, right?

And so they're saying things like, "Okay, um, if I wanna build, like, a real agent, how do I, like, give it database, database access? How, how do I get, like, IT and these other units on board? They, they are not as AI-pilled. They don't know what this stuff is," right? Um, and then the exotic behavior of, like, everybody is using some different flavor. They're using different models, they're d- using different harnesses and stuff like that. So, um, I led AI enablement at Block for the entire company and then went really deep with the engineering org, so I understand these pain points, uh, quite a bit.

5:25 Speaker 0: And, um, one-- I'll tell you about my experience. Like, one was that

5:32 Speaker 0: just being an engineering executive alone was not enough to move the needle. Like, I had to get a lot of other departments on board. Um, finance for one, 'cause we need to buy stuff and we need to buy stuff quickly, right? So it's, uh, like, if I need to buy a new subscription to an AI tool because that is now the hottest tool out- I can't go through this three-month procurement process, right? And so, um, that was one. Also, like, the IT or, um, developer experience departments, I needed to get them on board so that they can also help us move fast. So if we needed to build up new things that we needed to roll out to all the engineers, I needed them right there.

And then security. So I basically set security, like, pretty much next to me, um, and we work remote, so not physically, but you know. I'm like, "Listen, watch my every move. [chuckles] I'm gonna move fast and, uh, pull my coattail if we're going in the wrong direction." You know? That kind of thing. And so I think having, like, those trusted sources that are also dedicated to this initiative is super helpful. Um, and then as far as exotic flavors, I think it's, like, really too early, or it definitely was then. Maybe it's settled a little bit, but it was too early to, like, bet on one specific tool. Like, say, "Okay, this is a Claude Code shop," or, "This is a Codex shop," or whatever, you know?

Um,

7:03 Speaker 0: so I pretty much let folks have

7:07 Speaker 0: anything that they wanted, um, essentially, so that we can try them all out and see what works well. And I think there are comp- lots of companies that are kind of taking that same approach, because developers don't like their tools, uh, dictated to them, you know? Um, but on the flip side of that,

7:27 Speaker 0: that means you have to kind of manage the artifacts that go with all of these tools. So things like, if we wanna have skills for the team, if we wanna have MCP servers that the team can share, you know, um, all-- when we needed to roll out hooks to say, like, "Okay, before code gets committed, then we want these things to run, um, in advance," and we put that on everybody's machine. That means, like, whatever flavor agent all of these developers had,

7:56 Speaker 0: we, we needed to make sure it supported these things. So now,

8:01 Speaker 0: I encourage teams to opt into agents or harnesses that respect, um, open standards so that it's easier to just kind of build a plug-in, and you can share this amongst the different types of harnesses. So, just wanted to touch on that briefly. Um, what is this? Top, top eight questions? Okay, thank you. [laughs] All right. Um, oh, okay. A follow-up to that question.

8:33 Speaker 0: Angie, has your state on that question made absolute sense? Okay. The leadership has been really helpful. Didn't think about it early on. Every onboarding now includes the leadership question. Great. Perfect. Okay. Um, I'm going to, I guess, go to the next question.

8:52 Speaker 0: Uh...

8:55 Speaker 0: Oh, these are lots. Sort and view.

8:59 Speaker 0: How do I do this? I don't know how to work this one.

9:02 Speaker 1: I think Taras here in the chat, uh, did provide great help to sort by the most upvotes. Uh-

9:11 Speaker 0: Okay

9:11 Speaker 1: so, mm, unfortunately, Discord does not allow-

9:15 Speaker 0: Ah

9:15 Speaker 1: you know, sorting by itself.

9:16 Speaker 0: [laughs] When I clicked it-

9:19 Speaker 1: Yeah

9:19 Speaker 0: ...

9:20 Speaker 0: um, it just brought me to-- It didn't bring me to the Slack, to the, the actual thread. It brought me to the whole thing. I'll try one more to see if that helps.

9:34 Speaker 0: No, it's not helping. Sorry. So sorry, y'all. Okay. Um, so next question is, I'm starting to work on a-- Did I read this one already?

9:44 Speaker 0: Um, work on agents where output accuracy needs to be as deterministic as possible for the user to complete their work. How can I build a strong suite of evals to meet this need? Should some tool responses be provided straight to the user without going through the agent? Okay, great question. And I think one of the main themes that you'll see throughout week two is that we still need a lot of deterministic to- code. I think that it is, um,

10:16 Speaker 0: very short-sighted. Like, we got this shiny new tool, which is an LLM, and, you know, it can figure out how to do things. We put it inside of an agent, and then it can actually do the things. And I think that

10:31 Speaker 0: people then became hands-off, and it's like, "Oh, I can just, like, give this agent work to do." Um, and that is not [chuckles] real life. Um, you'll, like, quickly get burned by that, and you-- When people are using this stuff, like, for real and not just, like, on a side project, you quickly start noticing, like, "Oh, I can't just let this agent run wild in my shop," right? You have to put some-- a lot of things around it, and that's what we do in this course. So we have a model. We build an agent. We go through several scenarios to determine what does this agent need to behave in a reliable way. Um, and so I think that evals are definitely one part of that, but so is the, like, deterministic code, just in and of itself.

So throughout the, the week, we built a lot of deterministic code, and I talked through, through those decisions in the course to say, "Okay, should we use a model here, or should we just write code for this?" Anytime you can just write code to do it, you should just write code to do it. It's cheaper, it's faster, and it's much more reliable. You should only be pulling for an LLM when you need some sort of, like, judgment or, um, reasoning in the process, right? Um, otherwise, just use code. And then evals are super important.

12:05 Speaker 0: Um, we talk about that in the lesson that hasn't dropped yet. It's the last lesson of the course. But I'll go through evals, uh, quite a bit. They are super important. Um, they help us make decisions. Uh, you might even see that in today's lesson where, uh, I needed to make a call on, okay, do I, do I use this design or this design for this problem I need to solve? And evals help me answer those sort of things, 'cause I can try out... Like, code is pretty much free, right? So I can have the agent try it this way and try it that way, and then let's run a bunch of evals a bunch of times, um, so that we can determine which one is actually...

which one performs better, you know? Um, so yeah.

12:52 Speaker 0: Okay. Um, next one is, how do you stop a model from ignoring instructions in a complex prompt?

13:00 Speaker 0: So this one says that, uh, our system prompt isn't one block of text. They are assembling lots of partial texts, some rules, some examples, stuff like that, and putting all of that into the system prompt. Um, but then the prompt ignores it. I'm not, I'm not surprised that some of this gets lost. So, what I stressed in the first lesson was that the system prompt is submitted on every turn. So what you have in there should be, like, something that is of the utmost importance, but it should also, uh, be as concise as you can make it, right? Um, and a lot of times when people are building now, they get the agent to help them.

And so your agent might even build your system prompt for your new agent that you're trying to develop. And these things are verbose. This is what they do for a living. They generate text, right? Um, and some of that is much too much. And I even saw as I was developing this course, like,

14:08 Speaker 0: if I would ask my coding agent to, like, add a feature to another agent, it would automatically want to tack that onto the system instructions. And so I would have to interject

14:21 Speaker 0: more than once to say, like, "What? What are you doing? That does not go in the system instructions." So you need to be really mindful about what goes in the system instruction. It should be something that you need the agent to, uh, know every time that it's called, the model. You need the model to know every time that it's called. Um, and a lot of that is not relevant. That's why it ends up getting ignored. And so,

14:45 Speaker 0: the better approach to this is to move that stuff out into other files that you can bring into context when needed. So things like skills, which we talk about in, um, today's lesson, or, um,

15:00 Speaker 0: uh, any other files that you can just kind of bring in when the time calls for it, right? Versus putting everything in your system prompt. If it is too complex, things get missed. Sometimes by the time you add another file in or you add another skill in or the user says something, you even have cases where the instructions are contradicting each other. And so, of course, the model is confused, and it looks like it ignored something, but it probably just chose, uh, a conflicting thing instead. So, that's my advice. Don't stuff the system prompt. Be very, very intentional about what goes in there.

15:42 Speaker 0: All right. So working with AI agents in orchestration in bigger teams and corporations. Okay. So this is a great question about, like, enterprise pay-as-you-go pricing models and stuff like that, where token maxing is not an option. Okay. This is a sore subject for me because, um, one, like, I see people who work at Frontier Labs,

16:09 Speaker 0: and they have a different perspective than folks who don't work at Frontier Labs, who work at startups, or even if you work in a big enterprise setting. So at one point, myself included, um, when I worked at, you know, the enterprise company and we have 3,500 engineers, uh, and we're just now starting to use AI agents, the directive was, "Oh, use as much as possible," right? Um, "We want to see you all token maxing," and people were throwing up leaderboards and all of this ridiculous stuff. And then the time came when the bill was due, [chuckles] and executives realized just how expensive all of this stuff was, and so then you kind of got back to reality.

But for a lot of AI influencers or YouTubers or people who are on Twitter or who work at the Frontier Labs, who you look at and you go, "Okay, they're the ones building this stuff. I should listen to their advice," what I found is that their perspective is skewed by their, their situation. They don't have to worry about paying for tokens, right? Which is not the case for the rest of us. And so, um, what I actually see folks doing at companies, like, so startups are buying things like, they might have, like, a enterprise package that is custom to how many seats they have, um, but then they also might allow for, uh, individuals to also get a pro plan on top of that, right?

So stuff like that. Um, I've also seen bigger companies start doing a lot of kind of routing in the background. So they'll put a gateway, um, on this, and based on the request, they have some logic that routes it to maybe a cheaper model, 'cause your developers are gonna turn it up to max thinking, um, you want the best performance, and so you think that that's the best thing to do when you might not need that most expensive model. So, um, that, that's how I see people approaching this.

18:24 Speaker 2: What?

18:28 Speaker 0: Okay. Um,

18:32 Speaker 0: what is your vision for agentic systems in the coming years? So Claude published a post about sub-agents, um, will be more like skills and tools.

18:47 Speaker 0: Yes. So let me tell you, I, um, I'm a maintainer for Goose. Goose was one of the OG AI agents. It came out like even before MCP existed. And so,

19:03 Speaker 0: we

19:05 Speaker 0: introduced sub-agents very early

19:09 Speaker 0: in the process, like, uh, sometime 2025, like very early 2025,

19:15 Speaker 0: before the other harnesses had them. And I'll tell you, at that time, you needed something like, like sub-agents because we didn't have the big context windows that we have now. We had like 100, uh, [chuckles] those context windows were like not one meal, they were like 100K or something very small like that. And so sub-agents, what they do, they operate very much so like

19:42 Speaker 0: functions, if you think about it architecturally, where you can just kinda delegate a task off or call a sub-agent/function with some inputs. You don't have to take the context of everything that's going on. You can just say, "Hey, here are some inputs. Go execute this thing and give me the result." And then your main agent takes that and does what it was, does what it wants with that. So in that context, you, you needed sub-agents. Now that the models are smarter and the context windows are bigger, you don't need sub-agents as much. I won't say that they, like there's no need for sub-agents anymore. I think there's still cases where you need sub-agents, but what, uh, the Anthropic folks are saying is, "Hey, why don't you look at a lot of the systems that you've built with sub-agents and look at it with the, the models that you're using today.

Let's run some evals," um, which are again very important, "to see if we just made those sub-agents, uh, like we made some skills for them, uh, and we called on those skills when needed, would your main agent perform better or worse?" Right? And the evals will tell you if, if that's so, which, which, which way does it go. And so, um, I think that this is something we'll have to do for a while as this technology advances. Um, again,

21:22 Speaker 0: they-- Anthropic was the ones telling everybody to build sub-agents, um, and their tools provided for that. And then as we learn more and the, the technology progresses, we say, "Hmm, we might not need that complex architecture anymore, and we might be able to simplify it." So again, sub-agents came before skills even existed and, and so now that we have skills, yeah, some of that we can simplify.

21:49 Speaker 0: Okay.

21:50 Speaker 0: So building agents is like [chuckles] uh, positive parenting.

21:58 Speaker 0: Um, enabling, giving them means to explore their environment, self-improve, and we set up some guardrails and stuff, but we avoid rigid instructions so that they can be creative. Um, I would say that is a good analogy. Um, yes, I think you, you need to let the models cook. Um, [chuckles] let them be free enough to figure things out in the best way possible. However, don't let them YOLO in your enterprise setting, right? And so, yes, you put up guardrails, um, you provide guidance on what does good look like, um, and then you kind of let them cook within those constraints.

22:47 Speaker 0: All right. So how do we avoid becoming a QA engineer for our agents?

22:56 Speaker 0: And this one is specifically about, looks like they wanna build a mobile app testing agent.

23:06 Speaker 0: So what tools, workflows, agent architectures would you recommend so that we don't gotta, like, babysit the thing?

23:14 Speaker 0: Um,

23:17 Speaker 0: so testing is really interesting with models. They are not that great. So I have like a background in software testing. [chuckles] Um, so I know what good testing looks like, and the models are not that great at it. So if you're building an agent for that, I mean, it can work, but you're gonna have to put in

23:39 Speaker 0: a lot of work to make, to teach it, like how it should do good testing, um, what it should be looking for, and even the persona. Um, if you think about yourself as a developer, when you're building code, I know for me at least, when I'm building code, I'm not as great of a tester of that code because it's two different mindsets. One is trying to create something and one is trying to make it fail. And when you test your own stuff as a developer, what you're doing is trying to prove that it works, versus as a tester, you're trying to prove that it doesn't work kind of thing. Um, so it's just, it's just a different mindset.

So you're gonna have to do the work to build that agent to be what you want it to be. I would not rely on the model to just kind of get this right out of the box.

24:32 Speaker 0: All right. Um, is JEV production ready?

24:38 Speaker 0: When is it reliable enough for production? So JEV, we used JEV in, um, I think lesson two. And yes, this just came on the scene. Um,

24:53 Speaker 0: Typesafe says that JEV is production ready. I'll tell you in

24:59 Speaker 0: one of the videos, I think it's in tomorrow's lesson, um, I actually had a moment where I'm like, "Huh, should I use

25:10 Speaker 0: JEV for this?" And I talk about it in the course, and I then actually ran evals. Um, [laughs] evals tell you the truth. So my hunch told me that JEV was the right thing for this particular task, but when I ran the evals, JEV was actually not the best for this particular task. Does that mean that JEV is not production ready? I don't know. Uh, I think it is production ready, but just like we saw with LLMs, they were production ready two, three years ago. Were they as good as they are now? No, of course not, right? And so hopefully we'll see these decision models get better as well. But whether that happens or not, I still think that as an engineer, before you reach for this, you should try eval.

So test it out with JEV, whatever your situation is. Test it with JEV, test it with an LLM, um, test it, I don't know, with like deterministic code if you can get away with that, or a skill or whatever it is that you are considering for the job. Test it out with evals and see which one performs better.

26:25 Speaker 0: All right. How do you handle long-running background tasks that are launched by an agent?

26:31 Speaker 0: So it says like a background task, it might need extra user input and stuff like that. How do you handle this? So there, there are some approaches to this. Um, for example,

26:44 Speaker 0: if you use like any of the popular harnesses, you'll see where they, um, might ask you a question or something like that. There's tools that they use to be able to do that. And then for the background tasks, a lot of the, um, whether you're using a harness or you're building your own agent, so like Mastra, for example, um, and by the way, I have no affiliation with Mastra at all, uh, but I, I did use it for this course because it was open source and it was pretty... It had a lot of the, um, features that we needed to build agents. But, um,

27:24 Speaker 0: they have a lot of this stuff kind of out of the box for long-running tasks and things like that. And so, um, yeah, I think you can just kind of get away with some of the built-in stuff. I wouldn't try to roll that myself.

27:40 Speaker 0: Okay. How do you handle dynamic multi-step workflows in an agent?

27:46 Speaker 0: So,

27:47 Speaker 0: oh, let me take a step back. Someone asked, "Can I go into more detail about Mastra? I'm still really confused.

27:56 Speaker 0: I had no idea what I was doing." Okay. Um, yeah. So Mastra is a, a framework for building agents, and there's several of them out there, open source as well as commercial. So like LangChain has one, um, OpenAI has, has like an agent SDK, Anthropic does, you know. Um, and all of these folks kind of know what I know, is that like pretty soon you're gonna have to build these agents yourself. Like a lot of people are already building these agents themselves. Um, and again, this is not just like, "Oh, let me build an agent for a fun project at home," but like enterprise businesses, they, they want agentic software. And so Mastra is a framework, one of many, that allows you to build that.

So it'll include like a lot of the functionality that you need for agents, things like memory or, uh, workflows or, um, like easier ways to orchestrate things or like these long-running tasks, you know what I mean? And so a lot of the stuff kind of comes already out of the box, and you can piece it together to build your agent. So in the course, we see where it's basically just like code that you are constructing so that the agent is an object, and then you add things to it. So you can add like which model it's gonna use, what's, what's the system instructions, um, what tools should it have, should it have any skills, um, those sorts of things you add it to the agent.

So it's just a framework for building an agent yourself.

29:39 Speaker 0: Okay. So, um, dynamic multi-step workflows in an agent. So this was a good example where we could see how complex even a chatbot can get, right? Where the user says, "Find the flights," the agent then proposes some things and asks it to confirm, and then the user says, "Yes, but first do this, and then I need you to confirm this with my wife." And so you're like given a lot of tasks. You might even interrupt the agent and stuff like that. So this is a l- this is the complexity kind of what I'm talking about. So like in Mastra, for example, they have workflows where you can set up like steps in how, in the order that these steps should run in.

Um, but you also need to be flexible enough that when a user interrupts or changes their mind or anything like that, that the agent can handle that as well. And so we do cover that in a couple of the chapters in the course. Not exactly like this, but pretty much all of these pieces where the customer changes their mind mid-flow, where the customer needs to interrupt. And you also see like built-in stuff for, um, some of the harnesses that we use. Like, for example, um, Codex allows for steering, right? Where you can kind of interject, um, and provide more context. And what's happening is that the harness itself will take all of that context, um, and it'll maybe like have a couple of calls to the model and deal with everything and then give the answer back to the user.

So, um, this is another great example of why you don't want to like hand roll this stuff. Sometimes people ask. Um, in fact, this one looks like that kind of question, like, "Why would we use Mastra or similar projects?" You don't want to hand roll a lot of this stuff yourself because it's super complex and- It just becomes like a component that you can just already use. Like, I don't want to think about, um, how do you do compaction well, right? I would rather use Mastra to say compact. Or, um, I don't wanna solve agentic memory from ground level, [chuckles] you know? Um, so they have a memory system, and so you can just kinda turn these pieces on.

You know that these are the components of an agent. They give you the building blocks, and then you kinda add whatever you need to it and customize, but you don't have to roll the orchestration yourself. You know what I mean? So that, that's why you would wanna use one of these types of tools versus, like, trying to build and maintain this stuff yourself. It is not a good time. Um, okay. So writing good evals. Um, in case you all have not noticed, I am a big fan of evals. [chuckles] Um, so they list a course here.

32:41 Speaker 0: That's a good idea. I think, like, eval is similar to testing.

32:46 Speaker 0: Nobody, like, wants to do this. Like, [laughs] you wanna build stuff. And, and so now people go, "Oh, I can just delegate that to the LLM." The LLM is not

33:00 Speaker 0: good at tests, and it's definitely not gonna be great at writing, like, all of your evals as well. I'm so sorry. You need to be a part of this, or I don't know, you all can hire. That's what they did for software testing. People hired dedicated testers who actually like doing this shit, you know, um, and who are good at it. So you could do the same thing for evals, like hire professional, uh, eval writers who actually, uh, think through this stuff and build great evals for your agents. But if you do wanna do it yourself, which you don't, that's what this person says, "I don't actually wanna do it, but I know I should."

Yeah, you're gonna need to probably

33:45 Speaker 0: do some training, just like you're doing training for working with agents, building agents. You also need to understand how to evaluate agents. Um, and it's not as simple as it sounds. Like, we get into it in the final video of the course, but I'll go through, like, uh, like, it working one time, for example, like a deterministic test, that's not a thing. Um, so you have to kinda run evals a bunch of times and then kinda monitor the results and take the average of that and all of that. Like, it's, it's, it's, it's fun for some people. For most people, it's not. Um,

34:23 Speaker 0: I can't give you, like, good tips on this, like, right in this thing, but I think, like, a course is a great idea, um, for, for learning how to do it effectively.

34:36 Speaker 0: Okay. Oh, another part of this is how do you reverse engineer why it's degraded? So the other important thing about evals, um, which we will get into on the last day of the course, is then taking the information that comes from the eval and then determining what to do with that information. So they are going to... It might not be your eval that degraded, but, like, this is failing all the time because something in your agent build has changed, right? That could be the model has changed. That could be, um, the skills have changed or whatever. It could be a combination of things. So as models, um,

35:19 Speaker 0: get better, sometimes you'll see, like, "Oh, I don't even need, like, half of this stuff anymore," or, "This stuff is actually, like, all of these skills or all of this information in my system prompt or all of this information in my agents and D is actually degrading performance because the model has gotten better." And so the eval is telling you something, and then you go and tweak. So that's what the final lesson is about in week two, is like, okay, we're gonna run these evals, we're gonna get the insight, and then we're gonna tweak stuff to, like, fine-tune our agent. And that's just a part of how this is gonna have to work. Um, [laughs] sorry.

35:57 Speaker 0: Okay. Um, how do you handle clarifying questions in the user's replies to them in an agent?

36:07 Speaker 0: Okay. Um, so there's actually

36:11 Speaker 0: techniques to this. I think I talked about it a little while ago about, um,

36:17 Speaker 0: the tool. So there's a tool that you can add. Like, Codex has this tool. Uh, I think Claude had it first, where it's requesting input from the user. So the way this works is, like,

36:33 Speaker 0: the

36:36 Speaker 0: agent would, like, send a request to the model, and the model then would say, "Oh, I have so many questions about..." You know? And it would send all of those questions back, and the harness would prevent that, uh, present that to the user. It could be, like, five questions. I'll give you an example. Yesterday, I wanted to order my sister some flowers, and so I told my agent, like, "Order my sister some flowers. Here's her address." And then it sent that request. It started trying to do it. Once it was on the page to order the flowers, it saw a bunch of fields that I had not given it, like what phone number and all of these other things.

So it had, like, maybe three or four questions that it needed to ask me, which probably the model then, like, sent all of those questions to the agent. So instead of presenting that to me in text and then making me answer them, uh, one by one, which I probably wouldn't do, I might skip one or whatever, um, the harnesses now are using, like, a tool to kinda manage that. And so they'll present to me a little dialogue with a little answer box in it, and then I can, um, answer-

37:44 Speaker 0: one by one. And so, that's a way that we see, like, some of the frontier harnesses handling it. I think that's a good way to do it. I hope it gets better than that, um, but that's what we're doing for now.

37:59 Speaker 0: Okay. All right. I think I have gone through all the questions in Discord. Let's see if there are questions in the chat. [chuckles] Hey, Chris. [chuckles] Y'all are very funny.

38:29 Speaker 0: Um,

38:32 Speaker 0: if any- if anybody sees questions...

38:35 Speaker 0: Wojtek, do you see any questions?

38:39 Speaker 1: I believe there are still some unanswered, uh, questions on-

38:42 Speaker 0: Thoughts on Eve by herself. Okay, I have not tried Eve. I've read about Eve.

38:50 Speaker 0: And the whole... So Eve is, uh, their, their marketing of that is an agent is just a folder.

38:59 Speaker 0: Um, which, I mean, honestly, when I'm not building, like, production agents and I'm just building, like, some sort of thing to help me out with, you know, my day-to-day stuff, it does very much so look like just a folder. Like, I have a knowledge base, I have some markdown files, I have, like, some skills, and all of that just kind of exists in a folder. So, um, on that, I would say they, they might be onto something, and it could be a much simpler way to think about building agents. But I've not tried Eve at all, so I don't wanna say much more.

39:39 Speaker 0: Um, I asked a question on Discord,

39:45 Speaker 0: but

39:47 Speaker 0: do I need to refresh? Three new...

39:52 Speaker 0: Ah, okay. Thank you. Okay. So, how could I extract my own expertise and then help me skill up blazingly fast? [laughs] Uh, second brain is great, but the expertise extraction is hard. Cognitive fatigue, what to learn. I try to find a path that I can practice.

40:14 Speaker 0: I have not found, like, um,

40:18 Speaker 0: a shortcut to, like, doing this stuff blazingly fast. Um, it always just kind of backfires, and it works for a little bit, and then it doesn't, right? Um, the second brain stuff, like, there's a lot of

40:32 Speaker 0: folks thinking about how do you extract this stuff automatically. Like, oh, have your agent go to meetings with you and just kind of note down everything that's happening, and, um, you know, it's in your Slack, it's connected to your email, and stuff like that. I still have not seen that work very well. It works okay, you know what I mean? But it's not, like, amazing. So, I don't know. I think there's more work to do there.

41:02 Speaker 0: Um, okay. Why do enterprises build an A- AI gateway? Oh, great question. Okay.

41:11 Speaker 0: So, what problems does it solve beyond rate limiting and quotas? So, gateways are,

41:19 Speaker 0: in my opinion, a must-have if you are working in an enterprise setting. And the reason that companies are doing this is because, again, they have multimillion, multibillion-dollar businesses. You don't just let an LLM have its way, it's connected to all your systems, and just do whatever it wants, right? So, what a gateway does is it provides, like, a lot of guardrails where you can do things like, um,

41:48 Speaker 0: you can see every,

41:51 Speaker 0: uh, MCP call that is about to be made. You can then also compare those tool calls with, like, your own company policies, for example, to see, like, is this agent about to do something that we don't want it to do? And then you can prevent that from happening before the model does it, right? So, instead of, like, not letting the model cook and be creative, you let it cook and be creative, but before it gets to actually execute the tool, you have another layer that's checking that to make sure that's right. So, that's one reason. The other thing is, like, the, um, the financial part of that. And so, you can even set, like, limits on how much, you know, people can spend on whatever types of tasks and things like that.

So, that helps too. Like, we had a case one time, like, I don't know, this dude, like, had his agent running overnight, and it went into this loop or something, and we're like, "Why is the bill, like, skyrocketing?" We pointed back to this one guy who forgot this thing was running or something like that. So, you know, having a gateway helps you prevent, like, those sorts of things from happening.

43:08 Speaker 0: So, we see, like, a lot of enterprise companies adopt this now.

43:13 Speaker 0: Okay. Um, we need a new QA and testing paradigm, no? So, Cypress, Playwright, etc., cool. I need to be able to make sure a change is also UI/UX and that my proof of concept can run in isolation when I'm stress-proofing them. What's your thought about that? I think we definitely need new QA and test paradigms. I'm actually, um, about to work on a talk that I gotta give [chuckles] next week when I say, um, I talk about, like, okay, you're-

43:49 Speaker 0: Your, I think it's called, like, your next customer m- might be an agent or something like that.

43:56 Speaker 0: But, um,

43:58 Speaker 0: we have to think about a lot of things differently. Like,

44:02 Speaker 0: before, you could even do a customer journey and you could test that, right? And you say, "Okay, the customer is gonna click here and there and there, and then this should be what happens and this should be the result." Like, all of that is out of the window now, right? Like, agents find ways to use these tools that blow our mind. I don't know if you all ever use computer use, but you can ask the agent to do something and, and I sometimes just watch in awe, like, "Dang, they're using that tool better than me." They're, like, finding all of these, like, better ways to get to it. Even Bash, right? Like, you see them piping together these commands.

You're like, "Dang, I didn't even know you could do that." And so, the journey is gonna look different. Um, now I don't think that we as humans can even anticipate what the journey, what every journey looks like. Um, and so now the test comes for, the test becomes for, like,

44:59 Speaker 0: did the goal get accomplished? I don't really necessarily care how the user, its agent, or whoever got there. Yes, I wanna know if there's friction in getting there, but ultimately, are they able to accomplish their goal? And so that makes testing look a lot different than a Cypress-based test that you would have written before, a Playwright-based test, you know? And then we also have this whole concept of generative UI that, um,

45:30 Speaker 0: might be coming soon. So this is an emerging trend where you say, like, "You know what? Maybe I don't have to build my application in one way and make every user use it that way." You know? What if I had my UI generate on the fly based on maybe who the user is or what their goal is? That looks totally different as well, right? And so, how do you test for that is a really interesting question. So yeah, I do think,

46:03 Speaker 0: uh, a lot of this will, will change, um, and what it looks like to do it.

46:11 Speaker 0: Okay.

46:13 Speaker 0: All right. I think I answered everything. Do you share... Oh, a spicy one. Do you share Jack Dorsey's view as stated in that famous "From Hierarchy to Intelligence" article?

46:28 Speaker 0: Um, I don't even remember what he said in that article. That was, like, in April, right? So, like, two centuries ago. Um, somebody remind me in the chat [chuckles] what the, what the premise was. Um,

46:49 Speaker 0: how to manage spec files across environments.

46:54 Speaker 0: So when the behavior is driven by big spec file... You know what?

47:01 Speaker 0: Yeah. Um, I think, I think... Okay, this is a spicy take, and I could be wrong, so don't, like, take this to the bank. But I think the days of, like, those big spec files is gonna come to an end pretty soon. With some of the other more, um...

47:23 Speaker 0: The, some of the other technologies, like skills and things like that, that you can basically, the model can bring in when it needs it, and it gives it guidelines on what's needed, um, I think that those will become more prevalent. I also wouldn't be surprised if we see some other newer, more modern techniques for how to give an agent the information that it needs. So, like, I think the days of the big spec files and then trying to maintain them across environments and stuff like that is, is gonna go away.

48:00 Speaker 0: Okay. Um, can you elaborate on what a forward deployed engineer, what's their roles, responsibilities, a day in the life? So this is a, a relatively new concept, but also not. [laughs] Um, it's just kind of related to AI, right? And so, I think of it as... I'll tell you. I had, um, when I was doing AI enablement, I didn't do it by myself. I had, uh, about 50

48:30 Speaker 0: engineers that I recruited from within the company. And these engineers, we would all learn together, like, what is new, what is hot. We would try a whole bunch of things out. We would figure out how to do things, each of them for their team, and then they would go back and teach their team. They would deploy certain things that's needed for that team in their code base or whatever to make, like, using agents easier. Um, so a forward deploy engineer is essentially like an engineer who knows how to use all of this stuff, like, really effectively, and they could then go into companies or something, um, almost like, yeah, like a solution architect basically.

They could go into the company and teach them

49:21 Speaker 0: primarily, like, how to use whatever company's tools that's funding this thing, right? And so, if you were a Claude forward deployed engineer, I don't even know where they got that term from. It just doesn't even have a ring to it. But [laughs] um, if you, if you were Anthropic, you might send a FDE into, uh, one of your customer's shop, and then that FDE would work with their engineers to show them, like... All of the cool shit that they could be doing with Claude that they just didn't know, right? And that they didn't have time for. The difference from, like, just being, I guess, like a sales pitch or something like that is that the FDE could-- will really, like, kind of get in the weeds with that team and understand what their processes are and what they're doing, and then show them how to integrate the product versus, you know, a kind of hands-off pitch.

50:19 Speaker 0: Okay. Um, "Identify agents and sub-agents from a use case." So our team uses Java, hey, shout out to Java, um, and Kafka for backend event processing where actors already have defined responsibilities and communicate through messages. When looking at a business use case, what practical signs tell us that it would benefit from an AI agent? So I think, again,

50:49 Speaker 0: when people say, "Oh, we just need to, like, build agents," question that. Um, maybe you do, maybe you don't. And just because this is, like, trendy doesn't mean everything should be agentic. In fact, you would do yourself a huge favor to just build deterministic code where you need to. Um, I ask myself, does this require reasoning or does this require, um,

51:19 Speaker 0: basically reasoning? That's like my number one question. Uh,

51:24 Speaker 0: if it requires some reasoning that I just can't, like, build into the deterministic code itself, then that's when I pull for an LLM, um, and add that to the deterministic code, right? Um, and this can be an LLM API call, which is really cheap, much cheaper than, like, building a whole agent to do something. So that would be my advice there. Um, once you identify that agent, how do we recommend smaller tasks and sub-agents? So in tomorrow's lesson, or is it today?

52:00 Speaker 0: What's today's lesson?

52:03 Speaker 0: Is today's lesson about, um, multi-agent? I think that's tomorrow.

52:11 Speaker 0: Whatever lesson is about multi-agent stuff, we get into... Okay, tomorrow. All right. So we get into, uh, sub-agents, and I spend quite [chuckles] I spend quite a bit of time on how would you identify if you need a sub-agent or not. So, um, definitely check that out. Um,

52:36 Speaker 0: but to the point earlier, the question where it's like, oh, Anthropic is basically saying maybe try not to use a sub-agent. So what you should do, and I actually had this moment with the sub-agents. I was like, "Oh, a sub-agent." And then I thought, "Huh, maybe I don't need a sub-agent. Maybe I can just do this." And I wasn't quite sure, and so I ran evals to figure out which was the better approach. But, um, I go through a whole list of, okay, if it's this, this, this, this, and this, this might be a good task to delegate to a sub-agent.

53:13 Speaker 0: All right. "Every rule I add kills a little creativity in my agent. How do you balance this?" [chuckles] Um, this is true, but you need to kill creativity in your agent. Um,

53:32 Speaker 0: if you are building, like, production software that you want to be reliable, creativity is not a, a huge ingredient [chuckles] for such a system. Um, and so

53:46 Speaker 0: you need to figure out, like, where you let the models, um, have some freedom, but again, where you put up some guardrails. So it's a lot that doesn't need creativity. You just put that in deterministic code. Um, things that, okay, I do need the agent to intervene here, but I also don't want to get myself in trouble. That's where you have the gateways on the other end of that to kind of block anything that you-- that would cause some trouble. So,

54:19 Speaker 0: um, yeah, I think that's, that's a good approach. [chuckles] Uh, "Non-chat agents. Have you come across any notable examples of agents designed for communication channels other than chat? Please share your thoughts on A2A."

54:37 Speaker 0: Okay, this is great. Um, we get into A2A tomorrow too. Tomorrow is gonna be fun. So we get into A2A. I go through all of that about, like, when you might want to delegate. I'll give you a heads-up. So A2A is agent-to-agent protocol, and this is a way to, um, basically call another agent. So that's usually an external agent.

55:01 Speaker 0: You get into this when... And these are, like, non-chat agents, right?

55:06 Speaker 0: I think this is also something from the future too. So Google developed this protocol, and I think they're kind of ahead of the game with it. So I actually went to China, um, maybe like two months ago and met with a lot of companies there, and they actually use A2A pretty heavily because they build, like, these super apps, like WeChat and stuff like that, like, and Ali- Alibaba stuff. Like, all these companies build their own whole thing. And so they actually didn't have a big need for using external tools, like from MCP, but they had a bigger need to use external agents that are from a whole nother platform, right? Because everything is so, like, this.

Um, so the use case becomes, hmm, I need my agent to interact with this other thing that I do not necessarily have access to myself, right? There's no MCP server or anything that I can call, or I don't have the access rights to get to that certain data. They're not exposing that to me. But what they will do is expose an agent as a service. And so then I call that service, which is different than an API call, right? It's not like you actually needed to go and do something. You needed to have its own tools, its own access, its own logic and everything, and it can handle that in its trusted environment, and basically you delegate work to it.

So it's not just like asking a question, but it's like, "Hey, I need that agent over there to go do a task because I cannot do it." Me, this agent. I cannot do it. So your agent would find another agent, would discover like these other agents that companies have exposed as a service. Um, again, this is why you all need to learn how to build agents, 'cause people, companies are gonna wanna, like, build agents as services. Um, and this agent will discover, "Hmm, let me find an agent that can get this done because I can't do it." And it'll look for agents and say, "Ah, there's an agent. It exposes these capabilities. It has this access.

Just what I need. Let me delegate a task to this." So kinda like a sub-agent, but, um, you don't really have control of it. It's like a whole nother agent from another business that might be built in a different framework. It's like a black box, essentially.

57:41 Speaker 0: Okay. Oh, how do you judge whether adoption is helping engineering teams? Thank you for asking this. So when I led AI enablement,

57:50 Speaker 0: I started doing it like early 2025, three centuries ago. Um,

57:58 Speaker 0: and adoption was the metric to use then because people hadn't adopted it. Um, as people adopted it, as we got to like the 95%, uh, adoption rate, this became frivolous to me to continue monitoring, especially because we weren't seeing like developer velocity increase, so we weren't like shipping to customers faster. We weren't, we weren't solving problems faster. But hey, everybody's using AI, right? Um, so that no longer became helpful. Um, that was table stakes. Okay, people are using AI, whatever. Who cares? Um, other than they are running the bill up. And so I had to go much deeper within engineering to figure out how do we use AI effectively. So not just use AI, but how do we use it to move the needle?

How do we use it such that we can delegate work to it? We got to the point where agents then became a part of our sprints, where it's picking up tickets. So the first time we did that, the-- we had, you know, you do your little sprint and you're like, "Okay, here's all the work that we can get done this sprint." It was three-week sprint. With the agent involved, we blew that out in one week. So we had to pull more work in the sprint, and then we got through all of that, and then we had to pull more work again. So now we're able to do three times as much work as we would have done before, and this is real work that's like issues on the board that we have committed to, to solve our customers' problems or build new features or something like that.

That's when it became meaningful, right? So, um, yeah. [chuckles] Okay. Um,

59:50 Speaker 0: how to get people really in on this AI train?

59:55 Speaker 0: Yes. Um,

59:59 Speaker 0: oh, good. I was gonna say, if you haven't watched the video, watch the video. Yeah. So

1:00:04 Speaker 0: trying to get everybody on board is too difficult. I, I, I don't recommend focusing there because it's not just a technology problem. This is a change management problem. There's a whole bunch of emotions involved in this. Like, some people hate this shit, right? Some people are like very fearful of it. Some people have ethical concerns. Like, there's a lot. And so just like trying to shove this down everybody's throat is, I don't know, it just didn't work very well. Um, what I did instead, for those who didn't watch the video, is I thought about this, this rule. It's called the 1-9-90 rule, where, uh, if you think about any community, whether that's on social media, anything, any community, there's gonna be 1% that are actually the creators.

The, like, these are the people that are posting all the time on social or... And even at your church, 1% of the people are doing all of the work to keep the lights on, right? And then like 9% of the people, they'll pitch in here and there, like, "Give me my one task for the year," kind of thing. And then 90% of the people are consumers. So yes, they are present, but they're not like the builders. And so I took that model and thought about, okay, if I apply that to AI adoption, what does that look like? And so for 3,500 engineers, I then found the 1%, so like the fi- the 50. Um, and this was very strategic.

It wasn't like, "Oh, who's AI-pilled? Raise your hand." It was like, no, I need like one or two people from

1:01:41 Speaker 0: each of our strategic repos, from, um, you know, different varieties of repos. So iOS or Android. Uh, I need s- I need web. I need some backend folks. Oh, we got a, we got a hardware team. Give me somebody from there. You know what I mean? And so it was this very diverse group where we can represent all areas of the business, and then we went deep on this stuff. And so now-

1:02:12 Speaker 0: We are, um, learning about everything that's new. We're trying every tool that comes out. We see what works across different domains, and there was no clear answer to that. There was no like, "Oh, y'all, we found a solution that works no matter if you're like back end or front end or iOS or Android, just across the board." There was rare things like that. Um, what we found is like most of the stuff, like, okay, this works really good in this context, um, and people would bring that back to their teams and then even embed it. So things like embedding skills into your repo such that

1:02:53 Speaker 0: all your engineers don't need to learn how to be the most effective or the best, like, um, agent user. But when they point their agent at the code base, because you have skills already built in, because you have a really clean agent MD that you're constantly repairing as the one percenter, um,

1:03:18 Speaker 0: because you have these things kind of built into the repo itself, the agent can do a really good job. And so that got people out of the like, "Oh man, I tried it, but it was slop," and this is now a skill issue. You got rid of the skill issue because you actually built skills inside the repo itself, right? Um, so that's my, my, my advice is not to try to

1:03:44 Speaker 0: get everybody AI, on the AI train, but build

1:03:50 Speaker 0: it into the system itself such that people can just kind of do their work and this stuff just

1:03:57 Speaker 0: operates the way that we need it to operate because you built it into the system versus trying to get everybody AI-pilled. Now with this, people also saw their colleagues who were part of this 1% go from, "Oh, wait,

1:04:14 Speaker 0: Adam used to ship like three PRs a week, and now he's doing like 30 PRs a week. What's up with that?" They wanna know, "How are you doing that?" So then it, it went from becoming something that you push on people to now people were pulling and they were asking, "Please teach me, um, [laughs] teach me your, your new skills," or whatever.

1:04:37 Speaker 0: Okay. Um, "Sub-agents or single agent for sequential tasks? Um, Anthropics suggests the opposite. Keeping one agent for sequential tasks generates less cost and time." Uh,

1:04:55 Speaker 0: um, I, I don't know. I, I haven't read this.

1:05:00 Speaker 0: Oh, yeah, I did read this actually.

1:05:02 Speaker 0: So I talked about this with the sub-agent part, um, so I kind of answered this already about you need to essentially do evals, which is what they recommended as well.

1:05:14 Speaker 0: Um, they think, like, for a lot of the things that you needed sub-agents for, you probably don't anymore because of new

1:05:23 Speaker 0: techniques like skills, uh, the models are better now and all of that stuff. So I, I think you should test it out. The other thing that's considered here is what model you're using. So when they give this guidance, they are using Claude Fable. [laughs] They are using like Opus 5.5. Like, they are using the best of the best, where a lot of companies, they might not be a Claude shop or they might be restricted on which models they're even able to use, right? So if you, um, are not using one of those frontier models, this might not look the same for you. Again, that is where evals are super helpful.

1:06:05 Speaker 0: All right. "Are we living in a dawn of specified software engineers?" I think they meant specialized. Um, "What do you think about old specialized tracks as front end, back end?" Yeah. Um, "Can we be agnostic?" Some-- I'm always going to think being T-shaped is good, I think. So having like any sort of expertise, you can-- you-- I'm sure all of you have seen where that has been beneficial to you as you try to use or build agents, right? But,

1:06:41 Speaker 0: so for example, I'm a Java champion, and somebody was teasing me like, "I don't see you talking about Java anymore." And it's like, okay, like what's there to talk about? Um, I don't--

1:06:52 Speaker 0: I no longer think like being a specialist in Java per se, um,

1:06:59 Speaker 0: is something that I need to continue pursuing. And I still love Java. Shout out to anybody using Java. Um, or any language. I feel the sa- exact same thing for any language, right? So I think like, I wouldn't say you, you should shy away from being a specialist, but I wouldn't, at this very moment, I wouldn't invest a ton in becoming like a specialized back end or front end engineer. Like, I-- at this moment, right? Um, I think right now is learning, one, how to work with the agents, how to build agentic software, um, systems thinking, s- systems architecture. Like, those are the skills I would focus on right now if I was like advising someone.

Um,

1:07:48 Speaker 0: and then yeah, like I think whatever you are a beast at, like that'll show up in those areas, but yeah.

1:07:59 Speaker 0: Okay. "How do we manage MCP discovery, access control, and user identity?" Yeah, you want a gateway. [laughs] You want a gateway. Um, MCP discovery, so we had like, we built like 200 MCP servers, um, for our company, and we put them in like, um...

1:08:21 Speaker 0: This was back before, like, things were really taking off, but we put them in, like, a library or whatever. Now you have things like control planes that will expose, like, all of your MCP servers, all of your skills, and stuff like that. And then you also have gateways for, like... I think the control planes also handle, like, access control, user identity, and stuff like that. So I would advise looking into, uh, a really good control plane. I don't have a recommendation of which one, um, but yeah. Check, check, check that out.

1:09:00 Speaker 0: Okay. "Before building a dedicated agent for my website, can I validate the idea in an existing harness?" Okay. "I see many similarities between extending its capabilities and building an agent within an application." Yes, true. Um, "Could I first equip a coding agent with appropriate instructions, then test how it handles the task I eventually want the agent on my website to perform?" Um,

1:09:31 Speaker 0: I guess you could. It's not an approach I would recommend, mostly because if you're using a coding agent, especially one off the shelf, and it's not open source, then you can't really observe a lot of what it's doing internally and how it's managing the task. Whereas if you were building an agent yourself for your website, and you were gonna build that whole thing with, with, like, an agent framework, um, you are exposed to quite a bit more. I don't even think that it's that hard to tr- build it and test it out yourself. I think you would probably spend more time in trying to get your coding agent, um, to pretend to be this other agent that you, you wanna put on your website.

So, I would recommend kind of building out one with an agent framework first, like a simplified version, um, and then testing that out and then enhancing it.

1:10:28 Speaker 0: Okay.

1:10:32 Speaker 0: Somebody said, "Use Vercel." Do you work at Vercel? Are you the same person that said Eve before?

1:10:40 Speaker 0: I gotta try Eve out. I've heard all about it, but I gotta try it.

1:10:46 Speaker 0: Okay, Strands. Yes, another one. Okay. "Are there any agentic platforms for enterprise? What do you recommend?" And then somebody said, um,

1:10:56 Speaker 0: the Strands one. Yeah. So there's, like, yes, there's lots of agentic platforms. So if you're already a coach, uh, a Claude shop, like Anthropic has one. Um, I think it's called, like, Claude Managed Agents, um, which provides, like, a lot of this kind of functionality out of the box. Uh, OpenAI has one. They have their Agents SDK. Google has, uh, an Agents SDK. Uh, everybody has one. And then there's open source ones like Mastra, like LangChain, like, uh, Vercel. Yeah, Vercel's Eve is actually one you could use too. Um, so yeah, there's lots, but if you don't wanna go with an open source one and you wanna use, um, you know, one that you can get more support on, then Anthropic and OpenAI ha- and Google have some as well.

1:11:51 Speaker 0: Okay. "What are the most critical things needed when creating and implementing a strategy for AI enablement? If you were at a company,

1:12:00 Speaker 0: um, with Cursor and Claude seats for all developers, but little guidance beyond use it, where would you start?" So I think this is where my AI enablement story and the AI champions comes to play. So, um, I won't repeat it again, but I, I did talk about it in, um, a talk that I, I gave at AI Engineer, uh, if anybody wants to read that. And then I have also blogged about it on my website.

1:12:31 Speaker 0: Um, I did share the code.

1:12:35 Speaker 0: Uh, it is... If you go to, um, GitHub, and my username is Angie Jones, then you'll see a repo there as well.

1:12:46 Speaker 0: Okay. "How can I learn this stuff on a personal budget?"

1:12:52 Speaker 0: I know, right? It's, it... [laughing] I can't tell y'all how much this annoys me when I see, like,

1:13:01 Speaker 0: the influencers and they're just like, "Oh yeah, do this and do that," and, "Why are you worrying about tokens and all of that?" Uh, 'cause I'm paying for it, um, which is much different when this stuff is not subsidized, right? Um,

1:13:17 Speaker 0: there are, like, free models. They are not as good as the frontier models, but they are getting a lot, lot better. In fact, I think you could probably get away with it. So one of our maintainers, he's a kid, and he actually, um, has no job, so he has no money. Uh, and he uses the, um, he uses open weight models to actually do, like, really impressive stuff. And so I know that they are good enough, but I did give him, like, um, some credits to try out, uh, like, one of the Claude models, and he, he said it was noticeably better. Um, but things like Kimi, um, Qwen is getting really good. So some of these, like, Chinese models, um, are really good if you, if you wanna try that.

So that might be a good way to learn on your personal budget. "How do I manage my mo- my mental context?" Y'all, this almost killed me. So, you know, I'm like, "Oh, okay, yeah, I gotta be running all of these agents at the same time." And it was just doing a number on my... mental. [chuckles] You know, like the context switching was, like, way too much.

1:14:35 Speaker 0: Um, and so I stopped. I've stopped. I, like, a lot of it I kind of have... If there's something that I would need to do periodically or that's conditioned on something, I've set those up to run automatically, um, so that I'm not necessarily managing them, and I check in here and there just to make sure, like, they are still okay and they don't need me for anything. But other than that, I, I no longer, like, have

1:15:03 Speaker 0: five agents running at a given time where I am the person managing these agents. Um, I don't know, they might say I'm not gonna make it. I don't know, but it was, it's much too, it was much too difficult to do.

1:15:19 Speaker 0: Um, "Can JEFF be misguided by a prompt?" So, you mentioned that JEFF is a decision model, but this is still AI making a decision, if I understand correctly. Okay.

1:15:33 Speaker 0: The thing about JEFF, um, so can it be misguided? I guess, right? So with JEFF, you're providing, like, some context and you're asking it a question. So that, that could be a yes or no. That could be, um,

1:15:50 Speaker 0: a list of options to choose from. The way that we use JEFF is to determine which route to take in the code. So if we think about it like that, then yes, this could be harmful damage where maybe JEFF says, "Yes, you should give this person a refund," and then we do, right? So can it be misled? Yes, of course, um, pretty much just like, uh, an LLM can. So you wanna put some safeguards around that.

1:16:22 Speaker 0: Okay.

1:16:25 Speaker 0: Did I answer everyone? Okay, more questions. "In your opinion, what is the most challenging aspect of agent-based AI? What's the part that surprised you or that you warn a newcomer about?" I think the,

1:16:42 Speaker 0: the most challenging part is knowing that just because an agent is capable of doing something, that I should not necessarily just delegate towards it. And so, um,

1:16:55 Speaker 0: the, the

1:16:58 Speaker 0: boundary here is like, where should I use deterministic code? And I'm gonna keep, like, drumming on that, beating that drum, because

1:17:08 Speaker 0: I think people see this, like, magic genie and it's like, "Oh, it can just do it for me," and then they hand it everything to do, and then are surprised or upset when, you know, it doesn't go to plan or the agent, like, goes off the rails or something like that. And when we think about what we're doing, like, this is much more expensive and much more unpredictable than deterministic code, so why would you

1:17:38 Speaker 0: use this tool? And if we think back to pre-GenAI, we all as engineers would always say, "Use the best tool for the job." And everybody forgot that rule [chuckles] when this stuff came out. So I would encourage people to continue using that rule and thinking about, is an LLM the best,

1:18:00 Speaker 0: um, tool for the job? Is an agent the best tool for the job? And then decide accordingly.

1:18:06 Speaker 0: Okay. "How do I handle prompt versioning in my own setup?"

1:18:14 Speaker 0: Um, I don't have versions, but I do have evals on my,

1:18:20 Speaker 0: like, my systems instructions and my skills. Um, and so they basically are tested all the time. So anytime I wanna change anything in my agent,

1:18:32 Speaker 0: then I have these evals run to see if that made things worse or things better. But I don't do versioning.

1:18:42 Speaker 0: Um,

1:18:44 Speaker 0: okay. "Collaborative multiplayer agentic work." This is my favorite topic.

1:18:51 Speaker 0: So, ooh, y'all used to do pair programming mob, and now everybody does their own thing. [chuckles] You know, I started noticing, like, the amount of Slack messages decrease as people started doing, using agents, because they no longer needed to talk with each other much anymore. But what I am starting to see now

1:19:15 Speaker 0: is, like, um, have y'all heard of Buzz? Uh, so it's like a Slack kind of thing where you can bring your agents into the, the Slack workspace as well.

1:19:29 Speaker 0: Um, and y'all can collaborate like that. So when you're asking questions, um, or like you, you're walking through a design, you do it in Slack versus in your harness on your desktop. And that way other people and their agents can kind of chime in there as well. So, um, I think that's an interesting approach. I, I'll be honest, I haven't done much of that. Um, I think everybody has to be on board with that. So I have like my, [chuckles] my dot, uh, which is my, my always on persistent agent from ChatGPT. I have it in Slack, but I don't think people really like my dot talking to them. Um, so I think everybody kind of needs to be on board there.

But, um, let's see. We have these team skills and plug-ins. We have our personal stuff. Most of our team and personal is global. That's great. Um, can we retain each person's personal tool and configuration, or is there a better approach? "For parent mobbing with agent." Yeah. Okay. So, um, I think I kind of answered it with the collaborative space.

1:20:46 Speaker 0: Okay.

1:20:47 Speaker 0: All right. "How can we leverage agents for personal productivity?"

1:20:55 Speaker 0: Which works well. "Week two seems about building agents for a company. How do the two weeks connect?" Great question, right? So, um, I specifically wanted to bring that perspective to the course, um, because a lot of what I see taught or talked about online

1:21:18 Speaker 0: is from, from folks who are building their own thing or they get to, like, use this stuff for fun. And coming from an enterprise setting, like, that's not my experience, right? And I find most people are in my situation where, like, yeah, I can use it on a side project, but that's not the, the main thing, right? The main thing is I have to use this within a company that has rules, um, you know, that I have to work with a team of people, so it's not just me. I don't get to just do what I want. I don't even get to choose my own models or my own harnesses or anything like that, which looks a lot different, right?

So, um, I would say both skills are very necessary right now. So week one taught you about, okay, how to use these agents effectively, and I think a lot of what Theo covered is not just applicable to your own side projects, but they are also applicable to how you use agents to do your coding work. Um, and then my lesson then says, okay, beyond the coding work though, how do you build agentic systems? Which is gonna be the next thing you're gonna be asked to do within, I don't know, I don't wanna predict, but I know this is coming. So, like, you'll be asked, like, "Oh, we need you to build the agent," or, "We need you to add, like, some agentic capabilities to this existing code base," or whatever, right?

And so that's a skill that you're gonna need to have.

1:22:52 Speaker 0: Where should you start? Right here. Start in the, um, in mega.dev. [laughs] Um, okay.

1:23:02 Speaker 0: I think that's everything.

1:23:10 Speaker 0: Any other questions?

1:23:15 Speaker 0: I think we're almost at time anyway, so there-

1:23:19 Speaker 1: Yeah. We, we have around five minutes left, uh-

1:23:23 Speaker 0: Okay. [laughs]

1:23:23 Speaker 1: Angie, so probably we would need to take maybe some finishing, uh, questions from the chat. There have been some, uh, questions mentioned, um, earlier,

1:23:35 Speaker 1: um, but I think we have managed to clear Discord for everyone. Uh, in case there are any questions on Discord that you don't feel like have been answered, remember that Angie will still be throughout the week with you guys on Discord and will help to answer any questions that, uh, sort of remain there.

1:23:58 Speaker 1: Um, and for now, I think we could take some, uh, uh, questions still from the chat if you guys have anything

1:24:07 Speaker 1: that is still-

1:24:07 Speaker 0: What do I use Dot for? Everything. Um, y'all, I love Dot.

1:24:13 Speaker 0: So Dot, for those who don't know, ChatGPT rolled out Dot like a month ago or something. Um, it is their always-on persistent AI agent, and, um, I basically have given it access to, like, my Slack, my email,

1:24:35 Speaker 0: and my Google Drive.

1:24:37 Speaker 0: And with that, it's become essentially like a personal assistant, and so it kind of, like, keeps me on task. Like, when there's things that maybe I'm blocking people, it'll flag that for me. Um, it used to be a lot more,

1:24:54 Speaker 0: um,

1:24:57 Speaker 0: proactive. I don't know. I don't know if they've cut that back or I'm low on credits or whatever. I don't know. But it's, uh, it's not as proactive as I want it to be. Maybe it's a skill issue and I need to tweak it, but it used to, like, find things that...

1:25:14 Speaker 0: Like, say somebody asks me a question. Now it'll be like, "Oh, Sarah is waiting on you to answer this question for her." Whereas before, it would say, "Oh, Sarah had a question and I went and found the answer. Here's... Do you want me to send it to her?" Like that, right? So I think that's how it's supposed to work. Um, why not GrokBot, Claude, Duet, Hermes Agent? Um, Muse. [laughs] Yes, everybody is coming out with one. Um, so

1:25:45 Speaker 0: the honest answer to that is for my harness, I prefer Codex/ChatGPT work. I prefer that over Claude Code, um,

1:25:59 Speaker 0: Cursor,

1:26:02 Speaker 0: uh,

1:26:04 Speaker 0: yeah, all of them. So I was already in that ecosystem. I don't know if it's better. From what I hear, it's not even better than those other ones, but I don't want to, like, have five of these things that I use, right? I want to have one or two, you know, max. Um, ideally one, and so that's, that's the sole reason. I don't, I don't, I don't know if it's better. I haven't even, um, used, like, Muse and all of those, but I hear they are better, so [laughs] I just don't want to be... Yeah, T3 Chat, all of that. Yep, I know. I know. Um, but I'm using Dot. [laughs] Muse got hacked.

Oh, wow.

1:26:46 Speaker 0: So I use, actually Goose, um, is the open source AI agent that, like, is the OG one that supports all of the models.

1:26:57 Speaker 0: We support, like, 80-something models, and it has, like, all the features. Like, that one comes out with the features before they exist in all of the other harnesses, actually. Um, so I'm well-versed in, like, how this stuff works. Um, and so I use that one,

1:27:17 Speaker 0: and I use, um, ChatGPT because ChatGPT, I can connect with multiple, um, works, Google Workspaces. So that one I use for work and stuff.

1:27:29 Speaker 0: Yeah. We all have our preference. I don't even know if it's a preference at this point. It's just like I'm tired of switching [laughs] and I'm try- tired of trying them all, so I was just like, "Okay, look, I'm just gonna use this one." Um, which is also the importance of things like plug-ins, 'cause I have all of these skills and, and MCP servers and everything that's connected now to, um, ChatGPT.

1:27:53 Speaker 0: Um, and so, like, if I were to try something else, I need this to be like a plug-and-play. I need you to just already be able to pull in all of my stuff. If I gotta set everything up again, forget it.

1:28:05 Speaker 0: Yeah.

1:28:06 Speaker 0: Okay. Oh, one more about compaction.

1:28:09 Speaker 0: Okay.

1:28:14 Speaker 0: Oh, is this, is compaction the harness's job or the provider's job?

1:28:21 Speaker 0: So the provider will auto-compact, but I think it's

1:28:29 Speaker 0: also a good idea if the harness

1:28:33 Speaker 0: can call for compaction. And that's because the harness is gonna have, like, all of the context about what you're trying to do, and it could also probably do a better job of, like, saving and persisting anything that, you know,

1:28:52 Speaker 0: is worth keeping, um,

1:28:56 Speaker 0: outside of the compaction.

1:28:59 Speaker 0: Um, so this person is, like, it looks like they're pre-advanced. Yeah. So where should context management live? Should context management live in the harness or the agent or the provider? Yeah. Um,

1:29:15 Speaker 0: so I touched on compaction just a bit, but, like, the, the agents will provide or the harnesses will provide, like, a way to do compaction. When we looked at this in, like, in developing Goose, um, we basically just set a threshold that was configurable that the customer can change as well. So the model will auto-compact, but we try to compact before the model compacted, um, and we would do that at, I think the default was, like, if you've used, like, 75% of the context window or something like that, and then you can configure it to be like, "Oh, I wanna compact earlier. I want, I wanna go all the way to, like, 90 or 100%," or whatever.

Um, again, as these models get better,

1:30:04 Speaker 0: compaction gets better as well. So these are things that you can also, like, kind of revisit as stuff gets better. Like, when the model is great at this stuff, then maybe it's a, a task that the agent or the harness will no longer need to worry about. But for now, I think it's a good idea for the harness to consider when might be a good time to compact, and then is there anything that needs to be persisted before that's done?

1:30:33 Speaker 0: Okay.

1:30:35 Speaker 0: Okay, folks. Um, if you have any other questions, feel free to put them in Ask Angie. I will be around, um, throughout our Mega Dev time together, so I can answer those ASAP. Thank y'all so much for hanging out with me,

1:30:56 Speaker 0: and I hope you enjoyed, uh, week two.

1:31:00 Speaker 1: Thank you, guys.

1:31:04 Speaker 3: Thank you.