Skip to content
Linkdrop
Links

S03E03 · AI-Native Product Development: MEGA lesson video by Kent C. Dodds

Kent C. Dodds explains how AI agents can verify pull requests using screenshots, preview environments, system recaps, and multiple AI reviewers. He describes this as the second pillar in a workflow for agent-driven development, alongside reducing the blast radius of mistakes.

videoKent C. Dodds · hq.mega.dev · Saved Nov 19, 2026

mega course, automated code review, pull requests, production deployment, agent reliability

Transcript of the lesson video for S03E03 of the MEGA.dev course, by Kent C. Dodds. Agenda: Exploration · Planning · Execution · Feedback · Engineering · Delivery. The video plays on the lesson page in MEGA Student HQ (needs a login).

0:05 Even with good inputs, agents still make mistakes. Step two is catching them before they hit production. And I don't do it by reading the code. There's just too much that's happening, and I can't read all of that. And so instead, the agent proves its own work by a preview and explains what it changed in the system, and then other agents review that. Then they iterate, and once they've gotten all green, then depending on my comfort level for the repository, the agent will just merge it themselves. So this is step two of the three pillars for getting a really good flywheel, uh, of having your project on autopilot. So, of course, we want to have quality pull requests made in the first place.

That's the first step, the first pillar. Uh, but we assume that mistakes will happen. This happens with humans, it happens with agents as well. And so step two is a kind of like a little sandbox or like a playground for the agent to, uh, put up its changes. And we're going to talk about what that looks like. It's, it's literally just a pull request, but there are a lot of things that happen as a part of that pull request that makes this effective. And then step three is, well, maybe some bad things manage to get through that process as well, and so we want to limit the blast radius of things that could go wrong, and we'll talk about that later.

And then once you have this all set up, then you can just get agents to do a lot of work for you, um, and it's pretty cool. So we're going to talk, talk about step two. So, um, this is one real example of what a pull request looks like in my project. So here we've got, uh, Cody. It's keep the footer at the viewport bottom on short, uh, account pages without a blank chin. And so this is very much a visual thing, and so you can see this is the before. This was a, a bug. It looked really bad. Uh, and then here's the after. It's, um, down to the bottom. And, um, this visual thing is, is useful for me as a user or as a, like, a reviewer of this stuff, but of course, I'm not reviewing this stuff myself.

And see, like, look, goodness, it's, it's all over. They've got all of these. I'm not reviewing these myself. The agents are reviewing them. But by making the agent, um, create these screenshots, um, it's forcing the agent to verify for itself that it actually did do this. And, um, so depending on your level of comfort and, uh, autonomy that you give the agents, having, um, a screenshot in here, of course, the agent is going to be able to check its own work, but then you also can visually check its work. And even now, as I have my project just really kind of on this flywheel where it's merging stuff without me even looking at it, um, there are some things that I want to be able to see what it actually looks like.

So getting your agent to put screenshots in the pull request helps a lot with that. And sometimes the thing that they're changing is actually a, like, multi-step process or some, uh, interaction sort of thing, and so having a video is helpful in those cases as well. Another thing that you'll find in my pull requests is a preview, a preview environment. And this was actually a good idea always, like forever having an environment where you can test out changes in a production-like environment is super helpful. And so I just have a GitHub action that, uh, deploys a, a version of the worker, uh, to Cloudflare and, uh, actually fake versions of some other dependencies as well.

So, like, we have mocked out versions of third-party, uh, things that we don't want to actually touch in our preview environments. And then once the pull request is closed or merged or whatever, then those get cleaned up automatically. But we, yeah, like, we get D1 and, and, uh, KV, and, uh, we actually have several workers that all need to be deployed, and then we have, um, the mocks worker as well. And you want to try and drastically reduce how much of the, uh, your actual source code is affected or, or has some logic around the mocks and the previews. Like, you want to limit or completely eliminate that entirely through the use of environment variables or that sort of thing.

But, um, having a preview environment like this is super critical. It allows them to, uh, verify things like webhooks and, um, and things that just kind of require a production-like environment. And here's an actual example. So we had a webhook fix. There's 24-hour grace when signing secrets rotate. And having this preview environment that is deployed... Man, this one went through a lot of iterations here. We'll talk about those AI reviewers here in a second. Uh, but, uh, so you get that preview deployed. You also get seed data as part of that preview so that the agents don't have to, um, like, go through a whole bunch of steps to create an account and all of that stuff.

But then the Control Cody CLI that we have, uh, in this project, so that Control CLI that we've talked about, um, that also has utilities for talking to these preview environments. And so it was able to, uh, set up the, uh, necessary, uh, test data in this preview environment for the agent to actually test whether or not, um, the, its changes work in that deployed environment. And so here's that, uh, Control Cody here again. Um, and like I said, the, um, agents actually created all of this for me. So I, honestly, I, I can read this, and I know what all of these things mean, but I did not write any of this stuff.

And we'll talk about how you can get your agents to be more proactive about improving the code base once we get through the flywheel of the three pillars, and now we can just have signal. This comes from the friction log, and it just helps the agents build out this CLI. It's so, so cool. Um, but yeah, right here we've got this, um, Control Cody preview with the PR flag, PR number, and then we want to check a particular URL. And, uh, that allows the agents to, once it's made the pull request, we get the preview environment, and the agent can verify its work, uh, there as well. Uh, even reviewer agents can do that.

And, um, as the, uh, changes are happening to that, um, pull request, the preview environment is also getting updated. And I guess I should call out too that, like, you don't necessarily have to do this at first. Agents are, like, perfectly capable of pulling up the preview environment and clicking around and everything, but having the CLI makes it a much faster and more repeatable. And, uh, the other thing, like, kind of elephant in the room is like, "Ken, what about tests? Like, don't you just use tests for all this stuff?" And yes, you can absolutely, um, have the agent write tests, but sometimes you just want to do, like, a one-off check, and this doesn't really re- uh, require a long-term test.

If I had a test for everything that agents use the Control Cody CLI on, um, then that, like, I would have so many tests, and many of them would not be necessary. When we're developing software or when we used to develop software the way we used to, uh, we did tons of manual testing, and only some of those things became actual automated tests. And so having a CLI to make it really easy for agents to check their work so they don't have to click around, uh, themselves, but also, uh, so they don't have to write a, uh, automated test and keep that in your repo, slowing everything down, uh, it's a nice balance. So there's a, a place for everything, and the CLI is super helpful in combination with those preview environments.

Another critical part about the PR is the system recap. So this helps a great deal. It, uh, forces the agent to look at the primitives and decide what primitives it's actually using in the pull request. And I've seen agents that look at this and like, "Oh, I didn't even realize there was a pull or a primitive there." And so they, they go and they, uh, adjust their implementation based on this. And so you've got your, uh, user meter, entitlements, and email primitive for this particular, um, pull request. This is the group they're in, and this is the impact. We're creating it, we're deleting it, we are composing it, we're extending it, we're, um, put, you know, whatever.

And so if it's all just composing different primitives together, that's really good. That's, like, a good sign. This is why it's a low-risk change, uh, because it's just composing these. Uh, there's also the user flow. I really like these sequence diagrams. It's helpful for me when I'm looking at them to, uh, understand what the change, uh, is actually, uh, influencing. I'm looking at the im- the output, uh, or rather the outcome, not the output, uh, or the impact of the change that is happening here. And so the, uh, having some mechanism for seeing visually what the changes are and is, in particular, on the primitives that are happening here, whether we're extending or deleting or, or composing, uh, and seeing that level of risk, that helps a lot.

Now, I've got to say that I actually don't really look at this myself. That's part of the, um, not necessarily part of the point. Like, once you get this flywheel that's going and the agents are doing the implementation and the review and it ends up in production and with that smaller blast radius, once you have that, you're not really looking at these all that often. What it does is it forces the agent to think about these primitives and whether, uh, they fit. And, and if there is a high-risk thing, then maybe you do want to be flagged, depending on the scenario, uh, that you're in. So, uh, having the agent classify these itself is really helpful.

It's helpful for the reviewers, the review agents, so that they know, "Oh, this is supposed to be a low-risk change, and they're saying they on- only compose these primitives, but I'm looking at this and I'm seeing that they are expanding this primitive, they're deleting this primitive," whatever. And so it helps with the reviewers as well. Speaking of the reviewers, I've got three on my project, and, uh, each one of them catches something else, uh, something different. And so that actually is really interesting. So I, here I'm using, uh, Cursor BugBot and CodeRabbit and Devon. And full transparency, each one of these give me free use to experiment and, and try out their tools and, and leave them feedback.

Um, but, uh, I, I like them all. Uh, if I only had to choose one, I probably would, uh, stick with Cursor BugBot. That one seems to find things, uh, more often and the things that I'm looking for. But I should also say that I haven't even tuned these. Uh, these are just, like, raw, how they work. And so I probably should go in and tune these because they have a lot of different knobs that you can twist for these. But, uh, these are super helpful. One thing I want to call out is that they do end up finding different things in each one of these pull requests, and that leads me to believe that, like, okay, I wonder if I introduced a fourth AI reviewer, would it find additional things?

And if that's the case, then the fact that I don't have it means that, like, there are additional things that could be found. Uh, and so that, like, you have to take all of that with a, a grain of salt. Uh, the same thing happens for human reviewers. Like, sometimes a human reviewer is going to miss something. And so this is why that step three, like reducing the blast radius, is so important. Uh, but I would say for the most part, I haven't, like, with the months that I've been doing this, using these tools, I haven't had a situation that went off into production and I was like, "Oh my goodness, I can't believe the reviewers missed that."

Uh, like, I honestly, I can't even think of a situation where that has happened. So we are catching all of the most important things, and we also, and we'll talk about this later, but we also have mechanisms for cleaning things up after the fact, um, if something does come up later. But yeah, I've, I've not had any big incidents or anything like this. These AI reviewers are quite good at this. So I, I do use all three of them. Um, Devon is, uh, particularly really good at finding security problems. In my history as I've kind of gone through everything, uh, Devon seems to find those security problems really well. But, uh, yeah, they're all really, really good.

So you could, uh, really just use any one of them and get good results out of that. And there's, of course, another option. You could just make your own. But if you do make your own reviewer, here's what I recommend. You do not want the reviewer to be the same as the implementer. And this is actually true of humans as well, right? Like, you would have the human who would implement the, the thing, they make the pull request, and then they'd request reviews from other people. Because they're, like, the, the context refresh is actually really valuable. So if you are going to do your own reviews, you could even do them locally. Just make sure that they are a different person, uh, or rather a different agent.

And, um, in general, it's actually useful to use a different model as well, and just you can set up, like, these adversarial councils if you want to about, like, you know, force the agent that implemented it to defend themselves against these others. Um, that's kind of what I have in here, where I've got these agents who are posting things, and then my CodyBot agent comes back and it says, "Oh, I fixed that," or, like, "Here's why that's not the way, you know, this way." So having some sort of back and forth, you're, you're kind of simulating the way that humans built software. And I don't know how long that's going to totally be necessary.

Maybe eventually the agents get super, super, super good, and they don't need to be reviewed or whatever. Who knows? Like, that's, that's a problem for another version of you in the future. But right now, having a, a set of, uh, agents that are doing the reviewing- Uh, typically ends up giving you better results. And so that's why I've got the three. Um, and I like the cloud services. They are, like, really fine-tuned for this. But if you want to build your own, that's useful. Just make sure that the reviewer is different from the implementer. But you can't just rely on AI reviewers for everything because, like, they are expensive. Like, they do cost money.

And so having deterministic review is also super valuable. And frankly, this is, um, maybe where you thought I was going to go with when we started here, like having testing and linting and, uh, automated or type checking. Like, as much of the deterministic stuff as you possibly can is super valuable. So, uh, I've got my actions in here. This is what blocks, uh, any merges. This blocks deploys, uh, to production. All of that, uh, you want to have some sort of validate process in here. Uh, my validate job handles all the, the static analysis stuff. It also runs this nip thing, which, uh, like reduces dead code. So it's, uh, any exports that are not imported anywhere, that's going to fail the, the nip check.

Um, there's also a slop ratchet, which is effectively just... And, and this wasn't actually my idea. This was my agent's self-improving the, uh, the software. But the slop ratchet is effectively like if files are too long or if there's like these giant banner comments that agents like to put in there, my agents were like, "That's stupid. We're not going to let them do that." And so they put that in the slop ratchet. Uh, you can also have linting in there to, to catch things and just tell your agents, like, "Keep an eye on things that are like the slop sort of thing that, that you see happening and, um, make some sort of deterministic code to avoid that."

So determinism is actually really valuable. Um, also, like, the, my, uh, action for validate, uh, handles, uh, primitives and decision doc checks and deploy guardrails, all of that stuff. Um, the npm run validate runs 33 checks at once. It's, it's a pretty beefy thing, and I've done a lot of optimizations with caching and everything to make that run as quickly as possible. Um, but time, like, as, as valuable as time and, and, uh, optimizing that is concerned, like, I, my time is really valuable. And so making sure that there's a really good quality bar that, of deterministic things that make sure that we don't end up shipping mistakes, that is more valuable than a, a minute or two more on my CI.

And so I am more than happy to put as much deterministic checks in my CI to make sure that we don't end up shipping something that I'm going to have to revert myself later or whatever. And so this has worked out really well. You definitely want to have some sort of validation process. And one tip for this is I use NX for my, uh, validation and, or for, like, really for all of my scripts. And so when the cloud agent is running the tests and the type checking and linting and all of that stuff inside of their cloud environment, I make sure that they have what they need to be able to write to the cloud cache.

So it's just some environment variables. Uh, they write that to the cloud cache, and then when it runs in CI, uh, in the PR, then it just hits the cache, and so it's pretty fast. And so you can kind of get some, some benefits from caching stuff. So, like, there are other, um, projects that handle this, Turborepo and that sort of thing. But, uh, get yourself something to, like, allow you to have these kinds of checks so that, uh, you don't end up shipping something to production that's bad, but make it fast, uh, through some good caching mechanisms as well. Oh, and I should mention also that every single time a reviewer comments and the agent updates the, uh, pull request, so like, uh, in response to one of those review comments, or even if there's a build failure, the agent will, uh, update the pull request as a result of that.

When that happens, then, uh, we're going to run through the whole validate process again, and we run the reviews again. All of that, uh, is valuable. And this is part of the reason why the cloud, uh, reviewers are really good, because they're pretty good at handling those updates to the pull requests. And then I should mention too that validate is, also runs the tests, and agents like to build out so many tests. And so we're going to talk about how we manage that, uh, later on in episode five. So, uh, keep an eye, uh, or look forward to that. But, uh, yeah, you definitely want to have tests be, uh, run as part of this validation as well.

The last thing that I want to show about this is that much of this process is not some big skill. In fact, it's a very short skill. Uh, just pretty basic, and it's pointing to the Kodi CLI to run my ship PR package. So this is not an ad for Kodi, but this is so, uh, a recommendation that I give to you that you need to take your skills and turn them into determinist, uh, deterministic code. So you can do this with scripts, but I wanted to show you what my Kodi package does so that you can build a similar script if that's something that you want to do. And this is also open source.

You can look at this, uh, on Kodi and look at the files and everything as well. But, uh, yeah, the basic idea is that, um, every time the agent runs this, uh, npx command, it's going to kind of get through each stage for the PR that we're talking about here. And so the stage starts with marking the pull request as ready from draft, and that's when all the AI reviewers start. And then, um, it also checks on CI and makes sure that CI is green and, um, AI reviewers are good, and you iterate with AI reviewers and CI until those are both green. And then the next step is to actually merge based on the risk policy.

So for, um, Kodi in particular, I say, uh, you pick low, medium, or high. This is what, uh, how you determine what that is. And, um, and then we auto-merge if everything, uh, it looks good and it's not a high-risk change. If it's high risk, then you can only merge if Kent explicitly says that it's okay to merge, which frankly I often do. [laughs] But, uh, so then, uh, if the agent decides, "Okay, I'm going to merge it," then, or once the PR is merged, whether it's me or the agent, then it's going to deploy. And it's going to watch that deploy. So the workflow runs, uh, the, uh, sh- uh, Shaw, and once it gets into production, then it's going to check the health endpoint.

And once that health endpoint is all thumbs up, then we're in a good place, uh, and it knows that it's actually been deployed. Uh, then it refiles friction, and it reports to Discord. Uh, we're going to get into these pieces in a future, uh, lesson, so don't worry about these pieces right now. But come up with some way to have some deterministic software that gets your agents through the whole process from, uh, like creating the pull request all the way to actually merging it with deterministic code. You're going to save on tokens, you're going to make it more reliable and more repeatable, uh, and put it in some sort of portable place so that you can use it regardless of whatever agent you happen to be using.

Oh, and one other thing that I wanted to mention about this. Uh, so I am able to have such a small skill here because every time this CLI command is run, it gives the agent exactly the context that it needs. So I don't have to describe every single thing that happens. I said literally just run this script, and you're going to get the output that set- gives you the instructions of what to do next for this step. So it's a progressive disclosure thing, and that's really valuable from a context perspective because it means that I don't have to blow it up with all of this context only for it to find out that we're actually not going to merge this anyway, and we're going to close it.

So, uh, by having it be like distinct steps, and it's, it's effectively a state machine inside a deterministic code, but every step gives it the necessary context that it needs. So you don't blow it up with context and get it confused about like, what is the most, you know, thing that I need to give my attention to? Uh, so this is another valuable piece of turning your skills into deterministic software is because you get that progressive disclosure, uh, just by virtue of the fact that it's just running through the state machine. And that's it for step two. So the agent creates a good pull request, that's step one, and then it iterates with AI reviewers and CI and the verification with the preview deploy and the CLI that helps it with that, and the deterministic code that helps it go through the whole process of PR open to PR merged.

And, uh, then I never have to read the code myself because I'm confident in this process. So thank you very much for your attention.