Skip to content
Linkdrop
Links

S03E04 · Verifying Product Outcomes: MEGA lesson video by Kent C. Dodds

Kent C. Dodds explains how feature flags, deployment checks, and tested backups can limit the impact of mistakes made by coding agents. He covers rollout metrics, agent reports, and disaster recovery drills using examples from Kodi.

videoKent C. Dodds · hq.mega.dev · Saved Nov 19, 2026

mega course, production deployment, agent permissions, feature flags, disaster recovery

Transcript of the lesson video for S03E04 of the MEGA.dev course, by Kent C. Dodds. Agenda: Acceptance · Review · Verification · System Diff · Quality Gates. The video plays on the lesson page in MEGA Student HQ (needs a login).

0:05 So, so far, we've gone through the first two steps. So you've got quality in to make sure that the agents make good code in the first place, and then step two is about catching mistakes that do manage to make it through that, uh, through AI review and, uh, validation and all of that stuff. And then step three is about making it so that if a mistake does make it, it's contained, that it- it's not that big of an issue. And once you have all these done, then you can set up your system with really high signal in, and agents do all of the work for you. So you're just stuck with the product and architectural calls, like the, the big systems thinking stuff.

And so the, the question is, if a mistake does make it into production, some sort of mis- like, it, uh, an accidental deletion of user data because of a bad migration or, uh, some sort of feature that, um, users start actively using, and then you realize, "Oh, no, that was like, that's not the direction that we want to go long term for the product." Those are the sorts of things that we're talking about when we're talking about mistakes. And so the very first thing that we're going to get into for avoiding this being a massive problem is we're going to talk about feature flags. So this is not a new concept. We've had feature flags for a long time, and feature flags are one way that you can turn a one-way door into a two-way door.

So a one-way door would be the, uh, sort of change that is going to be impossible or very difficult to revert. An example of this would be you ship an API to, like, a very active application, and users start using that API immediately, and they start integrating. Now, pulling that out is going to be a really difficult thing. Or a migration that ends up deleting or reformatting data in such a way that you don't have a mechanism for switching it back. So that's the idea of a one-way door. You can turn a one-way door into a two-way door, or at least get it a little bit closer to a two-way door by using feature flags.

So here's an example of this from my own work. So in Kodi, you have this idea of packages, and those packages can be associated or can use credentials. And I decided that I wanted the package to declare upfront what credentials it's allowed to have so that we can have some security checks and a variety of other things that make, um, that a lot safer. And so to do this, we already had a whole bunch of packages that were using the, the credentials without having this declaration upfront. And so the agent decided to put this work behind a feature flag. And so here's the issue where it's discussing the change that we're going to make, and here's the pull request where it actually made that change, and it said here, this is a two-way door because we have a flag, uh, this credential need enforcement flag, uh, and that's off by default.

So you can turn it on for those people who are experimenting, and they are expecting things to break here and there, and so, like, it's not a huge deal if they go to use a package and they get an error. Like, they're expecting those kinds of things to happen because they're eager. They want to use the latest features. And, uh, and then if something is really bad, we can just turn it off. And so, like, it doesn't end up being a huge problem. And once this shipped, we found that there were actually 211 packages that were affected and needed some manual migration, which we then did, and ultimately, we were able to, uh, turn the feature flag, um, back on for everybody and ultimately delete that feature flag.

And there are services that help you with feature flags, but I just built my own because why not? You can do that. And so here's the admin interface for what that looks like in Kodi. So we've got this JEFF search re-rank, uh, feature that I really should probably just roll out to everybody by now, and execute invoke, another one that I should probably roll out to everybody. Uh, connection profiles, this is one that I'm actually really trying to think about. I'm not sure that I want this one. And so having a fe- this behind a feature flag means that I can ask people to try it out and experiment with it, but ultimately decide, "You know what?

We're going to pull this out." The other thing that's important here is that feature flags should come with success metrics. So in particular with, uh, the execute invoke, for example, the suc- success metric is whether the dynamic worker day event count decreases. And we can, uh, we keep track of that, so, uh, like, the code is instrumented such that it's tracking whether, um, that number is actually coming down for the affected users, and that helps us know whether we can actually turn this on and whether it's going to improve whatever metric that we have in place. And then for each one of these, you can have a rollout to a percentage of users, uh, whether that's everybody or the people who have actually opted in for these experiments, or you can even have, like, specific people who have a particular feature enabled.

And so with this, you're able to really experiment on a lot of different things, and you can pull things out and, like, feel really flexible to be able to try those experiments and, and have it be a limited blast radius. And especially if there's a significant change that you're not sure it's going to improve things, that, or the agent makes some sort of mistake, having that behind a feature flag is a good way to avoid, uh, making that blast radius just massive across your entire code base. Another thing that I have my agents do to help me be more confident in what ends up in production is just being aware of what is in production.

And so the agent, once that pull request is merged, the agent is not done. Nope, not by a long shot. What it actually does is it will look for the job that is deploying its change, and it's going to subscribe to that job. And once that deploy is finished, then it's going to hit the health endpoint and make sure that what comes back is the commit that it just created. And here's what that health page looks like, uh, well, rendered in a browser. When the agent re- makes the request to the health endpoint, it's going to get some JSON here to determine, was this, uh, a deploy of the commit that I had, or a commit that is further along past my commit, to make sure that it actually landed in production.

And if it didn't, then it's actually going to look at that job, and it's going to make sure that it fixes whatever went wrong with that so that, uh, we end up in a, actually deploying the thing that it needs to deploy. And once that actually does work, then the agent is going to send me a report. I like to get my reports in a private Discord, uh, between me and my agents, and so it sends something that looks a little bit like this. So here, it's Cody. It says, "Hey, this shift here, agents can make credential-free audited package in, uh, internet visible without Kent's click." Okay, great. Cool. And it tells me the cost.

This one was included in my subscription. Uh, it gives me a diff, the model, the difficulty, whether it was a two-way, and, uh, any cleanup that was needed, uh, links to the agent and the repo, and a link to the PR here as well, as well as, like, a full-on summary. And so this way, as changes are happening, and I'm getting messages all the time, uh, but as these are happening, I'm able to keep a, like, general pulse on what's going on in the code base and be like, "Oh, hold on a second. I didn't ask for that, and I don't like that," or whatever the case may be. Uh, and so I can, um, make adjustments and just keep up with what's going on here.

And that's really nice if I don't want to have to open up GitHub. I just open up the Discord app on my phone. But what if something actually goes really wrong and actually deletes data? That is, like, the real concern. And so that's where we get into backups. But not just, like, having a backup, like, data that's backed up, but actually restoring from those backups. Because just because you have the data somewhere doesn't mean that it's easy to restore things. And every second that you have, like, data getting deleted or whatever is just more data that you could potentially lose. Like, for example, if you ship something that it no longer is recording the user's, like, desperately needed notes from their meeting or something like that, uh, or, and, and in fact, it's deleting it or something, like, all of the time that it takes you to actually restore from that backup is going to be additional data loss.

And so you want to make sure that not only do you have the backups, but that you have tested the, uh, restoration process. And you can do this, like, pretty, in a pretty straightforward way. You just have a version of your application that's deployed that's not the production application, and you, uh, you take those backups, and you take that restore process, and you apply it to that other version of your application. And if that applies successfully and you get all the data in there that you're expecting, then you're good. And you just do this on a regular, like, automated process to make sure that you can always restore backups. And so I've got a disaster recovery plan inside of Cody that describes exactly what you need to do in the event of some sort of, uh, disaster.

And this is actually an example of, uh, finding something in a drill. I actually ran a drill to make sure that the disaster recovery worked, and it didn't. [chuckles] And so we fixed it. So if you're not running drills, then you don't know whether your backups actually work. And I-I've got to tell you, this actually happened back in April. There was a, a company that h-had a disaster. An agent decided to delete a volume that it thought was the staging volume, turned out to be the production volume, and that's where their database was. And, uh, it, they were frantically trying to, like, put together their disaster recovery plan and come to find out that they hadn't tested the disaster recovery plan, and the backups were actually stored on the same volume as their production database.

So when the volume was deleted, the production database was deslo-destroyed as well as all of the backups, which they have now obviously rectified. Um, but this is the sort of thing that you would catch if you were being a, a lot more intentional about how you're building your disaster recovery. Let me show you mine actually really quick. So here we're gonna get that login code, verify that, and th-this is very important. So I'm, I'm putting mine behind Cloudflare Access. It's a really simple mechanism for adding, uh, authentication to something. I didn't want to have to bother with authentication. Cloudflare Access is really good for this. Uh, but mine is hosted on a completely separate, um, uh, box, in, in fact, a completely separate infrastructure.

So this is hosted on fly.io, and, uh, here we've got all of my backups, and we have that, uh, the D1 manifest, D1 verified, D1 is restorable. Uh, we've got our, whether the staging is complete, whether it's sealed, and whether that sealing has been verified. So this is all part of the process of, uh, validating that these, uh, backups not only exist but are actually, uh, able to be restored. And then we can, uh, I, I could seal this myself, uh, each one of these that haven't be-been sealed yet, but they'll get sealed as part of their regular process eventually. Uh, and then I can also run an isolated restore drill myself manually as well if I wanted to just make sure, I don't know, just feel good about myself to make sure that it-it's working.

But all of that is automated already. Uh, and then this is how I do a production restore. I literally say which one of these I want to restore, and, uh, then it will prepare that. And there's a, a, a step process that I go through that's all deterministic to actually restore the data to that time period. So you need to have this. You need to have it, uh, tested and verified, and, um, then when the disaster actually happens, you will have some, like, a process that you can go through to restore that data. So this is all, like, this is a good idea since the beginning of building applications, is having some mechanism for restoring data from some data loss.

But now it becomes even that much more valuable because we're trying to trust these agents even more. And, uh, these agents, they're gonna continue to get better all the time, but even as good as they get, things happen sometimes that are outside of your control. So you need to have some disaster recovery to, uh, reduce the impact of anything bad that happens in your application. So that's step three. You have agents watch the pull requests get deployed to production and make sure that everything gets where it needs to go okay. You have them report to you to make sure that you're kept up to date on what is going out into production on a regular basis.

And you have feature flags so that as agents are making changes, if they're significant changes and you're experimenting with things, you have a mechanism for doing that and for, like, effectively reverting without having to do a full redeploy. And then you have disaster recovery in the event that something really bad does happen, and not just, like, the backups, but backups that you've actually restored. And with that, now you have agents that can make good PRs, that can review and to avoid mistakes making it into production, and if they do make it into production, you've drastically reduced the impact. So with that, that is enough for us today. Thank you so much for your attention.