Skip to content
Linkdrop
Links

S04E04 · Automating Work Beyond Code: MEGA lesson video by John Lindquist

John Lindquist shows how to build agents that respond to events while a team is away, including a support-ticket sorter, customer-reply drafts, and a multi-step error hunter. The lesson covers durable execution, evaluating ticket classifiers with labeled examples and repeated trials, and routing uncertain results to a person.

videoJohn Lindquist · hq.mega.dev · Saved Nov 19, 2026

mega course, agent orchestration, agent evaluation, agent reliability, durable execution

Transcript of the lesson video for S04E04 of the MEGA.dev course, by John Lindquist. Agenda: Events · Hooks · Pipelines · Push Systems. The video plays on the lesson page in MEGA Student HQ (needs a login).

John Lindquist (MEGA.dev) S04E04

Work that starts without you

00:00:01,860 --> 00:00:41,118 [John Lindquist]

Your domain agents now help your whole team, as long as somebody's at the keyboard. But guests still write in overnight, and pages still break when nobody's looking. So today, let's have your agents help your team while everyone's away. Remember, an event can trigger an agent. Inside your editor, events already wake your agents while you work. Now let events from outside wake them too. A guest sends a ticket, or a request fails, and your app writes that down as a line in its log, its running record. Your app shouldn't wait around for an agent to think. It writes that line and moves on. So ask Main for a watcher: a small program that stays running, reads new lines as they land, and starts the work.

Durable host

00:00:41,978 --> 00:01:53,853 [John Lindquist]

Now, some of that work takes many steps. A step is one thing the agent does, with its result. And with nobody watching, one hiccup shouldn't wipe it out, or make it start over. So run it on a durable host: a small server that saves each step as it finishes. That's called durable execution. Now, durable means it survives a crash or a restart. It doesn't mean always on. While your laptop sleeps, nothing runs. So we're going to build it on your own machine first, where it's fast to test, and keep it ready to deploy. Picture a real job from tonight: hunting down an error. It reads what failed. It replays the request. It tries the requests around it. It sorts what worked from what failed. Then it writes a report. Five steps, and it could take a while. As each step finishes, the host saves the result on your machine, before it starts the next one. Now the server stops mid-job. Maybe it crashed, or your machine restarted, right in the middle of step four. Without saved steps, that job is just gone. Or it starts over, and makes every one of those requests a second time. On a durable host, when the server starts again, it reads what it saved. Step one, done. Two, done. Three, done. So the work picks up at step four. A finished step never runs twice.

Come back to a glance

00:01:54,739 --> 00:02:33,975 [John Lindquist]

But work that runs while you're away doesn't help if you come back to a wall of logs. Remember pillar four, glanceable. So everything today ends as one short record in a widget: what happened, with the evidence behind it. And anything about money, or a promise to a guest, still waits for you. We'll build three jobs today. A sorter for your support inbox. Drafts for the question guests ask most. And a hunter for the errors nobody reports. The sorter and the hunter wake on their own. The drafts, you start when you sit down. And two of those jobs are quick: one ticket, one step. So they don't need a durable host. The hunter takes many steps, so the hunter is the one that runs there. Let's start with the inbox.

Sorting new tickets

00:02:34,597 --> 00:03:06,141 [John Lindquist]

Tickets land in your support inbox all night. A question about a stay. A complaint about a price. A refund. Your Issues widget reads that inbox when you check in. But then you're still the one working out who owns what. So let a sorter run on every ticket, the moment it arrives. So keep the model's job tiny. Plain code finds the new tickets, and plain code saves the answers. The model only answers two questions. Who owns this: Booking, Pricing, or nobody yet? And is it about money? Anything about money gets a flag, and waits for you.

How good must it be: pass^k and a labeled set

00:03:07,000 --> 00:04:27,967 [John Lindquist]

Before it sorts real tickets without you, decide how good it has to be. At the keyboard, one good try out of a few is fine, because you're there to catch the miss. When you're away, nobody is. So every run has to pass, not just one. Researchers call that pass to the k. Run it k times, and it has to pass all k. So don't pick a sorter by gut. Take a set of past tickets, and write down the right answer for each one: who owns it, and whether it's about money. And don't fill it with easy ones. Put the traps in. A guest says their card shows the same weekend twice. They never say refund, or charge, or price. That's about money, and a keyword rule walks right past it. Another guest asks whether the street parking is free. Free sounds like money, so a keyword rule flags it, and no money is at stake. Add a complaint that the calendar is slow, which nobody owns yet. And add the ones on the border. The heating was broken all weekend, and the guest thinks you owe them something. Your own team might argue about who owns that one. So write both owners down as right. That's a labeled set. It's your answer key. Then ask Main to run each option through it, many times. Start with plain keyword rules. Then a cheap model. Then a stronger one. And read what comes back. A cheaper option can miss the same money ticket every single time. One run would never tell you whether that's a fluke or a habit. That's why you repeat. Then pick the cheapest option that gets every money ticket right, in every run.

Two kinds of model: Jev

00:04:28,558 --> 00:06:05,938 [John Lindquist]

There's a new kind of option for a job like this. Most models you use are writers. You send words, and they write words back, one piece at a time. That's slow, and you pay for every piece. And then your code has to read the reply, and work out what it meant. Now there's a second kind, called a System One model. A System One model works like a reflex. It doesn't write at all. It only picks. The first one is called Jev, from a company called TypeSafe. Here's how you use it. You send the ticket. You send your questions. And for each question, you say what kind of answer you want. For the owner, a choice: Booking, Pricing, or nobody yet. For money, a yes or no. What comes back isn't a sentence. For the owner, it's one of your choices, with a number for how sure it is. For money, it's one number between zero and one. So there's no reply to take apart. And it can't pick something that isn't on your list. And because it never writes, it's fast. On our tickets, an answer came back in well under a tenth of a second. It's cheap, too. You pay a few cents to have it read hundreds of thousands of words, and the answers themselves cost nothing. So you can run it on every ticket, the moment it lands. And both of your questions go in one pass. These models read more than text. Jev reads images, so a guest's screenshot can be sorted the same way. And Google's newest model of this kind takes audio and video too. So the same idea reaches a photo of a broken heater, or a recording a guest sends in. We're sticking with text today. And know what it can't do. It can't write a reply. It can't count. And it can't do math. It picks from a list, and that's all. So it's the wrong tool for a draft, and the right one for a sorter. So ask Main to add it to the same trial. Same tickets. Same repeats.

Unsure waits for you

00:06:06,347 --> 00:07:15,118 [John Lindquist]

Here's what came back for us. It got nearly every ticket right, on every run. And it got one ticket wrong, on every run: that card with the same weekend twice. But look at how sure it was. On almost every ticket it got right, it was sure. On the one it missed, it wasn't. Remember those two numbers that come back? On the money answer, near one means yes, near zero means no, and the middle is a coin toss. On almost every ticket it got right, that number sat near one end. On the card with the same weekend twice, it sat in the middle. The owner confidence works the same way. And remember the tickets we marked as on the border? Most of those came back with low confidence. The model was unsure in the same places a person would be. So add one rule, in plain code, on both answers. If the money number lands in the middle, or the owner confidence is low, the ticket is marked unsure. Code writes down why. And the ticket waits for a person. That rule catches the miss. And it costs you a few extra tickets to read. One honest warning. We set that bar after we saw the miss, on a small set. So we wrote fresh tickets, labeled them first, and ran it again. Nothing about money slipped through. And every owner it got wrong came back marked unsure. Do the same with your own tickets before you trust it overnight.

Test what goes wrong

00:07:15,994 --> 00:08:50,061 [John Lindquist]

Now ask Main to make that model your sorter: a small loop that watches your inbox. Plain code keeps a bookmark: the last ticket it finished. One ticket is one quick step, so a bookmark is all this job needs to pick up where it stopped. And it saves each tag on its own line, without changing the ticket or touching any money. Code also checks every answer before it saves anything. Even a model that only picks can time out, or come back with a piece missing. If that happens, no tag is written. Then test what goes wrong while you're away. Send the same ticket twice, and it should still get one tag. Cut the connection to your inbox, and the sorter should say it's offline, and wait. And watch for a quiet bug here. If the sorter hits an error, skips the ticket, and moves its bookmark anyway, that ticket never gets sorted. Nobody would notice. So when something fails, the bookmark stays where it is. Then the connection comes back, and the waiting ticket gets its tag. Notice the trade. Moving the bookmark last means nothing gets skipped, even in a crash. The price is that after a crash, the same ticket can come through a second time. So the work has to be safe to repeat. Keep that rule. You'll use it again. One more test, because this is the one that happens for real. Stop the sorter completely, while tickets keep landing. Nothing gets tagged while it's down. Now start it again. It reads from its bookmark, and tags everything that arrived while it was gone. Once each. When that passes, it's ready to run unattended. And when it's running, a new ticket has its tag a few seconds after it arrives. So when you sit down, the sorting is already done. Ask Main to show it in your Issues widget: each new ticket with its owner. And anything about money, or anything the sorter wasn't sure of, sits at the top, waiting for your call. So you start with the ones that need you.

Drafts that never guess

00:08:50,733 --> 00:10:22,800 [John Lindquist]

Now, some of those tickets are guests asking what a stay would cost. And you already have a Booking domain agent. So let's give Booking its first customer-facing job: a draft reply to each of those questions, ready when you ask for it. One that never states a price your code didn't look up. One that never leaks your internal notes. And one that never goes out on its own. It's the same idea as your widgets: plain code gathers the facts before a model writes a word. Here, plain code reads the stay out of the ticket and asks your app for the facts: the dates, the price, the total. That's a lookup. Code asking your own app for a fact, instead of a model recalling one. Then a model gets only those facts, with no tools. It can't search. It can't calculate. It can only put into words what it was handed. What about everything your Booking agent knows? Remember, its memories have two halves: internal notes for your team, and customer-facing notes. Don't hand the model both and tell it to keep the internal half secret. It'll often repeat your internal rules to a guest anyway, just in friendlier words. Here's a real one. A guest asks: if I click reserve twice by accident, what happens? Booking's internal notes explain how your app prevents a double booking. That's for your team. We handed the model both halves, and told it to keep the internal half secret. It tried to be reassuring. So it explained that mechanism to the guest, in friendly, everyday words. It didn't copy a single line, so a word filter would never catch it. So cut the internal half off in code, before the model sees anything. Same question, one more time. Now the draft says it can't confirm that from here, rather than guess. A model can't leak what it was never given.

Every amount has a source

00:10:23,640 --> 00:11:46,266 [John Lindquist]

Now test what happens when a lookup fails. Say your app is slow, or the stay doesn't exist. If your code hides the failure, the model often fills the gap with a price nobody quoted. It sounds helpful. And it's made up. So when a lookup fails, code says unknown. Right there, in the facts it hands over. And the draft says we couldn't check. Then add a check after the model writes. Plain code reads the draft and traces every amount back to a lookup. An amount with no source means the model made it up. That draft is stopped. It never reaches the guest. That check stops made-up numbers. It can't tell you a real number is the wrong one. Here's a test. A guest was promised one price a while ago. Since then, the price went up. Now they write in. If code looks up that promise, the draft quotes the price they were promised. Now leave that lookup out, on purpose. The draft quotes today's price. It's wrong for this guest. And it still passes the number check, because today's price is a real number your code handed over. The check only stops new numbers. Only the right lookup makes the answer right. And some mistakes aren't numbers at all. Read the actual drafts. Add one line of policy to Booking's customer-facing notes, and drafts start bringing it up when nobody asked. Some even promise the guest something your app can't do. Every number in that draft is real. Every check passes. And the sentence is still a promise you can't keep.

Sending stays with you

00:11:46,799 --> 00:12:05,894 [John Lindquist]

That's why a person starts every draft, and reads it before anything goes out. Ask Main to draft the pricing questions. Plain code looks up each stay, Booking writes the reply, and each draft shows up beside its ticket: the question, the facts code looked up, and the reply. Your internal notes stay internal. And nothing goes out from here. Sending stays with you.

The error nobody reports

00:12:06,559 --> 00:13:24,517 [John Lindquist]

Now, what about failures where nobody writes in at all? A guest tries to get a quote in the middle of the night, hits an error, and just closes the tab. No ticket. Just one failed line in your log. And guessing at a fix from one line is risky, because nobody's checked how big the problem is. So let that line wake an error hunter: an agent with one clear contract. Report, never fix. And it never touches your real app. Hunting means trying requests that might fail, over and over. So it works on a copy. And a hunt takes many steps, so this is the job for your durable host. Remember, it saves each step as it finishes. Here, each request the hunter makes is one saved step, stored under its own number. After a restart, the host hands back the saved result instead of asking your app again. And connect your watcher to your app's log. Each time a request fails, it posts that line to your durable host, and a hunt starts. Before you trust it, test what happens when that line arrives twice. It will. Here's why. Your watcher only moves its place forward after the host answers. So if it stops right before it saves its place, it starts up and sends that same line again. It's the same trade as your sorter's bookmark. And you don't want two hunts for one error. So every event carries an ID, and the host remembers the ones it's handled. The same event twice runs once.

Break it on purpose

00:13:25,266 --> 00:14:09,427 [John Lindquist]

Now try it. Break a quote on purpose, in a copy of your app. And look at what the hunter has to go on. If your log only records that a request failed, and not what was asked, the hunter is guessing. It can still find a failure by trying things. But it can't try the guest's exact request. And watch how misleading that gets. We ran the hunter that way. It tried one stay after another, until it hit a quote that failed. Then it reported the error as reproduced. That looks like a win. But it never tried what the guest tried. If there were two bugs in there, that report would send you after the wrong one. So ask Main to log enough to replay a failed request: what the guest asked for. And never the private parts, like an email or a card number. Then check it. Make a booking with a card, and make sure none of it shows up in the log.

A map of what broke

00:14:10,215 --> 00:15:38,389 [John Lindquist]

Trigger it again, with the question in the log this time. Here's the bug we planted: a quote fails whenever the stay touches one particular month. The hunter doesn't know that. Watch it find out. First, it replays the guest's exact request, and sees the same failure. So now it's the real one. Then it tries the neighbors, changing one thing at a time. A stay that ends just before that month. That works. A stay that starts just after it. That works too. A stay inside that month, at a different home. That fails. And partway through, we stopped the host. When it came back, the hunt didn't start over. It finished the requests it had left, and filed the report. So you don't come back to a note that says quotes are broken. You come back to the edge. Any stay that touches that month fails, at every home. And everything on either side still works. That edge is worth more than the error. It tells you how big the problem is, and it tells Main where to look. Watch for one trap. Small, fast models hunt well. But if they type out the report themselves, they'll sometimes list a request they never made, or put a working one under failures. So don't let the model write its own evidence. Give the hunter one tool for making requests, and have that tool number every one. Request one: what was asked, and whether it worked. Request two. Request three. Code keeps that list, not the model. Then the report can only point to those numbers. This one replayed the error. These failed. These still work. If the model invents a number that isn't on the list, code drops it. And code fills in what actually happened, and does the counting.

Main takes it from there

00:15:39,073 --> 00:16:20,752 [John Lindquist]

Then ask Main for a Hunts widget that shows each hunt as a card. One short box: what fails, what still works, with a numbered request behind every line. Select the card and send it to Main. The hunter only reports. It never touches the code. The card hands Main the report, asks what it would check first, and tells it not to change any code yet. Then you decide what happens next. So look at your workspace after a run like that, on your own machine while it's awake, or on a server later. A silent error woke a hunt and turned into a map. New tickets were sorted as they arrived, with money at the top. And for guest questions, a draft is one request away, written from facts your code looked up. All of it at a glance, and every real decision still yours.