Skip to content
Linkdrop
Links

MEGA.dev live Q&A with John Lindquist

John Lindquist presents ideas about building workspaces around agents, then answers questions with host Wojtek. They discuss agent coordination, custom models and evaluation workflows.

videoJohn Lindquist · hq.mega.dev · Saved Nov 19, 2026

mega course, agent orchestration, agent evaluation, generative ui, multi-agent systems

Transcript of the recorded MEGA.dev live Q&A with John Lindquist (Wednesday, Nov 18 · 102 minutes), John Lindquist. The recording plays in MEGA Student HQ (needs a login).

0:00 Wojtek: Oh, it's nice to see you guys.

0:02 Guest: Hello.

0:04 Wojtek: Let us know if we can, uh, see and hear us. We're starting our, uh, final live session with John Lindquist today. Hello, John.

0:14 John Lindquist: Hello, everyone.

0:17 Wojtek: Guys, as usual, this session will be 90 minutes. You will, of course, get the recording and, uh, chat and everything. So in case you are not able to be with us the whole time, don't worry, you will miss, uh, nothing. Um, uh, this is a Q&A section, but John, for today, you have a short presentation prepared as well, I believe.

0:39 John Lindquist: Yeah.

0:40 Wojtek: Yeah, that's cool. So we'll start with that. So don't worry, guys, uh, we will go through all the questions as well. And if you decide you want to add some questions on Discord, feel free to do that. Uh, our questions list will, uh, update. Also, if you want to upvote during the live session, this time all the upvotes will also work as well. So there will be like this small progress bar that you will see how often the questions, um, sort of refresh. So don't worry if you add an, uh, add an upvote to one of the questions that you like. Just wait a mi- like two minutes, and it will update on the screen, um, as, uh, as well.

Um, and yeah, uh, John, uh, I think, uh, if you want to start off, uh, with the presentation, uh, we can, we, we can just, uh, go.

1:31 John Lindquist: Great. Perfect. Thanks, Wojtek. All right. Screen look good? Okay. So I'd like to dive in since I'm the last week, and I feel the need to kind of share where things are going as well as where things are today. Since everything's moving so quickly, I figured I'd put together a little presentation of, um, what I think is coming up immediately that, um, everyone should be aware of and kind of start playing with if you, if you get the chance to. Um, so kind of beyond software factories, that overloaded term that, um, is really-- has a million different definitions. So we're going to start with a quote from Tariq. He, uh, was the lead DevRel for, uh, Anthropic and did a lot of presenta- presentations on Claude Code, and he just posted this a couple days ago and said that, "Software factories feel like a temporary problem.

The models will learn to build them for you." And, um, the more interesting question to me is how to build a workspace. So you probably noticed throughout the week where I talk in a lot of general terms about workspaces and try and throw a lot of ideas around workspaces, because I 100% and, and like I'm aligned with Tariq that software factories are a term we're using right now of kind of the wiring between, uh, agents that for little things that solve little problems throughout your, uh, throughout your project and how to, uh, fix all of those frictions as, as we've gone over. And like this is already happening where you can take any one software factory problem, say, "Use this as an example, uh, apply it to our project.

Uh, let's-- we'll run some tests on it. If that worked, then, uh, that's great. If it didn't, like spin up five variations and we'll try what works. Then I'll pick the one that works best, benchmark it a bit or whatever, and then figure out something that works well." And then in an afternoon, you kind of have a software factory that handles a variety of tasks for you. And that's, that's something you can do today. So yeah, it-- there's a lot of agents and factories that can built-- be built in a single afternoon. Um, for example, like this is a Rent-a-Place, uh, workspace. We're covering this Rent-a-Place, uh, product during the week where I just asked in an afternoon, "Can you spin up like a support desk for Rent-a-Place and have it be, uh, driven by agents where these tickets come in, they're assigned to me?"

I can go to refund timing. I can type, uh, summarize in here, hit enter, it spins off over to an agent, summarizes the ticket, and it fetches all these tickets from GitHub issues, like all the sort of stuff where you'd spend, you know, weeks writing the implementation of it. Like all of that was just done in an afternoon. Um, these agents are running off of my subscriptions and, um, eventually it will come back. And now I have a thread based on those tickets and like it all just worked the first time. Like it's be able to one-shot so much of this stuff, and if something doesn't work, you just iterate on it. So like there's so much we can build in an afternoon.

So what does it-- what does that mean for us? Um, it means that not one size fits all. Again, this is from Tariq's post. He's showing how he built a little kind of, um, mirror board or little canvas where you can scroll around. He's building a game. Obviously, a lot of people are building games because they're very visual. Um, but a little board he can work with as a workspace for the way that he's working and the way that he can investigate animations and clips and how things all work together for his game. Um, the one that I've been building the most recently is if you look at some of the, um, like the, the workspace I built around building my course looks something like this, where I can go through, I can draft together some scripts, I can scroll through, I can leave notes, I can hand that off to my agent, I can record straight from here.

It'll parse out all the audio. Like it does all this stuff for me, and I didn't look at any of the code for any of it. And it aligns all of the, the words to the, uh, to the audio, and I can re-record. I can select little sections to re-record. Like I'm building this workspace around the output that is customer-facing, is, you know, facing you guys. Um, so whatever is, whatever it is you're building, like think of a way that you can build a, a workspace around it. Um, so workspace is about, uh, who's working with it, how are those views being shared between desktop and web and mobile. Uh, glossaries are always going to be a big, uh, a big thing here, and, you know, which models to pick for it, kind of the big pillars there.

I'm going to speed through this a bit so that we can get to the Q&A faster, but, um, so if I jump over any slides. Um, but we're kind of in a, a bots and dots era right now, where you see, uh, Grok and OpenAI and Devon Bots and all these, uh, all these approaches trying to surface agents to the top, where you have a collection of agents and you talk with all of them. And, like, it's still something people don't quite want. People just want a single agent they can talk to, and they want that one agent to understand all of those tasks, where you have an agent and a workspace. And, like, you can definitely use Grok, and you can use-- Like, it's, it's so close to something that feels good, where you have all those different threads, but it's so easy to lose what's going on in them because you don't have a lot of visualization, you don't have a lot of feedback, you don't have a lot of glanceability in all of them.

Um, so when you set up a main agent and you set up a lot of things that surface visualizations to you, you can set up things that you can ignore. They can just be, "Do I want to take action on this or dismiss it?" Like, little notifications and little things that seem important. Um, and that's where some of the tech starts coming in, and this is where, uh, I mentioned HerderCore as something we could use. Um, HerderCore and some others have built a, uh, kind of this core module that has schemas, it has, uh, the wrappers around the, uh, the agents that are running so that it knows when agents are running, when they failed, uh, when they're done, when they're in progress, like actively working on tasks.

And so you have this, uh, this module that can send events and you can interact with, that can talk to any of the views. So you get this shared view across mobile and desktop and any sort of workspace you want to, and it can turn into this shared view between you and your clients. You kind of have this, like, this source of truth for everything. And then another emerging technology, um, that I'm paying attention to is, it's a library called Exact, um, from Expo. And this is the best one I've seen so far for, uh, the era of mobile, web, desktop, Android, Windows, that deploys to all of them. It does it based on Rust, and it does it based on this language called Contract.

And, uh, Contract is like an abstraction where it, um, it has that one kind of DSL, whatever you want to call it, that can compile to all of them. And, uh, I've used this success-- excuse me, successfully so far for building out variations of things, and you're able to one-shot, you know, mobile versions of things or desktop versions of things. Uh, definitely something worth looking into, where you have-- This can be the, the client side, something like a HerderCore that hosts the, like, the agentic brain can be the, the backend, and like those kind of concepts of, like, backend and, and client side are going in this direction as well. Um, and then just recently, adapting models, like none of us would think, "Let's build a model for our product."

Uh, that's something we should all start thinking about more, like, what if I had a specific model that is just for me? Um, how can I start training models that are just for our own products? Um, this is definitely like bleeding edge sort of stuff, but, um, realizing that you can start training, uh, like these models that are as good as like the Opus 4.5s or, you know, model, the frontier models from six months or a year ago that can behave really well and tackle a lot of the tasks that are specific to, uh, what, what it is you're presenting. And then even crazier than that, uh, there's been reports of frontier models, like manually tweaking the weights or tweaking numbers inside of, uh, inside of the models.

They'll go in and they'll just start hacking away at models, um, and actually getting good results, where you just hand it a model, say, you tell your, uh, you know, your fable or whatever, "Make this model be better at math," or, "Make it, uh, more customized for our specific game," or whatever we're building. And it can just go in and rewrite the model itself. Like, the frontier models are getting that good. Um, and then, like, I would say that one of the, one of the key takeaways from this week is that whole idea of, uh, having your eval loops in place, setting up some sort of lab, setting up a reinforcement learning of your own so that you are s- you're starting to train your own agents, you're starting to figure out, how do I make custom agents that fit into the specific domains in my, uh, project or product?

And this can be done in a variety of ways, um, and it can even-- You can have your frontier models just build out your own custom labs for the exact things you're doing, which is-- which can even be better sometimes. But just having that mindset of, how, how are we going to start, uh, doing our own reinforcement learning, uh, for our product? Um, this one's huge for me, is I absolutely adore, uh, this, this workflow that has emerged. Um, you could kind of call it video hill climbing, where the, uh, if you remember the black box testing lesson, you can hand your product over to a series of agents, um, and you don't let them look at anything.

So say it's a web-- Say it's the Rent-A-Place. Um, you hand it Rent-A-Place, and you tell the agents, um, "You can only look at the screen. Just take-- You can only look at the pixels. You can't even use, like, the agent browser tools to look at the, uh, accessibility trees or anything. You just have to look at the pixels, and then try and accomplish some of these tasks or think of tasks on your own." And then you just, uh, record what they're doing and see where all the tasks they inferred and where they failed at all of them. You get this big report of, "Well, I wanted to do this, but I ran into all these issues," or, "I saw these visual glitches," and, and, and on and on.

And you get this report of what they tried and what their experience was like, and then you take that video recording, you take, take that report, and you throw it-- throw that into like a Carbon, um, which is just an incredible model for, uh, video processing. And because it's multimodal, multimodal and actually supports reading videos where compared to the, uh, the Fables and the Astras, they take individual frames out of videos, so they don't-- They're doing images and audio, whereas, uh, Carbon actually understands like the video holistically. It's, it's a really notable difference if, if you get a chance to try it. Um, and then Carbon could be a model that you give it the video and you give it your code base, and you say, "Hey, here's everything that failed."

Like, "Here's-- I noticed some content layout shift," or, "I noticed the pop-up covered something," or, "I noticed, um, when they typed in this, some of the fonts were a little bit weird." And it can notice all of those things and just start improving on it. And you set up this loop where your agents start using your product over and over again. Carbon starts watching the results of the videos and starts polishing and polishing and polishing until Carbon doesn't, uh, notice anything anymore. And it's, uh, this is-- it's blowing my mind what this is capable of, especially with anything multimedia or games or anything. Um, yeah, just what I just said. Uh, this is just videos of what I just said.

Don't wanna linger too long here. And then, uh, System 1 models are definitely, um, if you have a chance to like explore one thing right now, um, I, I would say it, like spend time with Jev and the System 1 models, where you have instant, uh, almost free intelligence that are able to make decisions quickly. Um, your customers are gonna be coming in more and more with agents who are gonna be giving like looser and looser terminology, um, or just phrases and whatever. Um, and I know that we, we thought so long about MCPs and defining contracts and all this sort of stuff to expose to other agents. Um, the System 1 models can just take that raw string of text that you get, whether it's a few sentences or a payload or anything, and so say you get a payload that talks about rescheduling a booking, and the System 1 model in under a tenth of a second for almost free could align that request with like your booking domain agents, or it could align it with the-- where it needs to go to as far as filing an issue.

It's able to just instantly sort and assign it to the things where it needs to go. Um, and in my opinion, it's ch- it's challenging a lot of the, um, the structure and architecture we've built in around agents communicating with agents because it's, uh, it's pretty much instantaneous, and it's able to make decisions for you rather than having that layer of, um, the, the facade or whatever you wanna call it between two agents. You can have this intelligence between the two agents that, that routes things where they need to go. And, um, you can build incredible experiences because these turn into real time. Um, they're turning into-- I think I have a slide. Yeah.

Uh, turning into like every keystroke experiences where you type something like, and it knows where to send that sentence. Um, and it can always give you like what happens when I press enter. In this example is you type one sentence, you hit enter, it does one thing, or if it infers something else, it could do something else. Um, basically like what you're looking at on the screen, um, is always in context. And then I think I have... Was there a slide in there? Nope. Um, just a couple of days ago, uh, the Google released their own-- I'm starting to sound like a Google shill, and I think I'm wearing a Goo- yeah, I'm wearing a Google hoodie.

I'm sorry. Um, Google released their own System 1 model that also supports video and supports te- or supports images. So you're able to send, uh, images and video, and you can only imagine the experiences with, you know, wearing glasses or phones or whatever, where you start like really interacting with the real world and getting screenshots of sites and like with your glasses, looking at a product or whatever, and sending those to a System 1 model and sending that to your, to your agents and your, uh, your workspace. So a lot of things to play with there and to, um, to really get, get into the future of this. Um, other than that, let's... Those are just some summaries.

Yeah. So let's get into the Q&A from there. Um, I just want to, um, kind of drop all of those, uh, make sure that gets at least touched on during this week. So, um, I will hand the screen back over to Wojtek. Stop my share.

17:41 Wojtek: Okay. Uh, so now let me share my screen, and we'll have, we'll have the questions for you guys. Well, actually, your questions. So they're in the order of, uh, upvotes, John. So we're starting with the ones that are most upvoted.

18:06 John Lindquist: Okay. Um, want me to-- I can ask-- answer the questions in chat real quick. Is, is there a priority between chat and the Q&A?

18:15 Wojtek: No, no, no, no. Sure. If you see any good questions on chat, just go, go with them.

18:19 John Lindquist: Yeah. I can, uh-- I'll share the slide deck. I'll drop it in the Discord. Um, with Astra and Soul, I was on the, um, the pre-release with Theo. I had a pretty similar experience with it. Um, it just came down a lot to the very difficult to com- to communicate with and very, like, obtuse language that it responded with, but still, like, an excellent model for doing work and for iter-iterating. Um, you did have to, like, prod it a bit more. But overall, like, it was fantastic quality. It's just kind of like the experience of using it, um, which is weird to say. The experience of using the model just fell off. Um, uh, prevent Pi from running dangerous commands.

Um, yeah, you can, you can set up configuration-configurations for that. I don't... Um, videos. Okay, I think that's good. All right, so Q&A. Uh, how do you start a greenfield project, and do you recommend spec-driven development tools? So recently, um, it depends on the greenfield project. Um, I would say, like, I would start it with, um, that exact library from the Expo, uh, people, where I would start it as ambitious as possible, where, like, you want it Rust, you want it performant, you want it on the latest, and, uh, like, whatever experience is going to help you stand out from the crowd. Um, make-making sure that you're finding your specific niche and finding that thing that is going to, um, draw attention to, to your product.

As far as the spec-driven design, um, I would definitely start with, uh, like a glossary as, like, the first document I'd make for anything, where you sit down with a-- you sit down with one of the agents, and you describe your product, and you have it create a, um, a system of terms that will serve as the, like, the language source of truth between you and the agent, and the glossary becomes the, um, the way that you and your agent communicate about the various, um, pieces of the software. Because you're going to start getting into a lot of, um, interesting language as you-- as it progresses, and it's great to stay grounded from, from day one there.

Um, look into domain-driven design as, as something that can really help guide you there. And then, um, I would focus on using-- Like, one of the reasons to use Rust is because, like, any sort of code failure will be surfaced right away. Um, so another good reason to use Exact. But if you're using a TypeScript, like using something like an Effect library or things that really harden the code that you're never-- that you're probably not going to look at anyway, um, just, like, forcing it into the patterns with, like, an XState library or anything that, uh, makes the code as bulletproof as possible without you looking at it. Um, but Exact covers a lot of that stuff out of the box, so.

Um, so I would say bulletproof code with, um, a glossary is where I'd start. And, um, just, like, dig into domain-driven design a bit. Um, ask one of your agents to walk you through. Like, initial prompt is, "I want to follow domain-driven design principles as I'm setting this up. Uh, walk me through setting up this project. Like, walk me through the mind space of setting up this project that way." Um, so yeah, I-- That-that's how I'd answer that now. I think spec-driven development is kind of falling out of favor, um, with some of the latest. Like, Opus 5.5 at Medium is, like, my go-to for everything right now, and it's just so good, but I don't...

Yeah. Um, and yeah, the other, like, the last thing I'll say about that is make tons and tons and tons of variations of every idea. Like, anytime you have an idea or you want to start building your, your product, tell it to spin up, like, nine variations, put them all on a grid, and I'll pick my favorite one. And then you find your favorite one. Take that and make nine variations of whatever, and then I'll pick my favorite. And then once you get to something you like there, you can tell it to make modular swappable pieces of my application in line, and I'll-- and make it so I can swap between them so I can see them in line, and then pick from and, like, easily cycle through with the one through nine keys.

And you can pick, pick from there as well. And so, it can just continuously learn from your, like-- That creates a history of your preferences. It creates a history of, um, of the, the, the direction you're going in, because the more, um, the more the a-- like, the more your past conversation history can show what you rejected as well as what you accepted, the more it can help, like, go with you to your final goal. And often, if you do spec-driven design, um, you get that-- You only get that mindset of, "Here's the restrictions." You don't get that, "I don't like this. I don't like that, that, that, or that. I really like this."

Okay. Uh, what's the real value of terminal-based agents when Claude Desktop or Codex handles so much for us? Yeah. Um, this is a tough one because, uh, terminal-based comes down to kind of your history as a developer. If you've kind of grown up in that space where you're, you're used to managing multiple machines and remote machines and managing hard disks and managing all these commands and writing custom zsh configs and, um, configuring things in that way, uh, you can be extremely, you know, power user about that, and you can be kind of a, the system ops person who manages all of that. And you know that any, like, dev task that's, that's thrown your way, you have good foundations for it.

And, um, you can have-- even have your agents, like, create all of those configurations for you. Like, it-- there's so many configuration files I don't, I don't even look at anymore. Um, I'm, I'm not arguing for one or the other. I would just say that the terminal will always be more configurable for someone who's gonna be a, a power user. Um, it can do a lot more than people give it credit for, because everyone always felt so stuck behind how difficult, difficult it was to configure, because it's meant to be, like, the thing that works on every machine everywhere. Um, now that it's trivial to configure, you can do all sorts of motion and animations and components and, and things inside of the terminal, and it's not an issue anymore, and it works across every machine, um, regardless of OS and, and everything, um, and regardless of, of phone or...

Yeah. Um, now that-- Again, I'm, I'm still not even arguing for it, because I realize that the desktop applications of these are, are incredible. Um, I think what all of the authors, uh, Angie, Kent, and Theo are doing are incredible. Um, I will always be someone who's going to, um, want to tweak everything, and that's just in my nature. And, um, and I'm-- I love that I have a massive amount of history in my, um, in my Pi sessions of all of the things that I rejected and all the things that, that I want to do, because I can pull from that constantly to say, like, "I was customizing this and customizing that and customizing this, this, and customizing that," and, like, learn more about me by everything I'm trying to customize.

And when you take something off the shelf, um, that's awesome. It's just that you don't get, you don't get a chance to really express your preferences in tho-those agent sessions because you're not customizing it at all, and you're not giving the agent the chance to learn about you. Um, but different, like, the desktop apps are great for a lot of, you know, building games and building visuals and building, like, there are a lot of reasons to not choose the terminal because you just never need to u- uh, use it or anything. Um, all right. Uh, I think that answers that well enough. I'm not looking at chat. I should look at chat. Um... Okay, let me take a chat.

Uh, examples of customizations you couldn't live without. Um, so any sort of, uh, ZSH configuration, uh, keyboard shortcuts, and, um, the way to spawn various, um... One I definitely couldn't live without is, like, spawning different instances of terminals where I can black box run them through labs and stuff to, um, run agents against them. Um, all that sort of stuff is, is critical to me. And, um, just the-- any sort of interaction with it, um, and yeah, the ZSH configs. Uh, my ZSH config is insane, um, split across dozens of files. Um, but yeah, the-- Like, if, if you look at, uh, like, the Codex TUI or the Claude Code TUI or, uh, any of those, and you look at the sheer amount of, uh, flags and options and configurations they have when you run them and you start them up, um, it's-- I, I strongly recommend telling your agent, uh, any agent you have, like, "Run Codex dash dash help," or whichever one you're using, or "Pi dash dash help," and say, "I'm interested in learning about all of these configuration options.

I don't think I'm using this to its full potential. What am I missing? Like, what could I build here that would be cool?" And, like, diving in that way, and you'll discover things as far as tool usages and model switching and orchestration and things that if you just kind of ac-accept, um, something off the shelf, you, you wouldn't discover otherwise. Um, so just, like, keeping curiosity, like, maximized for the future, I think is, is a good strategy. Um, all right. Uh, how do you usually kick off hill climbing loops, starting with target metrics-based prompts or an auto res- Yeah, so, um, for hill climbing, I-- Everything I do, I start off with, um, I just ask the agent, like, "Do the best you can, and I'll review the first few passes."

And then I learn from the failures in those how I want to, uh, refine the actual end goal. And sometimes that takes a while. Like, it can take days where you're just, like, rejecting garbage and... But all of those things help me surface the edge cases and the, um, kind of what, like, real-- Since models are trained on the internet and such, like, it's what real people would expect. And you get to see all these different points of view. Uh, points of view is probably the wrong word, but agents, points of agents? I don't know. Um, and a-as they kind of reach the, the top of what they think is the hill, you can, um, strip that all down and ru-run it again with, uh, with the targets you have in mind.

Um, and that's, that's the general approach I take towards everything, is I just do the thing and let it fail and iterate. And once you have a strong grasp on all the failures, then I feel ready to write the targets. Um, I, I think it was one of the Codex DevRels said recently that to, uh, to develop taste, you gotta eat, which I love. Like, you have to go and just- Try everything to know what you like. And I think we're so-- we have such a limited understanding, and I'm talking about everyone in the world has such a limited understanding of what models are capable of. Um, like, even if we were still on, you know, GPT-3 or whatever, like, we never-- before anyone really got a chance to fully understand it or how to implement it in workflows, like, we just bumped up to the next model.

And, like, we just have no idea the capabilities of these. We just kind of do our best with what we have and wait for the next model. Um, so as much as you can explore them and, uh, kind of pull that stuff out of them, the more you can kind of level up your own taste and your own, um, preferences, and the larger the history is of, um, of what those are. Um, all right. How do you approach reflection or dreaming? Do you have any agents reflect on your sessions automatically? Uh, no, I don't have any automatic-- I have very few automatic, um, uh, like consolidation tools or reflection tools. I, I run those kind of at checkpoints, like once some work is done.

Um, and I, I-- to me, that's some of the most important work that it's doing is, like, those learning moments where you're going to steer it more towards your preferences and more towards the goals. Um, so reflection and dreaming are def-- are for me very, very manual and, um, something I, I look at, especially when it comes towards, like, making decisions in the product. Um...

32:10 Wojtek: Yeah, if I, if I may suggest, let's, uh, because there are a lot of good questions on chat, and I think, uh, when we have a relevant question to the question that is on screen, we should go with it.

32:23 John Lindquist: Okay.

32:24 Wojtek: But let's focus on the ones that are upvoted, that are upvoted by the community on Discord. And if we go through them, then we can go through all the chat questions, and I will be marking them with a blue, you know, reaction icon with a sort of-

32:39 John Lindquist: Got it.

32:40 Wojtek: -triangle or whatever.

32:41 John Lindquist: Okay. Uh, okay. Focusing on Q&A from here now. Uh, what does the next decade hold for senior software engineers, and how should we prepare? Um, this is, uh, an impossible question to answer. My best guess would be that, um, the-- like, more and more things disappear. Like, the, the-- what's-- there's some quote about the, the difference between science and magic is that science is what you can understand, and magic is what you can't. And everything's going to start feeling like magic, um, and, like, you're just going to have to turn into a wizard and understand magic, right? It's like... I don't know what sort of, um, glasses we're going to be wearing or what sort of implants people will have or what sort of whatever.

Um, there's probably going to be a lot of, you know, to make an omelet, you have to crack a few eggs, and we're going to go through a lot of security issues, and who knows what that, what that's going to turn into. Um, and there's a whole lot of, uh, politics that comes into all of it. So for-- so, like, stay curious. If you're an engineer, stay curious, stay on top of things. Um, stay kind of aware of the capabilities so that you establish yourself as someone... Like, I, I tell this in all of my, uh, personal workshops. Like, you want to be the noisy person in your Slack or your-- whatever your communication channels are of, uh, the person who stands out as the expert on this sort of stuff.

And so, like, hop in and address those and, like, post articles and, like, be the person like, "Oh, that's the AI nerd again." Um, like, be that person who, in the company, like, if you're in a big company, be the person in the company that keeps on tooting the AI horn. And, um, then, you know, that'll-- people will know you as that, and they'll, they'll want you around because you'll be the person they go to. Um, so, like, make yourself indispensable for, you know, foreseeable future. And if you're building your own thing, um, try and find your, your niche that separates you from others. Um, the OpenAI and Anthropics are going to keep on, uh, eating all the lunches of all the big, like, task tracking and, um, communication and all those, like, huge things that are, like, global products.

Um, I think a lot of us as, uh, startups are going to have to focus, like, find little groups and their little pains with agents and address those. Like, don't worry-- don't try and fight against the, the big labs, um, unless you're, like, unless you are way smarter than me, um, and you probably are. But, uh, in general, unless you're, like, brilliant and you understand the ins and outs of some of these models better than they do, I wouldn't try and, like, build something that they could possibly build or something that they wouldn't focus-- I'd build something they wouldn't focus on. Um, all right. That's-- I think that's okay for that one. Um, where should secrets live so the LLM never sees them?

Uh, I would say number one, wherever your specific harness recommends. Like, each individual tool, like, has a recommendation, recommendation for it. Um, I put mine in 1Password, and, uh, 1Password has a CLI. I think it has an MCP now, um, that requests my fingerprint anytime, um, it's trying to access something privately, and then I can extend that to be like a 24-hour permission where it doesn't have to prompt again. Um, but that's just for the local agents for, um, for secrets. For hosted agents, that's a follow best practice of whatever, um- GitHub or whoever you're hosting with, um, like, yeah. Whichever host you're using, they, they should have guidance around that. Um, I, I used to work for Vercel.

I'd-- like, they have a very, um... They put a lot of work into their, uh, s- like secrets and security stuff recently after, after an incident earlier in the year, um, which they-- which everyone's going through. Yeah. So just, I, I, I don't think there's a one answer fits all there, but personally, uh, for my personal stuff, it's 1Password with the, the tooling that can expose the agents. I think that's good. Next question. All right. Automate AI model selection. This one's so fascinating to me because, um, if you look at some of the benchmarks that the labs like, uh, maybe labs is the wrong word. There are companies like Factory and, uh, Cognition and Microsoft and others that are combining models, and they, like, they'll call them fusion models, and they'll benchmark them against, um, like they'll take something like a, uh, a GPT Soul and a Sonnet, like combine their two responses together and summarize them and have them, like, work together in certain interesting ways.

Um, and those will often benchmark better than, like, an Opus, um, and they'll say it's cheaper and faster and, um, more capable because you're getting multiple opinions from multiple models. I know this doesn't, like, address this, uh, question specifically, like, as far as coordinators and workers, but I think it's relevant as far as, um, there's just so much to explore there that, um, there's no right answer right now, and I don't know if there ever will be. Um, this has been tried in the past multiple times and hasn't worked out very well, but that doesn't mean that it won't. And that's the-- one of the hardest things in, um, the AI field is that for, for years, like since, uh, like we first got these tools, we've had incredible ideas and that have failed over and over and over again, but that doesn't mean that they're not correct or that they won't eventually, um, be the solution.

It's just that our ideas are out, out in front of the model capabilities. So as far as, like, um, model selection and multiple workers, I would say right now just let the, like, whatever provider you're using, um, what- whatever sort of sub-agent system they're using, uh, just re- rely on that. Like, they've benchmarked that, they've optimized it for, um, for cost and performance and everything. Um, and like, yeah, the, the other trade-off is like if you have unlimited money, then the answer is different. If you're trying to, uh, save every penny, then the answer is completely different. Um, in general, like right now, I just use Opus 5.5 Medium on everything, and then, um, for specific tasks, I will, uh, I'll use Gemini's newer models.

Um, I like Carbon for video. I like the Hydrogen stuff for, like, interacting with sites and understanding. I, I think the Gemini models have a much better, like, understanding as, like, how would a human approach this problem. Um, that's just my, my feeling right now. Like, I don't have benchmarks. I don't know how to benchmark that. I don't have a bunch of humans to, to measure that. Um, and then, uh, Astra is very technical for me and, and does great work there. Um, but it's really kind of Opus 5.5 Medium for pretty much, for everything. Um, yeah, that's-- I guess that's answered. I wish I had a, like a-- I wish there was a correct answer.

Uh, tmux versus Herder is Herder, uh, is-- ha- has the, the layers around, like, the, the status of the agents and can communicate, uh, that sort of stuff, um, to other clients, and you're able to build up clients that way. So if you look at the way they're architecting their, their stuff, um, it's, it's definitely standing on the shoulders of giants, right? Like, Herder is, is based on the, the multiplexing capabilities of tmux. Um, so, like, if you want to recreate Herder yourself with, um, with whatever agents, like, that is totally possible. Um, that's one of those tech choices where, um, like I mentioned with the terminal, I get deep in my zsh config, I get deep in the keyboard, um, customization.

I accept Herder as a good solution, right? If, if you're someone who doesn't accept Herder and likes tmux, you can go for it and configure that to whatever, how, how deep it is you want to go with it. Uh, my personal choice is I, I put my efforts elsewhere. I'm not saying that's more productive or better or anything. I'm just saying that's, like, my preference. If you are, like, the tmux expert and you find all these gaps in Herder, then, like, then go for it. Um, what, um, what I would say is that, like, with the Herder app and, uh, the mobile app is decent, it's coming along. Um, but with the Her- Herder app and Herder Core and their UI kit that they have, um, I've been able to build any sort of, um- Widget or customization.

Um, I have a, um, command-- I need to publish my command bar. I have a command bar that can search every pane that's open and shows like a status and makes it really easy to, like, navigate between all the panes and, um, yeah, there's, there's some really, really interesting experiences you can build because they just give you a UI kit that, um, can spin up any sort of, like, UI you want. Um, all right, so that, that can be answered.

43:14 Wojtek: John, uh-

43:15 John Lindquist: Yeah.

43:16 Wojtek: Uh, I see people asking if it's possible for you to go through the descriptions of the questions, uh, as they sometimes, you know, have some nuanced details.

43:26 John Lindquist: Uh, sure. Uh, on this one? Uh-

43:29 Wojtek: I, I think it's a general, um, ask.

43:33 John Lindquist: Okay.

43:34 Wojtek: But take a look if this one requires any details, and I will just check it as answered if it's okay.

43:41 John Lindquist: Okay. Uh, mind commenting the advance of using Heard versus tmux and create something bottom up? I saw a lesson today on the advantage of using HC file slash commands. I wonder if it'd be better to create some dedicated scripts to create folders. Yeah, I, I think this is answered based on what I said. Uh, yeah, Victor, if there's a specific question you want me to follow up on, happy to. Um, how do we let an agent run overnight without breaking system? In a software factory, how do we handle system permissions so the agent can run fully autonomously but can't harm the system configuration or itself? Should permissions be defined per agent? Uh, yeah, anything autonomous, uh, should always have permissions defined.

Um, the only, the only time, like, you lessen the permissions, uh, in, in my case, is the, the agent that I'm interacting with directly. Every other agent you build should have, uh, guardrails and walls and everything around it so that it, it's the expert of its own, own domain. Um, and yeah, like it's, it's in, it's-- At the end of the day, it's still inference. Like, it's making, uh, choices and, and writing text based on what you give it. And, um, each agent and its kind of the role that it has should have its, its own system prompt. It should have its own, um, rules on what it can and can't read, what tools it can call, and, um, yeah.

Yeah, I would say that's, that's definitely something to-- when you're black box testing something, to tell that to the agent, um, that is using, uh, or not implementing, but like actually using that agent. Say, "Hey, try and get this thing to, to break out of its system to, um," inside of a sandbox, obviously. That's what, you know, the big labs do is they put an, an agent in a sandbox and tell it to try and break out or affect something that it shouldn't, and kind of sniff those out. Um, and it's, it's honestly like a lot less work than you might be thinking to, to do those. Um, you can just ask an agent to spin that, a lot of that stuff up for you, and they're, they're good at it, good enough at it now.

Um, but, but yeah, it's a, a lot of rules and restrictions around them, um, and if they need it, loo-loosen the restrictions, but you can usually start them out as pretty, uh, pretty buckled in and, and tightened down. Uh, I think that's answered.

46:15 Wojtek: Uh, Victor pointed out a, uh, follow-up question in the chat, if you could take a look.

46:20 John Lindquist: Oh, uh, yeah. Pi specific to this question, how do you prevent Pi from running d-dangerous commands? Whoops, it's cool. Is there any consensus on an extension that adds something similar to Claude Code's auto mode? Um, yeah, I would... Off the top of my head, I'm not aware of an extension for that. Yeah, I'll, I'll have to get back to you on that as far as, like, what the community standard is. I don't, I don't have that off the top of my head. Um, I do... Like, Pi just has so much configuration, it might not even be an extension. It might just be asking Pi to, like, what are your configuration options around this? Um, I'm trying to think of the last time I did that for my main agent.

Yeah, I'll have to, have to get back to you on in Discord or something. Um, separate agents or one Claude with sub-agents? Do main, critic, and the domain agents need to run as separate instances, or can it be one Claude Code session that reads agents' files and spawns sub-agents on its own? What are the pros and cons of each approach in practice? For example, for a team that just uses Claude Code. Um, so critics and domain agents are, uh, you could call them, like, the most abstract level parallel agents that run separately from main. But they're, um, like a critic observes the main agent and can help summarize and give you information about what main is doing behind the scenes.

Um, but main can never control critic. Like, main is separate from it. Um, similar with domain agents, they, they can be run and spun up whenever something in its domain is touched, like if you touch a file in there or like a API endpoint triggers something. Like, they, they wake up on a trigger. And main can, like, cause that trigger to happen, but it's not gonna invoke one of those domain agents. Um, so when it comes to, like, spreading work across a task, sub-agents are the way to go, and, um, it can build and orchestrate its own sub-agents. And you can have sub-agents, um, like we don't really cover in this week, like custom sub-agent prompts.

Um, oh, you can define those. Uh, a lot of the harnesses just spin up the sub-agents for you, but you can, like, predefine those. Um- And at least in Claude Code and Pi, and I'm sure most of them. Uh, but tho- those are ones-- sub-agents are directly controlled by your main conversation, and they report back to the main conversation. And if they fail, main knows about it, and like they have that parent-child relationship. Um, so when talking about like the, the definitions of critic and domain and, and all of those, those, those can run, um, critic can, can run autonomously and just sit as a, a side helper. Domains can run on triggers and feed information into main, but main doesn't, um, interact with them otherwise.

For, uh, compared to sub-agents, uh, your main conversation will create those and run them and get the reports back and close them and spin them up again, um, whenever you've like fed it a really big task and you wanted to spread that work around. Um, and that, that can be a decision like either let the harness make that choice for you or, um, or you can even talk to the agent first about, "Hey, I have, um, I have all these sub-agents, like what's the best way to do this?" And it's-- this is another thing that varies from, um, harness to harness and model to model. Um, like I, I was reading recently about how people are impressed with, uh, Gemini's ability to swarm a bunch of hydrogen agents and to, um, tackle long tasks, tasks that way.

Um, I don't know if that's just them catching up a bit with what Claude Code has done in the past or if it's actually better. But, um, in general with sub-agents, I don't really worry about them too much other than saying, if I think it's a big task that can be broken down and done quicker, um, if you spread the work around, um, then I ask for sub-agents. Or if I like want it working in the background so that I can still talk to the main agent, I'll hand it over to a sub-agent. Um, that's, I think that's the extent of it. Um, heard of TUI versus the desktop app. I also started using the desktop, desktop app, but I asked my agent to compare available TUI packages with the desktop app.

It told me something around 8% work only on TUI, and the remaining 20% work for both and some for the desktop app. Would this make you change to TUI? If not, why not? Uh, so I picked the desktop app due to the, um, extra UI elements around managing, um, the spaces and the widgets and things of that nature, where the, the TUI, you know, like now thinking about it, could probably do most of it. But like the, the capabilities of, um, of some of the things I've done with command bars and widgets and whatnot, and it's-- and, and what I'm doing have just felt like the, the GUIs I'm building for them have felt much more like these native components that I'm building out.

Um, I honestly haven't tried, tried building those out in the TUI, but I, I like, like I like having that layer of, um, that layer of separation between the kind of the, the container and then like the view that's running inside of it. If you think of like the desktop app as the container and then the terminals as the view, that way their renderers are separate, and they don't-- like if one goes down, the other one doesn't. Um, but I'm, I'm not going to take a strong stance on that one. Um, yeah. You could probably do some really cool things with the TUI that I, I haven't explored yet. Uh, I think that's answered things.

Um, orchestrating at scale, how? Um, what a great question. I really enjoyed the combination of the domain agent demo and the lesson on orchestrating workers. The idea of giving a domain expert its own reviewed memories and glossary, then waking it when a relevant file is touched clicked for me. I also like the idea of workers coordinating their own context instead of a person managing every individual conversation. I'm wondering how you'd extend this to a large organization. Dozens of developers work- working across several domains and the same code base, often in separate local sessions. In our case, domain boundaries aren't always clear either, and some parts of the code base have significant cross-domain dependencies. I understand the lesson proposes keeping task progress and files or a database, making agents aware of relevant running sessions, observed activity, and sharing new domain agent knowledge in the repository.

What I'm still missing is how those pieces connect across many people's independent environments. How would you discover and observe threads and workers across developers, route relevant events, and avoid overwhelming everyone with activity from other teams? Where would shared task state and useful discoveries live, and how would they become trusted domain memories rather than a collection of s- session logs? Would you start with a shared coordination service or cloud agents, or can mostly local sessions and repository-owned knowledge work at the scale? How would you handle overlapping work and unclear domain ownership? Have you seen an approach to this work in a larger team, or is there a part of the lesson or an example I missed that addresses it?

What a, what a fantastic question because this is one of those, uh, million-dollar questions that everyone is trying to solve right now, where they want the tools to serve this for you and then trying to help you avoid building your own, like, custom solutions for this. But, like, there's no, uh, there's no perfect answer. The-- Like, in, in general, as much as you can follow, uh, kind of that code owner's pattern, where, uh, you, you mentioned there's, there's cross, cross-dependencies, and I, I get that. Um- But I'd say, like, in general, as much as you can follow the code owners and, uh, maintain those, uh, like the, the memories and the, the glossaries and, and the files that, um, your agents will be reading and using those to infer what's going on inside of that, that domain or the, the code owner's package, um, the better.

I would say, um, one, one thing with any big team, like adopting some sort of software factory or agentic workflow is, um, get a few people to try it. Like, don't, don't be afraid of, um... The, the, the hard part is convin- convincing your manager, like, "Listen, we just need to try this. Like, we need to see if this is gonna work." And, um, let- letting them know that it prob- like, there's-- it probably will fail. Um, like, this is an investment where we need to figure out what, what the right approach is, because it's not gonna be right the first time, and it's not gonna be right. Like, if I tell you some answer right now, you'll probably have to adapt it anyway.

Um, but yeah, like there's a-- The, the tough part is that staffing changes all the time, roles change all the time, um, as people come and go in the company. Like, you could try and set boundaries around, like, individuals or people on the team. You could try and set them around the roles, um, but those are gonna start changing with, with more and more agentic work. Um, I would-- Like, I take the general mindset that the, um, the code is the source of truth of the project, um, and that as much as you can make the, the code the thing you look at as the way you're architecting it and not, not the people working on the code, um, and how the code could kind of, um, slurp in agent decisions and things out of the, um, conversations from pull requests and from the, um, I think, I think you mentioned in here, the conversations you have locally, uh, as you're making decisions with, with your agents.

Um, like, keep on feeding that information in and developing that knowledge base. Um, I found that the people who stay as flexible as possible, which usually means a, um, a collection of markdown files, uh, next to the project, either inside the project or like a sister project that is like the, the knowledge base for it, almost like a wiki. Um, the people who stay as flexible as possible, uh, with that are able to adapt as those things change. Um, the people that buy into a, like a hardened system, they're the ones who end up, um, with another system coming along, and then they get stuck in a certain pattern, and it's hard to transfer that knowledge over.

Um, so the, the more kind of hierarchical raw text of markdown you can have as-- Like, it's a keep it simple sort of answer, which, um, people often want complex answers. Like, I- I'm just, like, recommending markdown files inside of folders that you review, you-- Excuse me, you review with your team often, and that if any PR comes up, any conversation in Slack comes up or whatever, and someone makes a decision, and you run that decision like, "Hey, check, check what this means against our knowledge base." And if that pings anything in there you need to change, then-- and go and have a discussion about it. Um, so I would say stay, stay flexible, stay, um, kind of stay light on your feet about it because things are gonna constantly change, and that, um, in general, the, uh, like, I'm never gonna argue, I'm never gonna, never gonna be upset if I just see like a, a collection of markdown files, um, and some of these agents, you know, how they have like million token context windows and can, um, understand huge knowledge bases in a single pass.

Um, and then you can build tooling around that, but as long as like the source of truth is, is that, then either inline in your project or outside of it. Um, that's probably more opinion than best practice, um, but, but I'll stand by it. All right. Um, yeah. Now, what models and providers do you use with Pi? Love to use Pi more. Um, yeah, I use-- I do use Claude Code, um, or I use my, uh, Claude Code subscription with Pi. Um, I use my Codex and Gemini subscriptions with Pi. Um, so you're able to, um, essentially, there- there's a lot of harnesses around all the providers, um, OpenCode and T3 and, um, Factory and like all of these, and they all do very similar things.

And even with open source, if you ever p- see a provider that's not supported, you can just say, "Hey, look at how they implemented it here. I want to implement it in my system," and then you'll be following the same rules, um, as, as those ones are, the, like the, the bigger ones that are run by teams. Um, I would say with like the Claude Code, they've had-- they've kind of waffled back and forth around how much they're supported, and their language has never been super clear. But for the past-- Like, they had, they had a brief stint of a couple weeks, maybe it was a month or so, where they were banning accounts because of other tools using the, the subscription.

Um, they backtracked on that, like in July or August, and haven't said anything since, and no one's reported any, any issues since. Um, so I can't say that they're not gonna flip-flop again, um, but that's like the current state of things. Um, I think that's my, my answer there.

1:01:11 Wojtek: Uh, Victor has a follow-up for that.

1:01:14 John Lindquist: For this? Um...

1:01:21 Wojtek: Yeah. Does, does Pi allow using Claude Sub? As far as I know-

1:01:25 John Lindquist: Yes.

1:01:25 Wojtek: -they expect API key.

1:01:27 John Lindquist: Yeah. Yeah. They, they do allow-- So, so Pi doesn't necessarily allow anything. You can build an extension or use a community extension, and there is a, like, an official Pi community extension that, um, does the same Claude Sub logic that, um, other, uh, other tools like T3 and OpenCode and all of the other open source ones do. And as long as they're using that same logic, it won't like-- it's not going to detect whether it's Pi making those requests or, or whatever. Um, there, there was, if I remember correctly-- Excuse me, there was a little while where you could use the OAuth stuff from Claude Code, and now they recommend you use their, um, uh, ACP or, or one of those.

Um, but the-- I just let the extensions handle it, and I, I've used it constantly for months, um, with the Claude Sub and haven't had issues. Um, and I have a lot of friends who, who do that as well. Um, and I know I can't, um, I can't guarantee you'll be in the same way or that Anthropic won't change things. And Claude Code, I think, is a great, um, a great interface, which has, uh, the recent mods and plug-in system and everything. Uh, it is fantastic, and, um, I have to say thank you to them for, um, trailblazing a lot of what, uh, the harness even is. Um, I think we owe a lot to them as the entire, like, ecosystem of harnesses.

Um, but yeah. Uh, your take on agent factories and big teams, 10 to 20 teams, 5 to 6 each. Um, yeah. This is-- I, I, I would say, um, you kind of have to, um, own your domain where, like, whatever it is that the type of work you're assigned to is, like, you're going to be the expert in where all the troubles lie there. And, um, once you create the agent, you just have to own it and figure out how to, to make that work and, and check that in with, um, everyone else around you. Um, if you're an automation engineer specifically, um, then you're going to need to communicate a lot. Like, you're going to need to be explaining what's going on a lot to, to your team, um, or telling your agent to communicate a lot.

And that, that sort of, um, those frustrations will creep in, in Slack and other channels where you see bots being used and failures and, um, all that stuff. And make sure in those communication channels with automations that there are feedback systems right there. Um, the most basic one is a upvote, downvote to help the agent learn that that was a bad response or, like, I didn't want you to do that. Um, just make sure that anytime some sort of agent response, some sort of automated response pops up, the way to reply to it is, like, right where you're looking with a simple type in a box and hit enter or do something and hit enter.

Um, as far as, like, multiple teams and information sharing, um, and automation sharing, like, that's a-- It, it's tough because it's, it's, again, different for everyone. Um, but I, I would say, like, as long as there's somebody who is kind of spearheading the effort and has a full grasp on what everyone is doing and somebody that, like, knows all the, the things that are going on, and that's being, um, fed back to them so that work isn't being duplicated too much, then you can help. Um, like, it's kind of like an org chart issue where someone needs to, to be out in front of all of that, um, or else it just turns into a Wild West of everyone building the same thing.

Um, we saw that at Vercel a lot where everyone would, um, because you could build things, you know, in an afternoon, everyone would tackle the low-hanging fruit, um, and especially, like, project managers or, or non-- people who historically weren't developers would end up building the same thing. And that's fine, and in-- well, we actually encouraged it, um, because it's an extremely valuable experience for them. It's just that once they get to that point and, um, make sure that they're showing that to somebody and that you help them learn from that experience of building something so that you can, you can take everybody who built things that are similar and toss them in a Zoom call or toss them in a room and say, "We notice you're all building a similar thing, um, and we encourage that, but we also encourage you all to communicate with, um, the experience and the, um, knowledge sharing of, like, how can we best, like, merge this into one thing and, um, make it that something you all agree on."

Um, so just communication is key there. Like, anytime anyone's building an agent, um, make sure that gets surfaced and communicated in the proper channels. Make sure that any of the automations have instant feedback, and make sure that anyone who's, um, building agents are aware of people who are building similar agents and that they get in a room together. Um, I would say those are my three big things there. How to learn? Wow, that's a fundamental human experience question. A lot has been said on how agents can do the coding for us and how we engineers are becoming their managers, but there are many other areas where agents can improve productivity, one of which is learning new techniques.

What are your tips for boosting or, or boosting our learning workflow? How to use agents to give you a personalized learning path on a technology, not necessarily AI-related, you want to get familiar with? Um, there's, there's some really, really fun ways to explore this. Um, I personally like turning things into games. Um, if you want to, uh... Like, one of my favorite games of all time is called Typing of the Dead. It was a way to train yourself on how to type. And zombies would pop out, pop out from the corner, and they'd have a big word on their face. If you didn't, um, type the word before they got to you, they would, you know, eat your face or whatever, right?

Um, but if you typed the word in time, you kept on progressing through the level. Um, yeah, you can, you can Google it. Typing of the Dead, great game. Um, and what that combined was, uh, the experience of, like, wanting to progress in this thing and wanting to learn and wanting to, like, get further than I've ever gone before. There's a lot of, um, creative ideas you can apply that to as far as knowledge is concerned. Um, and, like, if you imagine, like, zombies that give you three options that are the correct, you know, only one of them is the correct answer, or, you know, or, or whatever. Just the more you can make the experience engaging for you, and it's something you can prompt in a day, um, the better experience it can be, the more hands-on it can be.

Um, similarly, the more you can, um... Like, there's a lot of studies around learning and, like, where you learn. If you learn something in the library, if you learn something going for a walk, if you learn something in bed, you're going to recall that thing when you're in the library, you're going to recall it when you're on your walk. Like, there's a lot of, uh, location-based learning. And so there's interesting things you can do there as well. It's like if you, if you want to focus on a certain topic when you're on a walk versus focusing on a certain topic when you're in the car or on a commute or, um, in a certain room, if you could, like, set up an environment.

Um, like, one of the most fascinating things about, like, traditional schooling is that in this room you learn chemistry, in this room you learn English. And so, like, you'd go into that room, and you'd be in, like, English mode or chemistry mode. Um, and you can kind of establish that for yourself now. It's like, okay, I'm going to take my laptop, I'm going to sit on this chair, and in this chair, I'm learning this subject. Um, so I, I love learning. There's, there's a million ways of talking about this, but, um, I would just say make it fun. Um, make it engage-- Like, you can make anything engaging. Um, what I like to do, uh, you've probably noticed throughout the week, is, um, assign characters to ideas.

Um, you have a lot of image generation options here, and, um, we're asked to, um, we're asked to consume a lot of text, like huge walls of text. And if you can assign, um, like, uh, some sort of imagery or some sort of, like, thing that you glance at that is, like, in- like, encapsulates that idea, that thing will come to mind. Like, that thing will represent all, like, everything you learned about, and it'll bring to mind all those things. Um, so it was a very intentional choice of this week of using, like, these robots that represent agents. Um, and that's, um, another thing. Then, um, finding... I think, I think one of the key things that I haven't addressed yet is that any sort of topic that you start diving into, ask your agents to find the real-world scenarios for it.

Um, it's kind of like sitting in a math class, and you learn all the stuff about math, and I remember sitting through, like, trigonometry classes, and it was never useful. And then I got to, like, you know, playing with writing games and things bouncing off of walls and collisions and, and whatever, and then I taught trigonometry to myself because now it was real world, now it was interesting. Um, so when you dive into these topics, have them create these, like, real-world walkthroughs of, say, you know, Sally was using her agent, and then the agent stopped working, and she didn't know why. Let's investigate why that stopped working. And then you, like, follow Sally on her journey of why something wasn't working.

Um, anyway, um, I could talk about this question for the rest of the time, but, uh, let's move on. Uh, what are the three things that pops in your mind when you think about engineering mistakes, and what's the obvious thing you tend to avoid or fix them? Um, number one, uh, engineers don't mistakes, they learn lessons. So [laughs] that's, um, at least I hope. Like, if you're, if you're an engineer, you, um, like, you try things and you, you progress, right? Um, and so we're, we're in a field where we're encouraged to make mistakes, and that's like the beauty of code is that we, we try things and we learn from them and we grow and we build.

Um- Then we will overextend, we will bring prod down, we will break things, and then we'll become better for it. Like, the best engineers I know have the worst horror stories because they lived through them. Um, there are a lot of places, like after being in the field for plus 20 years, there are a lot of places I would never go because I don't have the horror stories, and I don't know... I'd be too afraid. Like, my pride would get in the way of me making mistakes. And, like, I'm just-- I'm too prideful to be the guy that brings down prod by working in this one place, um, so I just don't do that anymore.

Like, but a-as you're learning and as you're, like, growing in your field, like, go out there and make mistakes. Um, as far as, like, engineering mistakes are u-are usually like, um, communicate with your team more. Like, don't-

1:14:13 Wojtek: No, but-

1:14:14 John Lindquist: Uh, communicate progress. Um, and, and, like, e-even more than ever, just being, uh, just being someone who surfaces what they're working on, who, like, talks about the issues they run into, um, someone who's honest about not being able to make a deadline or, um... Other than that, just go and try stuff. Like, just go. Um, like, now, now is the time to have ambition. Now is the time to, to push the envelope. Uh, "Any real viable alternatives to Herder plus Pi with the same level of extensibility? The course mentions that we can use whatever we want, recommends open source, but I find it hard to find a good alternative where I can extend it at the same level.

Dream, I got to start to use T3 code. Now I cannot do the same things with as shared in John's lessons. Are there real alternatives to both UI and GUI?" Um, the, the-- I think the answer for this is that, that is a, like, a, a question that's t-today, and as of right now, like, Pi was the, like... Pi was the first harness that stripped away everything and asked you to, like, start from scratch and build on top of it. Um, Herder can do a very similar thing where it gives you this core piece of, expects you to build things on top of it. Um, and they're, they're fascinating in a way that, um, there's not a lot of reason to challenge Pi because you have to, like, disagree with their foundational, like, the fundamental things that make up Pi, and that's just very basic, like, core harness stuff.

Um, same with Herder. Like, Herder's not doing too much beyond managing pains and communicating the status of agents and... But it just does that one thing really well. So I, I, I think the-- it's, it's not so much a Pi and Pi alternatives or Herder and Herder alternatives. It's just that that concept stripped down to, like, its core is, like, there's no real alternative to a thing that's just, it is what it is. If you make another one, it's just going to be the same thing. Like, you can't-- You could fork it and have, like, different, like, fundamental principles, but you'd probably end up being so close to it. Um, I don't know. Uh, it, it, it's hard to see too many others spinning up, um, because they're not interesting problems to work on, um, because they're so fundamental, and they don't touch the user experience too much.

Um, but I don't-- Like, I don't have... Like, the, the other ones will probably work backwards. Um, I am not very familiar with T3 code, I'm sorry, Theo, um, or, um, where a lot of the GUIs are at, at this time. But, like, you see how Claude Code is, you know, introduce mods, and all these are introduced plugins, plugins and systems. Um, and they'll probably start working backwards to make their systems more flexible, whereas Pi and Herder started, um, working from the ground up. So, I don't think one of them is going to win one way or the other, but, um, I don't think transferring between them is too difficult, um, especially with agents these days, as far as, like, "Please convert my config and my shortcuts to this other space that I want to try."

Um, so... And, and there could be ones out there that I just don't-- that I'm not aware of. Um, so I-- that's a very real thing. I, I know there's one called Rex that's a closed-sourced one, but that just, uh, I'm-- I don't touch closed source right now. Like, it's just something I avoid. Um, so yeah, that's my answer. Thanks. "Agent memory and global memory." Another great question. Uh, "Memory is becoming an increasingly important part of working with coding agents, and I find it can be very beneficial. However, I haven't yet found a reliable tool set or architectural pattern." Welcome to the club. Um, "What tools and end-to-end architecture would you recommend for memory shared between agents, including agents built on different tools or frameworks?

Um, project repository memory, decisions, conventions, domain knowledge, lessons, global user memory, preferences, and usable knowledge. I'm particularly interested in complete life cycle, what should be captured automatically versus curated, where each type of memory should be stored." Um, yeah, the existing tools and proven architectural patterns. So, this is one, um, I mentioned in the, um, in the, I think it was day one, might have been day two. Um, I mentioned making a memories widget which would observe your main conversation and would look at decisions you made and ask you to-- ask you if you wanted to save those as memories while you're working. And it would put a big backlog of, like, save as memory or reject.

And, like, I believe in that pattern of it's not automated- Um, it's not curating them automatically, it's curating them as a backlog for you to approve. Um, and you can just dismiss them or save them to just like a, a memory markdown file. Um, I personally keep that, that simple, but I would say that, um, there are projects, one's called, um, Basic Memory. [keyboard clacking] Um, is it this one? Yeah, I think it's the one at-- Let me triple-check before I say that out loud. Yeah, uh, basicmemory.com, um, is the open source one I've used in the past that actually manages some of the markdown files for you. Um, and I can say that I found that a, a, a pleasant experience, and while it's still, like, managing local markdown files, it introduces, like, searching, indexing, and, and that sort of stuff.

Not, not too much on top of markdown. Um, but as of today, like, that wouldn't be hard to build yourself. I don't know why they have, like, a pricing plan and stuff. Um, but I would say, like, the, the key aspect is, um, I would, I would never automate memory. Um, for, for your project as a whole, I would never automate memory. Um, this is different. It, it's funny how similar it is to, like, the, the reinforcement learning where you put something in a, in a lab and you have agents pound on it until the agent behaves as you want. Um, you're kind of doing that at the macro scale with your project when you're working on it, as you're, um, trying to accomplish these tasks.

So, um, yeah, let the, let the memory surface through some sort of tool that you approve or reject, and that way you have control over what, what goes in there. And they could even surface, um, from, uh, I imagine there's probably one that, like, looks through your PRs or something that surfaces, like, uh, it, you know, notify five people on the team, do they all believe that this memory should be saved? Um, I don't know if a tool does that. Uh, it seems like a good idea. Um, but all those decisions should be brought before people, because, um, the, the problem with memory and specs and things that constrain your system is that they, um, prevent you for- from the, a lot of the exploration work that, um, that keeps the system from looking for new ideas and new spins on things that are outside of your, um, your point of view.

And so often, all you need to solve a big problem is a perspective shift, and the memories can often, like, lock that perspective away behind a, like, a series of guardrails and walls. Um, so in general, from, like, the, the experts I've talked to, you want to keep memory to kind of a minimum to allow the, the model to be as creative and flexible as possible, to allow those, um, those explorations. Um, so I wish I, I wish there was one, like, good answer I could give you. Um, I do like basic memory. I do like, you know, um, a big markdown file knowledge base. Um, and I do like having, uh, options presented to me and me approving them, and then going through and looking, looking at that with the team.

Um, that's the, the best I can do for you today. All right. How do you handle complex UI/UX changes? I'm a front-end team, works with designers, you know, state of the art models, obviously the design is big, it doesn't do a good job. I had one good size ticket I have to co-design that the model makes it good in one shot. So there's many small tweaks that it misses. Um, I do like Impeccable. Um, Impeccable, I think it's Impeccable MD, um, is a design, uh, vocabulary and framework. Um, but I, I would say other than using, like, an Impeccable system for the, and the skills that it brings in for design, that I wouldn't necessarily recommend Storybook per se, but I would say that, um, asking your agent to create the core components in a massive, um, HTML file that's just like this raw HTML file that you can throw away, and, um, as you go through, you rip out the ones that become the, like, the source of truth for how things get, get built out.

Um, and you just kind of have an approved set of components for anyth- anytime something new is being built. Um, and this is, you know, kind of going beyond, like, beyond like a ShadCN or a Tailwind component library or something. Um, having those core set of components and having, when you ask for them, make them fully interactive and make it so that, um, um, I, I've mentioned this before, but it's, it's really fascinating to have it build an entire component set and then to tell agents to go and use the component set as if they're users who have never experienced but, ex- excuse me, experienced it before. Then creating videos of that and then, like, watching those videos with the team and talking through, um, what they look like or how they behave and the, the experience around them, if it's something that, that you want or not.

Um, it is a really fascinating way to work where, um, you just have these, like, video artifacts of agents driving components. And, um, sitting down with-- and, and just, and just watching them. It's a very, like, relaxing thing to do as well because you don't have to put too much effort into it. Um, and then as the, as the video goes along, um, if you build the tool like the, the workspace I have for, for making videos, at any point in a video, I can take a note at that frame in the video, and it captures the frame, and I say, "At this point," I can draw arrows on it and say, "At this point, uh, this looks off, and that looks off, and that looks off.

And the reasons are because of this, that, and the other." And so then you have this video artifact with the frames, with your notes and everything. You can feed back in the agent and iterate that way. Um, that's, that's what I would do. That, that's what I am doing, I guess. Um, when I think of complex UI and UX, I think of something that has like a ton of state and a ton of, uh, options, and you can have the AI throw a lot of different data sets at it and stress the edge cases and whatnot. Um, formal verification methods and gated policy. What's your take on formal verification proofs and the contribution to a robust software factory?

And a follow-up, have any recommended approach to how to implement these verification layers as well as policies that govern different levels of autonomy? Um, yeah, I, I don't know. This, this starts getting into the, um, like the lab level type of proof where your, like the enterprise level of verification proofs, um, and something that you're going to lock into for a long time. Um, there's, there's something like-- there, there's a level of cost involved in this and a level of effort involved in this that, um, my projects haven't been reaching recently because I try and stay more flexible. And I-- unfortunately, I'd have to do more research and talk to some friends about, um, like getting these level of verification layers in place.

Um, and with the way, like how quickly models change and like this is like locking into a model for a long time because you put so much effort and so much cost into it. Um, like I, I prefer to stay more flexible than that. Like it's a-- it's so hard. It kind of goes back to the spec-driven design. The more you lock things down, the more you're locked in, and the more you prevent yourself from moving forward the next improvements that are released. And so I've seen more success in people who stay like as light and flexible as possible rather than dumping-- like I just feel like the effort is better spent elsewhere. Um, I'm not saying-- like there's definitely scenarios where this is, um, you know, airlines and stuff that these are, you know, business critical or, you know, people's lives are on the line.

Um, but I, I-- there's not something I can, can recommend at this point. Um, I, I can research and get back to you.

1:28:51 Wojtek: Uh, John, just letting you know that we are approaching our 90-minute, uh, uh, time. So if you decide-

1:28:57 John Lindquist: Okay.

1:28:57 Wojtek: -to go on, no problem. Um, we can go through a couple more questions. But I also just wanted to let you know that there's a good follow-up from, uh, Brendan, uh, voted here in chat, if you could take a look at that. And for everyone, uh, remember that whatever questions will be left unanswered today during this live session, John will still be on Discord and, uh, will be able to answer them, uh, later.

1:29:27 John Lindquist: Yeah. Um, I, I can hang out for a little while longer. Um, let me get Brandon Brown to talk a bit about creating a workspace for collaborating on UX with product owners. How'd that fit into the SDLC? Where should acceptance happen? Um, is that-- let's see. With product owners. Yeah, I, I would say, like you could bring-- I was talking about video artifacts and, um, establishing those scenarios. Um, since, since you're able to essentially one-shot a lot of those experiences for them, um, and bring that into the room where they can play with it, it's almost like the idea of a prototype is like you used to build a prototype, like one prototype, and it took you a few days, and now you can build 100 prototypes, and it takes a, an afternoon.

Um, I would, I would say a lot of that is having those, like video or the artifacts that, um, the user experiences that you can play through and watch as if a user is using them. Um, and that, um, you could even like put a narration on top or have a, um-- it's very cheap now with either ElevenLabs or any of the open models. You can run and collaborate super cheap to like put a user voice on top of it. And you could like fake a user talking through, like have, have the, the AI model say what it's trying to do with the product and the user experience. Like turn that into a voice that's talking through out loud and like have it use the, use the product.

And you can sit there and watch a video with a, uh, someone who's invested, the product owner who's, who's invested in the product, and say, "This is what, uh, we're simulating what a user would experience when, when they open this." Like what the things are looking at and whatnot. Um, and, uh- I mean, humans are better at doing it, but, uh, like actually recording humans going through the, uh, the UX of things. But you can simulate a lot of that with agents, and they can capture a lot of it. And, um, strong-- I strongly recommend if you get a chance to, like, get things recorded and, and throw it into the, like, the Gemini Carbon stuff and see what that can do.

Um, but yeah, just emulating that real user experience with the UX, like build out the prototype and then spend the time building out the, um, the user sitting in front of it and clicking and typing and, um, trying to accomplish ta-- like giving it a set of tasks to try and accomplish. Um, and then you can just, like, watch that video with them, and they can give feedback on, um, what they agree with, what they disagree with, and make as many variations as you like. Um, it's a crazy world right now. Um, can we go to the next question, QA? Um, let's see real quick. Yeah, how can juniors build engineering taste in the AI era?

Uh, wow, what a question. Um, and like defining engineering taste is, is difficult, and whether that's important or not is, is debatable. Yeah, the engineering taste, what a term. I'll, I'll have to like sleep on this one. Engineering taste is such a fascinating, like, even concept, um, because it implies that, um, there's like traditionally we've had good engineering and bad engineering, and now you have, uh, AI models writing programs from assembly, and like what, what does that mean? Like, typically the, the, the beautiful thing about the, the junior mind is that they don't know what's not possible. And so you put a junior on a task, and you don't tell them that you can't build this from, like, you can't build that in assembly.

That's, that's crazy. Um, they wouldn't know that. They would go in, and they'd try and build like the most performant thing possible, and then like out pops this thing that you didn't think was possible, and a junior built it, um, because they didn't have that same like pain that you've experienced in the past. And that-- it's a wonderful thing. Um, I've seen that in, in, in a product I worked on. They wanted to implement chat. It was many, many years ago, and I'm like, "You know what? Building chat out is just insane." Like, the code base is a mess. There's no way anybody would want to do that. A junior hops on the team, they tackle it, like they get really interested in it and use it as a learning experience, and they dive super deep into it.

Became a core feature of the, the product because, um, like they had that, um, that drive and that, um, they didn't know that it, that it wasn't-- they didn't know how hard it was. It was still hard. They ran into all those things I knew they'd run into, but they just plowed through it, and they didn't give up on it because it was the thing that they owned. Like, there's just a sense of ownership, like junior ownership is a amazing thing. Um, and leveraging, like giving them responsibility over tasks. Um, so like if you can get an excited junior on a task with AI, like just give them something to own and to go crazy with it and push it, and they might, they might surprise you.

Um, anyway. Uh, how do you deal with the, [clears throat] excuse me, mental fatigue of working with AI agents? Oh boy. In my opinion, resting and mental energy management are now super important. Do you have some golden rules for keeping a steady pace at work during hours or during work hours? I wish. Um, this one's tough. I take naps. I go for walks. Um, I have kids, so I'll go and play basketball in the backyard and take breaks like that. Um, and the, the hardest part is that because I can talk to my agents from my phone, and it's the same project and checking on them and stuff. Um, the, the, the main thing for me is that, like recognizing that if I spin up an agent that's gonna take a while at something, that it's okay to like not start another agent on another task.

It's okay to not try and multitask a million things at the same time. Just let that one agent and that one like chain of thought work, and I can go, you know, take a break. Um, like focusing on that one most important thing and not, um, worrying so much about all those other things that seem fun and important or whatever, because I, I think it's the other things that create the stress. I don't think the main one creates the stress. I think it's the other ones that all feel important because you start managing those as well. Um, I'm not saying I do a great job of that, but, um, I'd say like the more you can limit, uh, to the, the thing that means the most to you, whether that means making money or improving your life or helping your family or whatever, like whatever, whatever that means, the thing that means the most to you, like having that be your agent time.

Um, and then like the, the fun part about that is you can use the other agent time with friends and family and help them solve problems. Like if you're, if you're just trying to create problems and solve them for yourself, um, you're not leaving agent time and like mental bandwidth for your family and for some of the stuff that that can solve. Um, and like with one of my kids, they were memorizing a Jabberwocky poem, and I was able to, like, sit down with him and make a song for it, make a bunch of images, and, like, create this cool, like, bouncing karaoke thing where he could, like, learn the poem by singing the song.

And it turned out great, and he loved it, and he still sings the song, and it's, um... And that was fun, and I was like, "Agents are cool." Um, anyway. [laughs] I had no idea it was not okay to start. Yeah. Oh, man. Um, yeah, that's-- I think that's answered.

1:38:12 Wojtek: Uh, I see that the fo- next questions we have on Discord got zero upvotes.

1:38:18 John Lindquist: Okay.

1:38:19 Wojtek: So my proposition is that we leave them for, you know, for, for later for you to answer directly on Discord.

1:38:25 John Lindquist: Perfect.

1:38:25 Wojtek: And I can see [laughs] uh, a, a question that might be a good one to end the session with, and it was bumped by a couple people, and it's about cost management, or as Victor put it, "What's your monthly AI token bill?" [laughs]

1:38:42 John Lindquist: Yeah. Um, again, like, uh, I have two Claude Code accounts, and Opus 5.5 on Medium is like, is, is the get work done mode. Um, more recently, like, getting those... I, I'll-- I've had a Gemini account for a long time. More recently, I actually bought a second Gemini account because I've, um, running it through a lot of, like, video processing and understanding video because I have a lot of, um, interest in that. Um, that second account actually got banned because I ran automations against it to process videos, and apparently I had to go, like, fill out a form and stuff. So I need to be more careful with that. Um, but then I haven't been using, um, OpenAI's stuff as much recently.

Like, it just... Yeah, I, I would s- I would say the bill is just those two Claude Code, like, max accounts, and then, like, I have a personal, like, Ultra Gemini because it, you know, YouTube and family, I share it with five people and all that stuff and get all the, uh... It, it is-- Can I just say, like, the Gemini subscription for 200, excuse me, 200 dollars a month, the fact that they give you, like, all of these collab hours where you can train models and you can run all of these, like, video generations and all of these things in collab, um, and you get credits for their API services, and you get YouTube, and you get all this other stuff, like the value proposition there, even though Gemini doesn't have the best models.

Like, if they do, did, did they... I think they're releasing another model, model today. But anyway, the value there is incredible compared to, like, a Claude Code, which is like, here's a single agent. Anyway, um, I sound like a Google shill again. I promise, I promise this hoodie was free. Um, but, um, yeah, that, that, that's about it. Uh, it's... I, I have a couple, like, Cursor gave me, um... It was hilarious. I did an interview with them, like, years ago when they first started out, and they gave me more credits that I still haven't used sitting there in an account. It's like my emergency account in case I use up both of my [laughs] Claude Codes.

I tap into that every now, every now and again. Um, but yeah.

1:41:09 Wojtek: Okay. So if that's it, then thank you very much, John.

1:41:13 John Lindquist: Thanks.

1:41:14 Wojtek: Thank you guys for, for being with us. Thank you for your questions. And John, you'll be on Discord, uh, through the remainder of the week, right?

1:41:21 John Lindquist: Absolutely.

1:41:22 Wojtek: Great. So if you guys want to add some questions to the Discord forum, go ahead. If you want to upvote some questions that still haven't been answered, you can go ahead and do that. And for now, thank you everyone, and have a good day or have a good night, depending on where you are.

1:41:39 John Lindquist: Thanks, everyone.