It's just all about building trust — Matt Pocock with Joel Hooks
Matt Pocock and Joel Hooks discuss building trust in coding agents, from proving code works and automated review to keeping humans in the loop. They also cover agent workflows, Effect, and why human judgment still matters.
videoMEGA DEV · 58 min · Oct 23, 2026 · youtube.com · Saved Oct 25, 2026
coding agents, agent evaluation, effect, software factories
Description
Matt Pocock moved part of his course planning back to paper. Some tasks need the whole puzzle in one head, and talking them through with an agent costs more than it saves. In this MEGA conversation, Joel Hooks asks him where agents do pay off, and how a team gets from a single agent PR to a software factory without turning the app into a slop cannon.
Matt's answer: start slow and start with the brakes. It's just all about building trust. Agents need a way to prove their code works, every step up the trust ladder lets you run more of them, automated review catches the agents that cheat, and every one-way door still goes past a human.
Also: why your code base is kind of like a massive agents.md, Effect as a backend where restrictions on agents become inevitable, stateless agents vs Joel's LLM wiki, reading the software classics with agents, what loops really mean for mere mortals, and why the application of taste is still a human differentiator.
MEGA is four weeks with four authors: Theo Browne, Angie Jones, Kent C. Dodds and John Lindquist. Starts October 26.
Chapters
00:00 Intro
01:17 Back to paper: the communication overhead of agents
03:22 Too many lanes of work in progress
05:16 Brain dumps and animatics: seeing the course before filming it
09:24 Are we really building software factories?
10:40 The trust ladder
13:41 Bad trajectories: how agents produce garbage forever
15:56 Where should a team of 10-50 developers start?
17:14 Start with the brakes
18:08 Raising the floor of failure
19:29 Personal software is not production software
20:35 Reading the software classics with agents
22:15 Domain language, DDD and patterns agents actually read
26:24 Skills as a framework, and how to make them your own
31:49 Leading words: the most valuable part of a skill
33:15 One-way doors and two-way doors
35:21 TypeScript and Effect
37:22 Agents require restrictions
39:30 Agents cheat: lint rules, automated review, human review
41:46 Stateless on purpose vs an LLM wiki
44:53 Token austerity
45:35 Top tips for building your own factory
47:17 Your code base is kind of like a massive agents.md
47:47 Loops, OpenClaw, Soul MD and Vision MD
49:51 Tactical vs strategic programming
51:43 Turning the death spiral into a virtuous circle
52:11 Support, product design and human teams
54:29 The application of taste as a differentiator
55:48 The teach skill and giving your agent eyes
Links
- Matt → https://x.com/mattpocockuk, https://aihero.dev
- Matt's skills (grill-me, teach) → https://github.com/mattpocock/skills
- Joel → https://x.com/joelhooks
- Effect → https://effect.website
- Remotion → https://www.remotion.dev
- OpenClaw → https://openclaw.ai
- MEGA → https://mega.dev
Transcript
Intro
0:00 You need to get your brakes in first, because if you don't push the quality high enough, you're going to turn your app into a slop cannon. >> Personal software is not production software, and production software is still extremely difficult even [music] if you can produce software at unprecedented rates. >> You think of agents.md is the place where you put all of your instructions, but actually your code base is kind of like a massive agents.md, but only code.
0:20 >> They speak it and understand it and search it. >> As you trust your agents more, as they're able to verify their own work, you get a absolutely outsized amount of benefit. >> The agents cheat, and they're always trying to get around, and and they take the easy path. How do we stop them from from cheating? >> And so you get these weird trajectories where it will seed something bad in the environment, and then that will produce garbage forever.
0:41 >> I'm not austere with my tokens, man. >> Whereas I tend to be more on the more austere side. >> Stop building for a second and actually learn. I don't think we're in a post-learning society. >> I think the application of taste is essentially a differentiator, right? >> You're still useful, uh, whoever's listening. >> You could change jobs before your strategic mistakes come back to bite you.
1:00 >> Intentionally change jobs before your strategic >> [laughter] >> Just keep running, man. Just keep running. Whoops. >> [laughter] >> Yeah, I just walk away from the explosion. It's fine. >> Yeah.
Back to paper: the communication overhead of agents
1:17 Hey, Matt. >> Hey, Joel. How you doing? >> I am doing well. How about yourself? >> Good. Yeah. >> I read recently that you've uh reverted back to paper for for some of your planning processes. Um, I'm curious about that because like I'm I'm constantly, you know, being being crushed, uh, by by my computer usage. So I'm I'm curious what led you back to, you know, the the old school.
1:42 >> One of it is was there are certain types of tasks that I needed to retain the entire thing in my head at once. And what I needed, like this was specifically for planning courses, right? And planning courses, it's like a huge puzzle where you basically need to find the correct order of about a hundred different things. Let's say per section.
2:09 And this was a six-section course. And sometimes they have independencies per section, that kind of stuff. So it's a huge puzzle that you're building in your brain as you go. And what I found was the more I tried to get agents to help me with that, the less I could keep in my brain at once, and the more they would just sort of splurge out information and garbage. And even if it was relatively high quality, I would still have to take the time to understand the planning decisions they'd made and try to keep up. And so what I realized I needed to do was to take it back a step, to put it more in my brain, and then I could actually just get the thing done a lot faster without needing this sort of communication overhead with the agents. So it's almost like you know, should I do this on my own,
2:57 let's say, or should I do it with a colleague, right? If you There are certain types of tasks that if you just do on your own, you're going to get them done faster. And when I started doing it on my own, I was like, "Oh, wow. I I feel a lot more relaxed. I feel like I'm able to make better decisions." And I think people just overestimate or underestimate the communication overhead with agents, right? It is another person that you're dealing with, and that can be very tiring.
Too many lanes of work in progress
3:22 >> I mean, I found too, like when And this kind of goes back to like classic project management for me, when you have too many lanes of work in progress, everything to me breaks down, and you get this kind of frenetic chaos where you are trying to keep track, and you are trying to track the conversation, and the agents are so willing to please that they will divert and then start talking about everything at once, and then it it it goes from, you know, like a beautiful crafted painting to like a a lovely shade of brown. Like you're you're now in brown town. Um and and everything kind of goes downhill and for me it's really hard and it's a graph like at the end of the day, right? Like you you're you're building a knowledge graph when you're building a course or you know, like if you're building a
4:03 product too, I think at the end of the day because features and all this stuff and so you the more you start and the more you spread the context thin across a lot of conversation. Like to me like it it like it goes downhill and you like you said you can still get like there will be good results in there, but as a person like our ability to parallelize um diminishes. Like it and it's it's a real problem actually for me when I'm when I'm dealing with these things.
4:28 Especially if I'm trying to scale them up and get a bunch of agents going cuz that's, you know, like that's fun. Um but >> Totally. It's good in like really well-defined pipelines, right? When you know that like I know that uh my course goes from okay, the shape of the course, the learning goals for each section, and then I turn those learning goals into beats, which are the sort of vague things I want to hit in every video. And then those beats turn into uh animatics, which I then review and take a look at, and then those animatics get turned into videos. So, by the time I've organized the sort of initial planning stuff, there's obviously I'm trying to like figure out what's next and what's here and what's there, but also that has ramifications in scope as well. So, I don't know. It's I've just found that pulling myself out
5:12 of agents for a while and it's feels a lot nicer.
Brain dumps and animatics: seeing the course before filming it
5:16 >> So, I I want to can you talk about animatics? And I don't know that this is something that everybody necessarily needs cuz you're producing video and you're producing courses, but I think it's also an interesting um it's an interesting flow and process and I'm I'm curious about two things. One, do you when you're doing your hand notes, are you taking photos of them and then feeding that back into your agent to like continue the conversation? Um and then like what what is an animatic and how and how does that work and why is that helpful to you?
5:43 >> Yeah. Um one flow that I've been using recently um for all sorts of stuff, is brain dump and get the agent to restate, right? So, what I will do is I will just um have a bunch of cards out in front of me and um figuring out the shape of things, and then I will just essentially read out the cards and describe them to the agent in a huge brain dump. So, I dictate it out. And then I get the agent to restate what I just said back to me.
6:09 And usually that produces a nice little plan, and I can Okay, maybe don't do that, maybe do this, maybe don't do that. Cuz I find that with brain dumps you often get that slight anxiety of Okay, I've just talked for 15 minutes to the agent. Are we actually aligned, right? You know, maybe not. >> Yeah. >> Or it might pick up on something in your brain dump and really go hard on it when you actually didn't mean that.
6:32 So, the other part is uh animatics. And animatics is I I don't know what it is about Opus 5.5, but it is very, very good at vision, right? Understanding what's on screen. And you've seen a lot of stuff happening on Twitter of people producing really nice animations with Opus 5.5. And this led me to think, right, I would really like to, before I film a video, actually see a really high-fidelity mock-up of that video. Just so I could like, you know, understand it a bit better and uh get it to maybe verify the things I was about to do, which often takes a lot of time in filming. And so, what I've ended up with is before filming this course, uh my agents have run through the course for me, right?
7:16 They've actually done the stuff locally, and they figured it out. And, you know, figured out all the sort of gnarly stuff that I would have had to do myself. And they've produced these videos back to me. So, I've almost seen the course before I'm filming the course, you know? And I think a lot of people are sleeping on the fact that agents can produce these very high-fidelity videos for not that much in in of token spend, right? You >> don't even take that long, actually.
7:43 Like, on clock time, I found it's it's like a 10-minute, maybe 15-minute process to make a a few minute video, which is is is wild if you've ever done animation. >> Yeah, it's wild. It it's outrageous. And even if you just sort of take the fidelity down a little bit and like say, "Okay, you're only using still images.
8:02 You're going to use like a local voice model or something." Um I've got mine set up so it's, you know, uh really parallelizable. It is very, very very quick to create these animatics. And I think, especially when we look at like code review or things like this, things where you the agent needs to give you information in a very rich way, I think the video is actually something that we're going to be seeing more and more of.
8:25 >> Yeah, we saw a lot. I think there was the the Opus 55 came around and people discovered that this is good for explainer videos. And so, there's been this just absolute surge of explainer videos. I don't know if you I'm I'm sure you do, but Kit Kit Langton had it something fun. Uh uh he he put out uh just for for explainer videos. I have to I I can never remember the name of psychopomp.
8:48 Um is is what what he's called this. And it it's actually fantastic and it makes fun videos. His stuff is always fun. But just you know, like so there's frameworks evolving around making these videos. And and I think that's cool that we can we can get that um sort of turn around on our our uh exploration, I guess, in the capabilities kind of latent space of of the frontier is is a fun place to explore.
9:10 >> Yeah, and Remotion was pretty early on this, too. Like, I've been a fan of Remotion for ages, essentially creating videos in React. And they they've been, you know, really doing well out of this. They've got a very popular skills library. And I I'm starting to use it in my courses, too. It's very, very good.
Are we really building software factories?
9:24 >> So, there's this notion of a factory. And and people are everybody's talking about a software factory. You know, you're you're thinking about that. I think in some ways that's what we're talking about. Uh you have, you know, the the agents are working behind the scenes for you, running through debugging your course material, working through your ideas. I I assume that they are running in the background. You're not watching them as they as they work through these things. I mean, there's two kind of different ideas here that I think are are either pushing together or pulling apart. And And one is the factory work, right? Like we want to we want to produce the thing.
9:56 And then the other is away from keyboard. What What can you do when when you're away from key from the keyboard? And I'm And I think another interesting thought that that came up, and this was the conversation that you had with with Lauren Potato, um about the idea of where she prefers to think about the factory as a Michelin star kitchen, which to me is in in some ways just a factory. They're producing food. But But, you know, like the high the high meticulous high quality, you know, like relentless nature of of that sort of environment. Um Are we really building factories? Is that Is that the goal here? Is working AFK like What What do What do we What are we shooting for right now when we we think about autonomous agents running in the background for us?
The trust ladder
10:40 >> So, what Lauren talks about is a trust ladder, right? And the idea is that as you trust your agents more, as they're able to verify their own work, you get a absolutely outsized amount of benefits from that. And you can start running huge amounts of agents uh like at once. Because, let's say you trust your agent to 90% of the time um build the correct thing, right? That's pretty good. What would happen? Maybe that might take you to 50 agents running at once, right?
11:14 What about if you trust it 95%, right? Then you start getting even more. And so, what you're trying to do, I think, and this is how I sort of think about a software factory, is that you can think of the flow from like idea to production as currently full of checkpoints, right? Points where the human might need to intervene.
11:37 >> Gates, if you will. >> gates, let's say. And these [laughter] these checkpoints, I'm not going to be drawn into that. Uh they're essentially you can start removing them. You can start taking them out. And the first one that you might want to consider removing is initiating work, right? Is what work needs to be done. And I think of the moment when you're in a software factory, not and not just like, I don't know, uh producing work with agents.
12:08 I think the definition is when you have triggers that initiate agent sessions on their own. So, let's say you have a a loop that checks your GitHub issues on your open-source repository, let's say, and starts uh initiating sub-agents to work on that work. That is when I think you've reached the software factory. And when you have that sort of level of acceleration, when you have these triggers accelerating work, you then need to think about the brakes as well, the things that actually slow the agents down, that produce higher quality. And that's where you get to verification, that's where you get to testing, that's where um human review also needs to come in. Let's say, not necessarily on every single PR, but maybe you're sampling some PRs.
12:53 So, I do think that software factory, I mean, I don't really have a strong opinion about whether it's a good term or not. I think it is the term. It it is appears to be what everyone's saying, and it's it's this level of automation of removing humans from certain parts of the loop. But, I think as soon as you start thinking about removing humans completely, then there's a whole other class of problems that come in, which is how is this factory going to manage itself long-term? And I think everything that we've seen, all the people who are trying to work on state-of-the-art solutions here, they're all kind of saying, "No, it doesn't seem to work."
13:31 They get into weird patterns and bad trajectories, and eventually the factory grinds to a halt. So, yeah, I think we're all four men or four women in the factory now.
Bad trajectories: how agents produce garbage forever
13:41 >> What do you say? I mean, that kind of goes back to the discussion around paper and like multiple lanes of work in progress and the entropy that's involved with just the the way I think that maybe, you know, and I'm I'm not not an expert at the like the low level of how an LLM works, but I understand it to the extent that it feels like, you know, like it just gets lost in the latent space and all of the potential, you know, um for for the autocomplete aspect of it just goes crazy, right? Like the the it fractals out and you end up with with such a a space of choice that it starts making bad ones.
14:15 Um and and yeah, let me hear. >> The metaphor I would use is you can think of an agent as like any agentic process, let's say, and let's say you're working on uh code base is a process. It is a stateful process, right? It is not stateless. It is stateful because the code changes underneath it every single time. And so, as it produces more code, it has a bigger impact on its environment, and then the environment has an impact back on the agent. And so, you get these weird trajectories where it will seed something bad in the environment, and then that will just produce garbage forever. It will direct the agent onto a bad trajectory, and then it just gets worse and worse and worse. And so, without someone else to correct that, then, you know, I I can't think of an architecture that gets you out of that without a sensible
15:08 human intervening. And I think that is the challenge currently is how do you prevent the death spiral that comes from a dark factory? >> I think at the end of the day it might be like a tools and language kind of issue, and and and that's one of the ways I approach it anyways is at this point that's what I have in my in my my tool belt and we're we're going to talk about that. I think people um are super hyped on on AI in general, but I have a a few few few topics here that I want to explore. But on the just while we're we're discussing the idea of factories and and getting started like I think people are I know for sure I see the kind of desperation in the organizational level of this meaning like organizations that have a bunch of software developers and have
15:51 have are a little bit behind the the sea change, but see what's happening. Um you recommend starting
Where should a team of 10-50 developers start?
15:58 with one issue a day and as it like a starting point and I'm curious about that. Like how do you how do you get started? How do you get you know like what's a what's a good way for an organization, you know, that has 10 or or 50 developers to even start considering um like adding this to their workflows?
16:13 >> Yeah, you want to start slow. You want to start small because it's just just all about building trust. That's what it is. As soon as you can get the trust level higher, then you know, you really are good to go. And I think a lot of it is about getting the agent to provide evidence and giving the agent the ability to provide evidence that their change is working.
16:36 Most organizations have agents that are hamstrung by inability to prove that their code actually works. They haven't um many like many companies will have systems that depend on other microservices that are very complicated to set up that uh need, you know, Kubernetes it running inside Kubernetes in order to like figure it out, right? Like most uh applications are not factory ready, you know? And so before you even think about you know, starting and accelerating the software factory, you need your brakes, right?
Start with the brakes
17:14 You need to get your brakes in first because if you don't push the quality high enough, you're going to turn your app into a slop cannon. And so that feeling of when you see an agent PR, when you know that it has run the code using a staging database, let's say that's very similar to production, you can actually it screenshots it for you and you see the thing working. It just makes it so and also that's rated its own blast radius on how um dangerous this is to merge, it just becomes so it's a different thing altogether than just trying to pass through some code that the agent has written that it's not verified. So, [snorts] I think when people see that, they start to believe and they realize, oh wow, okay, we can really push the accelerator on this and start getting more code out there. But
18:05 start slow and start with the brakes.
Raising the floor of failure
18:08 >> I think what's interesting there is that this is not a problem that emerged with the the agentic frontier that we are at. This is a a life lifelong problem with software engineering in general and we've done the best we can, you know, if you have Kubernetes in Kubernetes, somebody made that choice. That wasn't probably Claude that made made that specific choice. And now now we're we're dealing with that, but we're dealing it with, you know, like this this agentic intelligence that that we have access to. And I wonder like for organizations in particular if, you know, like I think we go to features like first like our brains, we start thinking about software features, customer facing features. And instead it's like that devops layer, right? Like like pushing it back and coming into there and building the brakes and building like I always think of it as
18:54 the foundation and, you know, when you and Lauren talked, it was it was, you know, like this is this is kind of the same thing. We're trying to raise the floor of failure. Right? Like how far can you what what happens if if stuff fails cuz it will and and how do you catch it and understand it quickly and raise the floor before you start trying to raise your ceiling, right? Like at the end of the day, like we need to like have a have a a solid foundation upon which we can build that that is agentic. And, you know, like then we we can successfully at scale, and this isn't, you know, like at at home experimenting, doing, you know, like playing, building personal software is great.
Personal software is not production software
19:29 Personal software is not production software, and production software is still extremely difficult even if you can produce software at like unprecedented rates, right? Like it's is it good software? Is it software that that we can count on, you know, that our that our our kids eat on this software. So, you know, there's it's different. You know what I mean? Like it like building all the personal tools you want is one thing. Building production software to me is a is a totally different thing, and and that's like a um >> Totally. It's the it's the number of constraints, right? It's the constraints on your system. And I think like I I I genuinely think if if people were able to like we have so much more uh labor available to us now, you know, why not get your agent to go and like aggressively look for issues in your system, you know, get
20:18 it to run your system, look for performance traces, look for um ways that it can improve things, look for database queries that are running slow. You know, take the information that's available about your application and plug it into agents. I I don't know. I think it's a very exciting time to be messing about with this stuff.
Reading the software classics with agents
20:35 >> So, one of the things I've been doing lately, and I know you appreciate the classics, is I have a collection a giant collection of PDFs that I've parsed and made like super agent friendly and have semantic search over and all this. And so, I'll be like, "Hey, go like dig through the entire like software canon and find the patterns, ideas, and abstractions that would apply to this or make our system more resilient." And and it's it's it is wild cuz it has most of that stuff in, you know, like it it is in the the the space that it has that it that it knows. But then it goes in and gets the specifics, and now, you know, with like larger context you can fill it up with with It's it's been fantastic and it like But what's hard is is then you're evaluating them and then also like if
21:16 you don't understand those and I think that we share like a mutual respect for these classics and I think as people reading them is giving you an advantage over folks that have not read them cuz you know what to ask for. You understand you have these in your own context in your own you know like your own neurons have these to bring up and convert and set the terminology and I'm wondering how like cuz I know you've been you know going through some of them and and bringing them up for your for your teaching practice and I'm wondering what other advantages you've seen for yourself from that study like slowing down and and giving you know like speaking of paper going back to the source.
21:51 >> Yeah, I mean I think of skills as process. Right? And what I'm trying to do when I write skills is I'm trying to figure out the best process of producing software and turning that into text. And so you know, people have been doing that for a long time and those processes are in books. And so having the literal words that you can call upon is incredibly useful.
Domain language, DDD and patterns agents actually read
22:15 Now, I've been really obsessed by language to be honest ever since for a long time I think for a long time too. Yeah, for a long time especially you know as a teacher and before that even as a drama student. >> I think they might call you a word cell. >> A word cell? >> Yeah. >> What with with a with a C?
22:34 >> Versus a shape rotator. >> What's a shape rotator? >> I'm going to leave that as a exercise for the for the listener. >> Sounds good. So this idea of language is just been obsessing me and I actually it's on my mind right now cuz I just I was late to this because I was filming the domain language part of my upcoming course. So specifically how language affects um the thing that gets built by the agent and how easy it is for that um inter-agent or human-agent communication to come around. And so, especially things like DDD, I've just felt have this whole new life now because the thing that always made these books sort of a hard sell in organizations is that you had the struggle of teaching people this process, right? You had the conversion layer uh of like, "Okay, I've got this process. We now need to teach everyone
23:33 this process. If everyone would just follow this process, we'd have an amazing result." But now with agents, they actually read the things you give them, right? >> Well, you have the guy in the meeting, right? Like, "Oh, this is the flywheel pattern." Or, "This is the strategy pattern." And that's great. Like, design patterns are like I I love them and I like I've always been a patterns nerd.
23:51 But when you say that and you see like blank stares from all of your coworkers because everybody hasn't read Gang of Four or they haven't read uh domain-driven design or whatever. And you're you're the book guy and you love to read and you like you're always trying to learn, but you can't get them to understand. Now, like I think we have a real opportunity to like drop cuz Gang of Four is is legit useful and there's some interesting patterns in there. But like, who has time to read all that? And it's in Java. It was written in like '99 or whatever, you know, like whenever that came out like a long time ago. So, it's like it's not doesn't feel relevant. It's not very exciting to read. I've read it. Um but now we have this opportunity and we bring that language in you know, like all like the
24:27 I don't know, the full history of software development is is well recorded. Um and we have access to a lot of that. Not to mention the source code. >> Yeah, and I do think that we as humans have an advantage over agents because we can take all that information in. We can filter it through our experience, through what actually worked for us. And then we just have a better sense, I think, than agents do for when to apply it, for when to pull that in. Um I mean I suppose that's not always true cuz when you read these patterns, you everything looks like a nail and you start hammering everything.
24:58 >> Yeah, yeah. >> But >> Like every meeting you're the you're suggesting the strategy pattern. >> Oh, it's the flywheel pattern again. Yeah. >> Yeah, there you go. >> Yeah. [laughter] Exactly. But I mean I don't know. I I found an enormous advantage from reading those things. And the idea of constraints being useful and there being you know, no social pressure on constraints, right? It's like I've got a a fleet an infinite array of people who will do my bidding without there being any social cost to me instructing them, right?
25:32 >> Personally, I I roll, right? Like the human and I love my fellow humans, but like you'd say that and you start talking about patterns or start wanting to to add these constraints and they would think about oh, this is this is oppressing me. Uh this is adding to my workload. I I'm already busy and you're in here talking about your patterns again like I'll roll.
25:50 >> I took that one one level further actually, which is at one company I was at I wrote a custom lint rule to lint uh an annoying pattern that my colleague was doing, >> [laughter] >> So I did this. And I cuz and he and he was like This was my boss actually. He was very lovely lovely bloke. And he said, "Did you do that for me?" I said, "Yeah, I did." So now I can do that for agents, right?
26:12 Like I now and they won't you know, they won't bring it up at every all hands afterwards, you know. >> They will actually try to work around them quite quite vigorously sometimes when you when you have rules in place, which is another [clears throat] another
Skills as a framework, and how to make them your own
26:24 fun topic. I don't want to I want to push back a little bit cuz you talk about skills as process and I think at the atomic atomic level skills absolutely are representing representing process. Um what I see evolving and I see it evolving through you and and uh potatoes skills is a framework. And I I have been just a long time I learned how to program through frameworks like Django and Ruby on Rails and um you know, like just over the year like Angular is kind of the the progression as a professional engineer that I took and I I was always like framework first and there was other people that were like no, language first. You have to understand the core functionality. You You have to, you know, know JavaScript if you're going to use JavaScript and I was like I do not. What I need is a really good
27:10 framework to start and I think what's interesting with skills is starting with a framework, like starting like with what you've put together and then or maybe what what Lauren has put together. Maybe combining those and then like how do you how do you think about skills as frameworks and then like what's your recommendation for like breaking free of the framework to like then develop contextually your own framework based on, you know, like to where we are are using this as a la- you know, speaking of ladders, we're using this to ladder up. Like you you are providing us with a great reference. This is This is something that works. Use it. It's well explained. Your whole team can understand it. But now what's the discussion that we need to have internally to build this into something that we own?
27:53 >> So, >> [gasps] >> I think that most people who are doing what I'm doing have all tended to align around a common process, right? Which is that you align with the agent before you build, then you build the thing, then you review it, then you have some kind of compounding thing that goes in fixes the process so it performs better next time.
28:16 That's common to pretty much every um system you'll see. And then we where it sort of alters is the amount of documentation and structure around that, right? Are you keeping your specifications around once you've built them, right? Are you um adding extra documentation so the AI can navigate your repo better? And that's really where the opinions are. And so I yes, it is a framework, I suppose, and it's a framework that kind of emphasizes some things more than others.
28:51 So, I think my framework emphasizes alignment and domain language, and maybe even, let's say, code review on the other end more than Lauren's. Lauren's is often more focused on implementation, getting more juice out of tests, and um sort of giving you more detailed playbooks in that zone. And so, I think if you think about the process as, you know, the thing you're building is, you know, five or six steps, let's say, and you then realize, okay, it needs X bits of documentation in order to get it working. So, in my case, a glossary.md and some architectural decision records are really the only docs I add.
29:33 Then you start thinking, okay, what other docs might we need? Or what other parts of the process might we need? How might we build on code review? How might we um add a bit of security stuff, which my skills don't really have. So, I think if you think of the process, and you think of the docs, and you think of uh the sort of points in the process where you have to uh initiate work or check stuff, that's really the whole thing. And you can just fiddle around with any part of that, combine skills together. I think there's definitely like a mishmash of like mine and Lauren's skills that would work amazingly. Because, again, alignments or grill grilling alignments combined with awesome implementation is probably a really good setup. And I think really the only place that skills differ are these um are are in the docs and in the process.
30:26 >> Yeah, and I mean, back to back to my point and what I my personal strategy with this in building software that that touches users and and to run my business ultimately is to review widely. Like, when you release a new skill, I'm going to go go check that out. I'm checking out retro today, which is a great skill and it's something that that I think is is interesting and I can go test that out, but then I will probably not use your skill off the shelf. I will read it. I'll actually put my eyes on it, read through it, and then bring that into our systems such that like it fits the way that we build software, but the idea is good, right?
31:04 Like we should after we are done do a retrospective, go through, look what's bad, look what's good, um and then like adjust the systems as we go forward. But you know, like the phrasing, the language, the pointers, the documentation, what exists where for me is going to be different than it was for you, but it's this generalized idea. Um and and you know, like thankfully people are generous and have public repositories and they're just giving, you know, giving away their their alpha for free and I think that's wonderful and and but I'm going to take advantage of that by then bringing it in and and um altering it to suit my needs when it doesn't cuz sometimes like, you know, I think I think grill me for instance, it's like that's pretty good.
31:42 We can probably just take that, but we should always feel free to like bring that in and make adjustments at the same time. >> Totally. And I think
Leading words: the most valuable part of a skill
31:50 you can steal like two main things from skills, which is process and then the The things that I think are most valuable in skills are what I call the leading words, right? Are the phrases, the compressions that people have made where they're able to take a large complicated process and compress it into a few words, right? So grill me relentlessly is like a very very solid, you know, um leading word or because it's creates this emergent property from the agent where it's does the thing.
32:24 Um and discovering those is really hard work and just stealing them from skills is incredibly easy. So that's usually what I look for when I'm looking at skills. >> And you will see like those leading words get latched onto and that's that's just a like by the nature of the technology and and you mentioned that already. It's like you can have code patterns, leading phrases, like these things the the agent will often attach to, and it's not everything, but it will attach to a specific thing, and then it'll just propagate. So, it's like choose your words wisely, choose your patterns wisely because, you know, like this is, you know, now it's going to be be stamped over and over again. So, you need to to like be cognizant of that as the operator, I think.
33:06 >> Totally. It repeats it in its own thinking traces, so it re-emphasizes it within its own context window, which just comes up again and again, and it leads it down a certain path.
One-way doors and two-way doors
33:15 >> So, you talk um this is my last question on factories a little because I think it's really interesting, the idea of one-way doors and two-way doors. So, decisions and and implementations where once you do this, it is going to be extremely, if not impossible, to go backwards. So, can you can you like dive into that a little bit and how you think about one-way doors?
33:34 >> Yeah, this is a big difference between personal software and production software, which is that and you really I mean this is Okay, let me define it first. This is not my definition, it's an AWS definition, I think. This is something they talk about at Amazon, which is when you have a pull request, it can either be a two-way door or a one-way door. And the two-way door is that we can merge this pull request and easily walk it back. It's a door that you can walk back through. But if it's a one-way door, then it's you go through that door and you never coming back, right? Or it's very expensive to pull it back, let's say. And it doesn't have to be a complicated code change, right? It could just be, okay, this goes into a repository where it will get pulled out and an email will be sent to 50,000
34:20 people, right? That's a one-way door, you know, even if it's a small change to a subject line or something, because, you know, you can't pull it back. So, when you're in some industries have more one-way doors than others, right? right? If you're in a legal profession, a medical application, then a lot of your decisions are going to be one-way doors, and that is going to really hamper the amounts that of velocity you can get out of agents, right? Because every one-way door must be reviewed by human, is my opinion. There's no way that an agent should be doing that stuff because if that could be an extremely expensive mistake that some no one has taken account account for.
35:01 So, yeah, having this concept of two-way doors, like there are a lot more two-way doors in software than there are in other professions, right? Like, you know, civil engineering or something, if you get a bridge wrong, like that is a one-way door. So, I think thinking of it like that will let you prioritize your reviews a bit more sensibly.
TypeScript and Effect
35:21 >> So, here here's let's let's walk it back a little bit and go into the olden times, um, when we when we had TypeScript and uh we had JavaScript frameworks that we we used and and had bitter just blazing battles on on Twitter and and elsewhere about which framework was better and and um I think at this point TypeScript is good. I think it's it's still good. Um, it's interesting to see kind of the the drop-off cliff of people actively trying to learn the language and and I understand and I understand why. Um, I'm still appreciative of my knowledge. I never went like full Matt Pocock in my TypeScript understanding, but like I I know enough I know I know enough to get by.
36:01 But, one of my favorite things now and and like I truly feel like if you are setting up these constraints and you're setting up boundaries, like a strong type system, this is why I think Rust is so popular with agents and TypeScript is so popular with agents. You get like strong types are are really good. Now, you take that to the next level and there's this library called Effect.
36:21 And I think it's one that I looked at and I was like, "Man, this is a really good idea." And the smartest people I know were like, "Hey, this is a really good idea." So, for for years, even pre-agents, I was like, "This looks like it's a good idea, but I ain't got time for all that. I can't learn effect." And I've I've slowly I've actively used effect for about a year and a half without having like bothered to like truly understand it, and I've been trying to rectify that. Like, that's one of my my current learning goals myself is to, "Okay, stop building for a second and actually learn." I don't think we're in a post-learning society. So, can you teach me a little bit of effect and and kind of tell me like the basics of it and and help me understand some of the
36:59 the weird vocabulary and the kind of the the functional nature of effect. Um and I don't want to, you know, I don't want to We could probably do a whole podcast season. Uh we could build cold courses around uh effect, but we're not going to do that. Just like, what's the like elevator pitch and what's the basics that I need to understand to use it in my applications?
37:17 >> Yeah. So, I think that people get the pitch wrong for effect. And
Agents require restrictions
37:23 I think if you think about Okay, the elevator pitch is that agents are Agents require restrictions. The more restrictions you put on agents, the better they're going to perform. And they also need flexibility around boundaries as well. So, you need a very boundaried application, and you need to be able to insert things into things, and like the code base needs to be a certain shape in order to work well with agents.
37:53 And effect makes that I would say not only simple, but inevitable. Right? If you're using effect, you are going to have a beautiful code base. As long as you impose the right things and you and you don't let the agent cheat. Now, what I think people get wrong is that they think that error handling and the type-safe nature of effect is the reason you go for effect. I think the reason you go for effect is dependency injection, actually. It allows you to create these beautiful small modules that you can just swap out as needed and create these really nice looking tests with effect, too.
38:30 And I think to understand effect, you need to understand that essentially effect allows you to put programs inside values. So, you just have a program that can just sort of sit there doing nothing until you need to execute it. That's very very simple. We've had that for a while as well. You can just, we know, >> have a function like that. It's pure >> functions, functional programming. Like it all kind of like traces back to those ideas.
38:55 >> Exactly. But it keeps all the dependencies, like that that program in the type system knows everything that can go wrong with it. It knows everything that it produces and it knows everything that it needs in order to function. And when you've got that set up, it's just so nice to work with because the agent has so many feedback loops it can pull on. It understands so much more about the program than it usually does. And having all of that brought to the surface by effect just makes it the best framework to use with TypeScript on the back end with agents.
Agents cheat: lint rules, automated review, human review
39:30 >> I've been trying and you mentioned cheating and I think this is interesting just because the agents do tend to they'll work around like if there's an easy path and this is something, you know, again and I'm it was such a great interview. The potato interview was fantastic and and if you're listening to this, absolutely um go check out that out. But um yeah, so they cheat, right? Like the agents cheat and they're always trying to get around and and they take the easy path.
39:52 So, when the easy path is async await, they're going to do that. And we don't want that, right? Like we're trying to build an effect app and effect is is more beneficial the more effect you you like the the more you're doing it along with the the standards and idiomatic way to build effect apps, the more effective it is going to be in your application. So, how do how do we stop them from from cheating in our in our apps? Like you can I I do I use like left hook, so I'm I'm, you know, pre-commit hooks, all these things, and they'll be like, "Ah, that's fine, Joel, but I'm going to actually work around that and you know, no verify you um anyway." But, like what's your what's your approach to stop the cheating?
40:29 >> So, there's three layers, right? There's automated checks are the top ones. So, lint rules that actually run on the implementer agent, and effect comes with a bunch of those. So, it can prevent all sorts of bad patterns. It hooks into the TypeScript type system, so it's aware of all the types and can lint based on those. Those are really useful, and of course, then you've got unit tests and blah blah blah.
40:52 Then, you've got automated review. Automated review, I think, is the one that is massively underused by most people. Most people just use their implementer agent, and then they never get a second look on it, right? I think people need to be using automated review because it's so good for catching these things. You just get another agent to Like, if you tell your implementer agent, "Don't do X," then that's bad, right? It will start thinking about X, or, you know, that's that's that's interesting. But, if you pass that to a reviewer agent, that then becomes a positive instruction, right? Then, the reviewer agent is goes, "Okay, I'm looking for X in the diff, and I will attempt to um find it and excise it." And so, I think automated review is essentially the the main one people need to be focusing on.
41:37 That's how I tend to stop my agents cheating, and it's most effective. And then, of course, there's human review on top, and then you want to do as little human review as possible.
Stateless on purpose vs an LLM wiki
41:46 >> One of the things I think is interesting that you've talked about is stateless on purpose, where your implementer agent won't have access to, say, all of your coding standards, but your reviewing agent will have access to those. And I'm curious, like how that works in in I take a fundamentally opposite approach, and I What another thing that I find fascinating is that people will take opposite approaches and uh reach successful end game. So, I think that's important for everybody to understand when we're talking about these things.
42:12 But, like I do think it's fascinating to explore the other side. So, tell me about stateless on purpose and how that works in these systems and maybe how that relates to what we were just talking about with the effect gates. >> Yeah, totally. The I think that statelessness in agents is uh really good, right? It's very nice because you can essentially work from the same state every time. People have this sort of feeling of I want my agent to remember stuff over time or I want a memory system. But, you sort of start realizing, okay, that memory system is going to retain state and build up state over time, and that's not going to be very fun because it might get some bad state in them. And so, the stateless stuff on purpose is nice. The I always think of review, or rather, I always think of the implementer agent as
43:00 super overloaded, right? It's got loads of jobs to do. It needs to build the code. It needs to explore the code base first in order to build it. It needs to then verify that it's working. And then if the verification goes wrong, it then needs to debug and figure out why that's happening. Whereas the review agent, it's incredibly underloaded, right? It's receives the diff. It doesn't hardly need to do any exploration. It doesn't need to implement anything. It just goes, okay, I'll just look at this and look at this, and then report its findings back.
43:31 And so, you can really pile on a bunch of coding standards into that reviewer agent, and I'm not sure it's necessarily like a stateful stateless thing at that point, but it just feels so much better when you have a fresh context window looking at it. >> So, and for me, I each one of my repositories, each project, has a full LLM wiki that it that it has access to and it uses Tiago Forte's PARA method, which is uh projects, areas, resources, and archives, and and sorts that. And then there's a higher-level LLM wiki that is the system wiki that then they have access to and they are connected in a a large knowledge graph.
44:11 And I actually get success with this. They go through and they they're they're good at searching markdown. It's all I actually use MDX. I actually use MD SVX which is the Svelte version of MDX to accomplish this. I think you'd hate it. But like it's also very successful. So I think that's interesting just the the like total different approaches and and you can do that and I think you know one that's important but then also like to explore them is my best advice. Like see what see what's working. Try it out.
44:38 Like I want I want to go try stateless. Like what happens if I implement these and they go in and they just go in cold. >> Yeah, it's a totally stateful system that you've got, right? And like the cost I see with that is the amount of updates, the amount of tokens you need to spend just creating and updating that.
Token austerity
44:53 >> I'm not austere with my tokens, Matt. >> Quite Yeah, right. >> Just like you you you've known me for a while now. You can imagine how I am with tokens. >> [laughter] >> Yeah, the carbon steel, you know, top layer of to Yeah, I don't know. Just >> Just Just bring it. >> [laughter] >> Yeah, whereas I tend to be more on the more austere side, I think. And I tend to want efficiency and speed wherever I I can get it. Um and so yeah, state statelessness feels nice.
45:21 >> I'm I'm jealous of austerity. Like people that that have that kind of nature in in in my life just cuz I I'm like I I need more of that but I don't I don't [snorts] I'm I think that ship has sailed, frankly. >> Yeah, just be you, man. >> Yeah, that's all we can
Top tips for building your own factory
45:35 >> do. >> All right. So top tips on building your own factory then. Like let's kind of review this and and think about it. Like if you you you had a suggestion and and like what what should people kind of take away from what we've talked about so far in terms of building their own factory? >> So I think start with the brakes. Kind of covered that already.
45:52 Um when you get to the accelerator, then put it on very very slowly. And I think look for data sources. Look for um opportunities for passing data to your agent that you've not explored yet. So, if you have a bug tracker or something, find a way for the agent to access it via our CLI. Find a way to get the, um, issues list searchable and pulled into your agent. And find ways that you can start, um, creating sort of reusable cron jobs that pick up and pull the stuff These are loops, of course.
46:32 >> that's what That's what people are talking about when they talk about loops is cron jobs. Like for the most part, like at the end of the day. Yeah. >> Absolutely. It's, um, or maybe let's say triggers that trigger issues and pull them into, um, coding agent sessions that then distribute stuff. Anyway, so that's how I'd start. And I'd also really prioritize cron jobs that just look at your code base and improve your code base. I think that the you can just get so much improvement, so, like, so much benefit from just looking at your code base and making it better, right? It's outrageous how much benefit you can get because every coding agent session afterwards will benefit from that. People underestimate the amount of steering that your code base can do on your output. You think of
Your code base is kind of like a massive agents.md
47:18 agents.md is the place where you put all of your instructions, but actually your code base is kind of like a massive agents.md but only code. So, >> They speak it and understand it and search it. Like at the end of the day. So, you know, like your agents.md is is instructions and they take those very seriously. They're kind of at the system prompt level, but then the rest of your your code serves as context and and it is part of or included in the prompt that you're providing and the the output that is is generated at the end of the day.
47:46 >> Absolutely.
Loops, OpenClaw, Soul MD and Vision MD
47:47 >> Yeah. So, one of the things that I think is interesting kind of in this this line is that and it and it's loops and we saw this earlier in the year if if you aren't using loops, you're way behind and are aren't going to make it. Peter Openclaw did a vague tweet that that told us all how far behind we are.
48:04 And I think there's another thing that I think that I take very seriously is when you were looking at that, when you see Peter Steinberger tell you that you are so far behind, you got to understand that he literally works at OpenAI and has unlimited access to whatever cooler models they have and all of the tokens that you can burn. And that's amazing.
48:21 And I love that they exist and I love that they're out exploring that space. But we as as mere mortals without access to like the literal firehose need to take that with a grain of salt. But what I do recommend is going and looking at what they're actually talking about and what they're doing and it takes a little bit of spelunking past the vague post into the comments and then into the repos because they release a lot of this and Openclaw in particular had a lot of great ideas. I don't you know, product-wise has a lot of flaws too, but the thing that I took away is soul.md. I love that. Like I put that everywhere.
48:53 My little agent has a lot of soul. Um um he's my little buddy. And then vision.md is something that they were using in loops and I don't know that it got a lot of hype and I don't think people really understood what they're talking about, but it's this vision.md file. And instead of the agent that is spawned in your repo, this is for the agent that is above your repo, the looping agent that is analyzing all of your repos and looking at them and they use vision.md to understand what can be fixed, what requires human review, and what they can act on. So the agents that are surrounding that are that are looping over your entire system are looking at the vision.md file of each project and using that as kind of the guardrails for what they can accomplish. What is acceptable to just like fix and change?
49:40 Look at the issues, look at this, what you know, like what needs to be escalated. And I've got a lot of mileage out this and I don't know if you've explored vision at all or what what you think about it if you if you haven't.
Tactical vs strategic programming
49:51 >> I think about there being two um kind of activities uh in programming, which is tactical programming and strategic programming. Tactical programming being the kind of like on the ground thing where you're actually fixing the stuff. Strategic programming being the long view, the kind of like the vision, right? As you see it. And I have not yet really experimented much with making agents good at strategic programming, right? I just not something I've found agents to be good at.
50:22 >> They're not good at it, right? >> Like they're they're there's not and like that's the idea of vision is is I'm going to, you know, like I'm going to tap that out and I'm going to explain in detail what the strat well, in in modest detail. Too much detail isn't great either, but like explain what the guardrails are and what the what the vision actually is. Like here is the strategy. This is what we are doing. You are not making that part up cuz they are bad at that fundamentally, I think.
50:47 >> I think maybe maybe >> [laughter] >> I think maybe they're bad at it because humans are quite bad at it as well. Like so the sort of median human is probably not a good strategic programmer. And because strategic programming is really hard to be get good at cuz the feedback loop is so slow, right? You know, you could change jobs before your strategic mistakes come back at you, you know.
51:09 So >> Intentionally change jobs before your strategic >> [laughter] >> Just keep running, man. Just keep running. Oops. [laughter] Yeah, [snorts] I just walk away from the explosion. It's fine. >> Yeah, yeah. >> Um So but that is essential for getting AI to a point where it can operate a dark factory on its own, right? If we could get agents to be good at strategic programming or find them some kind of obsessed with this idea that there's some framework that you can give to an agent that will cuz as soon as you get the agent good at
Turning the death spiral into a virtuous circle
51:44 strategic programming, you turn the death spiral into a virtuous circle, right? And it will just produce great code forever and you can just keep feeding it great stuff. >> I think, is the >> Exactly. So, finding that would be very nice, but I don't know. I don't know if it's possible currently. Um but I don't I haven't invested much time into trying to make it possible. So, who knows?
52:08 >> Yeah, I'm I'm one like support has been my my my
Support, product design and human teams
52:13 constant lifelong last 13 years um just albatross that I've had to to wear cuz like if you sell a project product to thousands of people, they are going to email you. And and rightfully so, they've given you money. Like they you know, it's not a complaint about that. It's just like this relentless like it never stops. Like it's a it's a daily every day and it's like these are the same questions over and over again. So, that was that's been like my my first line of attack in terms of of getting these things to do it. And and it does require some strategy, but it has this like bounded fence. So, like it's easier. But I saw people, you know, like like it was Kent C. Dodds and he's he's asking Lauren on on Twitter, like how do you, you know, like maintain product velocity and and steer your
52:57 agents? 2,500 PRs in a month is is is a lot, you know, 1.3 trillion tokens worth. Again, the firehose of tokens really helps out when you're you're on the like bleeding edge of this particular frontier. But how do you how do you keep them busy? And and like to me the answer is like you work with a good team of human designers that they keep the product going because like if you were sitting there trying to like maintain and and like be the the maintenance engineer or the person that's gluing it all together or implementing the features, it is really hard to also be like the fundamental product designer.
53:31 We see a lot of it. Like there's a lot of you know, like at that point you're like an indie hacker, right? Like you're alone. But when you see potato doing 2,500 PRs, that is because of a world-class product design team that you get to work inside and other other developers. So like that is facilitating that. That gives you this this expanse of work that you can do. And I think I think people see that and then maybe are like, oh man, that is, you know, like how how does one person do that? Or how can I get my agent to do that? Or or how how can I develop the product where it's good autonomously? And the answer is you can't.
54:05 Like in a literal way and I don't see any near-term future and and who knows? Like you can't predict the future right now. It's it is it is crazy. >> Yeah. >> But like I just I don't know. Like teams of humans designing products for humans seems like it's got like I'm bullish on that con- conceptually. I think we I think we have a little bit of time left before we are drinking protein sludge and >> I think the application of taste is
The application of taste as a differentiator
54:31 essentially a differentiator, right? Like because I think agents can get you to a nice floor in terms of a product and I think they can raise the floor on your product. But I don't think they can get further than that. I don't think you can get to like a linear level of execution, for instance, linear being the application.
54:49 Um or any of the great applications of the last, you know, 10-20 years without taste and without making hard choices. And I don't think agents are that good at that. >> Yeah. Like 100%. Like they they I mean, will they get there? It really goes back to the the same kind of discussion at the end of the day, right? Like they don't they don't have that capability for strategy. And taste and strategy and like for good strategy to actually be effective, you you need it's the application of of taste at the end of the day. Like you have to make decisions and and whether, you know, like you're Sun Tzu Art of War style, right? Like what it what it however you you look at this, like you are making strategic decisions based on your taste and experience and like context and understanding of the world and the
55:33 connections that we make and and um we're still useful. You're still useful, whoever's listening. >> Oh, sorry. I thought you were talking to me. >> No, you you included. I think I think we I think we still we still have a place in this business. Teach and the teach
The teach skill and giving your agent eyes
55:49 skill and learning stuff and and using AI in in interesting ways that um uh aren't necessarily software. Like for me, I I took a a version of the teach skill and had it like reintroduce me to my my elaborate synthesizer and kind of walk through and um me to learn a thing. And and I think that's like something that that is fun and people are maybe sleeping on a little bit, but it's also like a good kind of side project to do to to get in and and think about agents a little little differently and a little better.
56:18 >> Totally. I found that so fun to write that skill. It's one of the rare skills that I I just had like an hour and a half on the tube or something and I needed to um just wanted to do something with my hands basically and just got all of my ideas about teaching out into a single skill. And it was a really fun skill writing exercise. And I haven't really revisited it much since then, you know, people absolutely love this thing. It's you know, some video goes viral on it like every couple of weeks or something.
56:43 And it's just a really fun way of seeing that knowledge is available to you, right? Like it is just using public information and um public resources and scientific papers and stuff and just translating that into a language that you can understand. It feels really empowering to me. And so I yeah, I you know, I use it um once a week or so just to like ask a question or like um figure something out or just organize information in my brain in a new way. So yeah, it's it's a deeply cool way of using AI. I love it.
57:16 >> I got to say what I've really enjoyed doing and what I did with the synthesizer exercise is I hooked up a webcam and um gave it eyeballs and said, "Hey, you have eyes now. So like you you can watch while we do this." And literally if you're you're doing like an Arduino project, like, "Oh, I've always wanted to get into Arduino, but like how do I do that? How do I even get started?
57:35 You put a webcam on the desk and say, here's what I'm doing. Let's go. And it it actually works. It's amazing. I would highly recommend going and going trying that if you haven't like given it eyeballs. It's It's a lot of fun. I was like, well, they don't have hands. And now I want to buy like a seven of articulation robot arm and and like actually have it like do the like wiring of the I I don't know. Like then I'm not doing anything outside of but it's really fun.
58:01 >> How's that austerity going? Yeah. >> Bad. Poorly. >> [laughter] >> We're not going to get into it, Matt. Like don't don't We're not going out like that. Come on. Anyway, I appreciate it so much. It's always fun to work with you. It's fun to listen to you and learn from you. I I can't wait to to see what's coming up next and yeah, I'll talk to you soon.
58:19 See you soon, Joe. Cheers.