How Developers Are Actually Using AI — ForrestKnight
ForrestKnight investigates why AI coding tools produce sharply different productivity gains across developers and teams. Drawing on examples and reports from Amazon, Anthropic, and others, the video argues that expertise, task preparation, and engineering workflows matter alongside the tools themselves.
videoForrestKnight · 19 min · Nov 9, 2026 · youtube.com · Saved Nov 13, 2026
coding agents, ai developer productivity, engineering teams
Description
It's not the size of the model that counts, it's how you use it.
Try out Grok Bot: https://jetty.to/grokbot
So as developers, we’ve been promised a lot. For about two years now we’ve been hearing some version of the same thing. With AI, you’re gonna actually become a 10x developer. You are gonna get so much more work done than ever before.
Now, I gotta be honest. On my personal projects, oh I feel this. I’m seeing gains that blow way past 10x. But my side projects aren’t exactly representative of the industry. And even within the industry, everyone seems to be getting different results.
I have dev friends in just about every part of it, from the big AI labs, to the Silicon Valley startup that just raised $50M, to enterprise like banks and defense. And what I’ve noticed is just how different their experience with AI is. Some of ’em are seeing only 2x gains, if that. Then you take a look at some others and they ship 2,000 PRs in a real production codebase in a single month.
So I wanted to find out why. Why is there such a big gap in the benefits devs are seeing from AI?
Sources:
Lauren Tan, “shipped 2,500 PRs last month to production” (X, Sep 21, 2026): https://x.com/poteto/status/2102050467505430555
From AI-Assisted to AI-Native: Building a Frontier Development Team, Clare Liguori (AI Engineer World’s Fair, Aug 28, 2026): https://www.youtube.com/watch?v=pqlWNihgdjI
Redesigning source control at Anthropic (Anthropic, Oct 13, 2026): https://www.anthropic.com/engineering/redesigning-source-control
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (METR, Jul 10, 2025): https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
How to Quantify AI ROI in Software Engineering, Yegor Denisov-Blanch (AI Engineer Code Summit, Nov 2025): https://ai.engineer/talks/JvosMkuNxF8-quantify-ai-roi-in-software-engineering-stanford
AI Engineering Report 2026: The Acceleration Whiplash (Faros AI, 2026): https://pages.faros.ai/hubfs/AI_Engineering_Report_2026_The_Acceleration_Whiplash_Faros.pdf
How frontier teams are reinventing AI-native development (AWS, Jun 10, 2026): https://aws.amazon.com/blogs/machine-learning/how-frontier-teams-are-reinventing-ai-native-development/
Kiro (AWS): https://kiro.dev
pstack, Part 1, Lauren Tan (X, 2026): https://x.com/poteto/status/2094457600259842065
Introducing Claude Opus 4.5 (Anthropic, Nov 24, 2025): https://www.anthropic.com/news/claude-opus-4-5
How teams build, Edition 01 (Linear, 2026): https://linear.app/data
Wait, I thought Java was dead? (ForrestKnight, 2026): https://www.youtube.com/watch?v=HhPs2sKlQYg
A portion of this video was sponsored by Grok Bot.
My Newsletter, Dev Notes (coming back soon) 👉 https://www.devnotesdaily.com/
My Socials
🐦 X: https://www.twitter.com/forrestpknight
🐙 GitHub: https://github.com/forrestknight
💼 LinkedIn: https://www.linkedin.com/in/forrestpknight
📸 Instagram: https://www.instagram.com/forrestpknight
Transcript
0:00 So, as developers, we've been promised a lot. Over the past two years, we've heard the same story said a thousand different ways. With AI, you're going to become a 10x developer. You're going to be able to get so much more done. And now I got to be honest, on my side projects, oh, I feel it. I'm seeing gains that blow past 10x. But my side projects aren't exactly representative of the industry. But even within the industry, everyone seems to be getting different results. I have dev friends in just about every part of it, the the big AI labs, the Silicon Valley startup that just raised $5 or $50 million, as well as enterprise, banks, defense, healthcare, what have you. And what I've noticed is just how different their experience with AI is. Stripping out any of the red tape, when they're all allowed to use all the same models, all
0:48 the same harnesses on the code bases, some are seeing smaller gains up to 2x if that. Whereas on the other hand, take a look at Lauren here. She has shipped 2,000 good PRs in a real production code base in a single month. So I wanted to find out why, not how she was able to ship all that because that's a whole video on that tweet that you can go ahead and watch. But why is there such a big gap in the benefits that devs are seeing from AI? So I read through everything I could get my hands on. And one of the best stories I found, they took 50 different teams in the same company working on existing systems with a normal mix of software engineers from junior to senior and gave them all the same AI tools. Well, except for a few.
1:30 And a bunch of those teams saw less than 3x improvements, whereas some other teams saw over 10x improvements. And there are very clear reasons why that we can replicate and do ourselves. But before we get into that, a quick word from today's sponsor, Grokbot, which I've found to be one of the easiest ways to really utilize AI in so many different facets of my life. Whether it's personal, like I needed to shop for a 2 to 4 TB SSD that has a Thunderbolt port, or I have to research on what developers are talking on X or what they're watching on YouTube. But it's also very helpful when it comes to engineering. Now, I created a fresh engineer bot here, connected it to my GitHub, and I'm going to have it take a look at my Jetty repo, look at the PRs, and tell me what each one is about and
2:14 which ones are highest priority at the moment. And as that works, I want to talk a little bit more about it. One of the best quality of life things about Grok bot are the connections. Take a look at all the connections you can give your Grok bot. If you're like me, you have it hooked up to Cloudflare, Vercel, Google Sheets, PostHog, X, Convex, as well as GitHub and Origin. And for the engineering tasks, it actually uses Cursor cloud agents. So not only are those really good cloud agents, but also it means I can control everything from whatever device I'm using, whether it be my phone, my laptop, or my desktop. And after just a few minutes, there we have it. The most urgent PRs in that repo are 81 80 85 and GitHub says they all merge cleanly. So what I'm going to do is go
2:56 ahead and merge the three most urgent PRs and then I'll take a look at the rest and see if I want those to be merged right now as well. So if any of these tasks are something that you're interested in automating or you're like me and you have thousands of unread emails, well try out Grok bot at the link in the description or just go to jetty.to/grokbot.
3:12 Now, I know I already showed you this Lauren example, but she's not the only one shipping at these high rates. You have Anthropic completely rebuilding their Git because Git couldn't handle their throughput of roughly 30,000 merges a day. And these aren't even close to the only two examples, they're just the most notable ones. So if the labs are shipping like this, why can't normal companies? Well, they kind of can. And no, it is not by just throwing money at it. I know that's the first thing everybody's going to say because of course it's going to cost money. And these AI labs have infinite compute and they're also heavily incentivized to ship with AI. I get all that, but money is not the deciding factor. What I mean is that if your company just started throwing money at a bunch of inference and tokens, you're not going to be
3:57 seeing these insane results. There are a lot of other more crucial factors at play here, which is exactly what I went looking for. I went through study after study, listened to stories from all these different companies, reading all these different reports. And what I found was exactly the same as what I was hearing from my friends. The results are all over the place. I mean, I saw anywhere from -19% increased output to, you know, 2x and above. But the story I mentioned earlier is what really caught my eye. To reiterate, 50 ordinary teams in the same company working on existing systems, not building something from the ground up, with a normal mix of junior and senior software engineers. But the results they saw being anywhere from under 3x, but we don't know how low, all the way to over 10x. We don't know exactly how high, but
4:45 that's the rough range from 3x to 10x. And the company that did that was Amazon. And I know, okay, they have money, this that and the other, but there are countless [laughter] there are countless companies that have, you know, three developers on hand and they're just throwing AI at it like crazy and they are shipping just as much as the examples I just gave you.
5:04 But that's never the example that people want to see. They want to see it in an actual team environment that isn't just a startup, but a more established company. And the only one that's giving these studies and these stories out to all of us, well, it's Amazon. But they didn't just wake up one day and think, let's just throw AI at 50 teams. It actually only started with one team. So I want to lay the groundwork of how everything happened because it's important. And they started it with the Bedrock team, which makes a whole lot of sense considering, well, that's the service that hosts Claude and GPT for AWS customers. But they needed a whole new inference data plane. To build this out, what they had estimated was they needed 30 people in 18 months to do it.
5:46 Instead, they took six engineers, gave them Kiro, which is their in-house Codex or Claude alternative, and built it in two and a half months, which sounds insane. And and it is. Don't get me wrong, but it's a little a little bit less insane once you realize who did it. The six engineers were some of the best in the entire company. People who already knew distributed systems and LLM architecture inside and out, and two of which being distinguished software engineers. So Amazon tried it again, but this time on Prime Video with a different six software engineers, again using Q and they gave them a 10-day sprint. And based on what they got done in that 10-day sprint, the project estimate that they were working on dropped from getting done in 90 weeks to only 24 weeks. But something very important to note here, a senior engineer had spent three weeks before
6:38 this even started writing out small, detailed, well-scoped tasks for them to just be able to churn out. And then we get to the actual 50 team test. Amazon Stores, which is amazon.com and the physical stores, they ran this real test. The 50 teams, the existing systems, the you know, mix of junior to senior software engineers and just let them go wild using AI tools, Q to be exact. And then they just kind of sat back and watched. All throughout 2025, they weren't looking at the amount of commits coming through. What they were looking at was how many things [snorts] made it into production. And this is where we get the number where half of them actually came in under 3X. The median was about 4.5X and with some teams being over 10X. So how with all of those variables in same company, existing systems, mix of engineers,
7:31 why is there such a discrepancy here? Well, it's actually pretty straightforward. So the teams that came in under 3X, they just took Q and just kind of >> sprinkled Q and some of the other tools that we have on top of their existing way of working. >> Any habits that they had in, you know, coming from previous sprints, they just they didn't change anything other than plugging Q in. That's the wrong way to do things. I mean, they still saw results in that lower half apparently up to 3X. But the teams that did really well fundamentally changed how they worked. And we know this because Amazon interviewed all of them and their takeaways were habits. And they call them habits instead of like hacks or tricks or whatever because it it truly does need to become a habit. You need to change the way you used to work into the
8:18 way you're now going to work with AI moving forward. And the first habit of these teams that saw all this success, they wrote everything down. Everything that was over in Slack, in their head, in the code review comments, and everywhere else, they took that and put it into their skills file, put it into their steering files, gave it to the AI as context one way or another for it to have access to. But not only that, every single time the agent did something wrong, they didn't just, you know, type a bunch of cuss words at it because, spoiler alert, AI isn't alive.
8:53 You as the human need to take responsibility. They thought to themselves, what did I not tell it? Like, what did I do wrong for it to come up with this conclusion, which is not at all what I was asking? Like, how can I make this process better? And then as the models began to improve throughout 2025, they started removing rules because maybe I don't need to be so explicit with some things or give it these rules because the models are getting better. And that is an overall trend that we have seen over the past one to two years is that is you don't need as many rules. And by me saying that is also not a rule, it's just something I I've personally experienced.
9:30 Obviously, people love, you know, the skills repose, the P stack, the mat po skills, you know, all of those things. I'm not saying those are useless by any stretch of the imagination, especially not P stack. That's something that y'all should absolutely go look into. But what I'm saying is that, you know, the more rules you have, the more context it has, you may just be blowing out the context. And what's nice about actually today than when this story took place is that you can connect your AI agent to all of these things. You can have it connected to see all of the comments in the code review.
10:02 You can have it connected to Slack, certain channels in Slack, so you can see all of those notes that you have written down. You you necessarily have to pull it anymore from all these different areas and give it, you know, put into the skills file or what have you, as long as you direct it and connect it to where it can get that information, as long as it's not in your head. And the second habit or thing that happened here is that all of the good teams got slower first. They noticed their productivity slipped at the very beginning. But again, that's because they are building out the process. They are building MCP servers for the AI agent so it can connect to everything that it needs to connect to. They're fixing error messages so the model could tell exactly what broke. They're even restructuring the entire codebase so the
10:45 AI agent could navigate it better, which I have thought about making an entire video on this of how to actually restructure your entire codebase from, you know, pre-AI era to now with AI era. If that's of interest to you, just let me know. And some teams even went as far as changing languages. They went from maybe Python or JavaScript to obviously the superior TypeScript or Rust because you have the type checking and you have the compilers yelling at the agents for you so you don't have to do all of the back and forth. I have talked about this ad nauseum in my past videos.
11:23 There is a little asterisk there that may I will be going over in the next video or two, by the way. So stay tuned for that. I know two video plugs in a row. I'm sorry. The third habit was they didn't babysit the agents. Listen, if you're going back and forth with, you know, a single agent or a couple agents all day where you are, you know, responding to it every 30 seconds or minute exactly when it is finished with a single task, you're never going to see the four, five, six X improvement because you're the bottleneck. You only have so much time in the day. What the frontier teams did, they actually only wrote one to 2% of their code by hand and for everything else they would run agents that were looping for hours. And obviously they would run multiple at once working on their own things, which
12:06 is is much more common nowadays. You have to remember that this study, this deal happened in 2025. It was a whole lot less common at that point to be looping, having an agent loop for hours. For example, Ralph Loop, if you remember that, I'm pretty sure that happened in 2026. It's been a long year. A lot has happened. So the way they got there is that they were able to set it up and more or less ask the agent to keep going, check its own work, and then come back to me after everything has compiled, ran, all tests are passed, which means that it's going and it's building and it's compiling and it's fixing those errors and then that'll have new errors and it's fixing those errors and you get the idea. This is actually kind of Lauren's whole thing too with P Stack. The idea there is to
12:46 get like one agent that you you build up to where you really trust it and only then do you start stacking them in parallel. The fourth habit here is that they write the spec before the code. Now this is more or less plan mode. I don't really use plan mode anymore. I used it for the first time the other day in in a very long time, but the idea of it is still sound. It's you don't want to just give the AI a vague prompt and then once the code comes back and however it's implemented, then you're arguing with it about the implementation and about the code and about changing it here there and and whatever. You want a clear picture to give to the AI of what you actually mean. This doesn't mean a whole entire spec list like this for every single thing you want. I actually think
13:28 that's overkill and that's that actually makes it worse. You don't need to over engineer your prompts, but you still need to be specific with your prompts even if they are smaller now than they were before. This also derives from the fact that Amazon engineers I think have always done spec driven development that which is also built into their tool Kiro. And also it it saves money to, you know, go back and forth in regards to a document, you know, in plain English then trying to go through all of these diffs that the AI had already generated of all this code and then trying to make the changes then. So not only is that the worst way to go about it, but it's also the more expensive way to go about it. And the fifth and final habit that they came to the conclusion of the, you
14:08 know, 10x teams within Amazon is implementing something that everybody should already have implemented. tests, linters, integration tests, performance tests, security tests, this whole entire pipeline, which is what I encouraged every single open source repository out there to build themselves if they want to continue taking external contributions because it just makes everything so much easier in the AI.
14:32 Well, they made sure it was built out properly. Not only because, you know, you should do this, but the better it is, the higher the ROI. Like the ROI is there now more than ever before because that is one of the elements that lets the agent continue to loop and test beyond the compiler, beyond the type checks, things of that nature. I guess that's included in there. But to run all of these tests and run through the entire pipeline until it fixes all of the mistakes that these tests in the pipeline find. And something interesting that they do is mocking out all of their services locally, on their own laptop, so that way they can have a quick feedback loop with deterministic responses.
15:13 >> Because it lets the agent do everything locally. Um doing everything on your laptop without having to spin up a bunch of other services and and connect to cloud services makes everything a lot faster. >> Because when you have a faster feedback loop, the more loops an agent can go through quickly, and then the more loops that agent can go through in theory and in practice that we found, the better the output. So that that's the gap. They all had the same tools. It's just that, you know, the under 3x teams, they didn't lay the groundwork. They didn't write down their context. They didn't have the pipeline in place and the tests in place and everything in place in order to remove themselves from the loop. But what's craziest about all of this, which I mentioned the fact, but I really want to point it out here, is that all of this
16:00 happened in 2025. Opus 4.5 was released on November 24th of 2025 at the end of the year. All of this happened pre Opus 4.5. This is with models like Claude 3.7, 4.0, 4.1, which if if use these models, you are keenly aware of just how bad they were relative to models today. I mean, you'd ask 3.7 to make a small refactor somewhere in a single file, then it would come back with 20 changes that you didn't want from every corner of your codebase. Imagine what the output of these teams looks like now with, you know, the models we have today. Oh, which reminds me, they're actually doing a new test in 2026.
16:39 >> And that's what 2026 is about for Amazon is how do we scale this out to more and more teams to the next 2,000 teams instead of 50 teams. >> And oh boy, I can't wait to see the results on that one. Because if you have the right systems in place, like the good teams at Amazon do, and you follow all that I laid out in this video, models do greatly improve how much you can get done, as well as the entire ecosystem being built around it. I mean, I hesitate to say this because I feel like it's just common sense, but I know some people are on the outside of, you know, coding with AI and things of that nature, so I'm going to say it anyway.
17:16 The amount of progress we have made from 2025, a year ago, when this story was taken place to today in terms of models and harnesses and other tools in the entire ecosystem and how they can all be connected, and also just how the models work, it is a night and day difference. So if they if some of these teams are seeing 10x back then, I can only imagine what they're getting done today. So if you're sitting around wondering, you know, where are all these 10x devs that that that we were promised, they're here. You're just not going to see it if you were solely relying on the model and giving in minimal effort into using it. It's the same idea, the same concept of working by yourself building something and working with a team building something. Those are two completely different systems and rules that you have to play by in order to
18:06 optimize and be useful from how your codebase is structured to how your tasks are structured to how everything ties in together. The other layer to that is AI agents. But that's all I got for you today. I hope you enjoyed the video. Hope you learned something. Hope you can take something away from this and apply it into your own development life cycle. Or maybe you just want to stay here and watch more videos. If that's the case, well, there's a couple on screen that YouTube thinks you'll like. So, give them a shot. Maybe you will. But uh anyway, I will see you in the next one.