How We Built an AI Code Reviewer for 200 Engineers
Serhii explains how they built an AI code review system for 200 engineers, including its context retrieval, feedback loop, and autonomous PR approval workflow.
videoAI Native Dev · 34 min · May 29, 2026 · youtube.com · Saved Oct 28, 2026
automated code review, context engineering, pull requests
Description
One developer. One bot. A pull request merged to production with zero human reviewers.
That's not a hypothetical — it's a real merge.
Serhii spent a year building an AI-powered code review system on top of open-source PR-Agent. In this talk he walks through the six decisions that made it work at scale across 200 engineers.
What we cover:
– Why they chose open-source PR-Agent over CodeRabbit and Greptile
– The two-stage context retrieval system that made prompts actually useful
– How they used Gemini Flash to filter 5,000-token documents down to 40 relevant sections
– Why feedback loops from day one caught a silent Gemini model degradation
– The "self review done" command that lets bots approve PRs autonomously
– A practical playbook you can run in your own company next week
Chapters:
00:00:00 Introduction
00:00:43 About Serhii and the talk
00:01:00 Background: AI-powered code review
00:02:09 Why they built it: the data behind repeated review comments
00:03:29 Tool selection: CodeRabbit, Greptile, GitLab Duo, PR-Agent
00:06:38 Architecture: from GitLab CI job to a Kubernetes listener
00:08:18 How the review pipeline works end to end
00:10:57 Working with prompts and why context gets hard at scale
00:12:19 The AI-powered context retrieval system
00:16:41 Integrating Jira tickets for PR compliance checks
00:17:19 Model selection: OpenRouter → Vertex AI + Gemini
00:19:03 Real examples and the self-review flow
00:21:27 Building a feedback loop with Grafana
00:22:34 Results: 300 reviews/week at MVP scale
00:23:05 The six-step playbook
00:24:49 Q&A
🌐 Tessl: https://tessl.io
🔔 Subscribe for weekly videos on AI-native development
What's your take — would you trust an AI to approve and merge your pull requests?
Transcript
Introduction
0:00 I'm bored to do the same stuff. And we are living in the AI era, so why not to try to use all these tools that we have available? And this is literally what happens. So this is like one of your merge requests from our like production GitLab that from the beginning till the end one developer and a bot.
0:22 And it was like merge without human interaction what we're going to talk today is about like six things that you can do maybe like in a week and do something similar in your company.
About Serhii and the talk
0:43 Oh, no. Oh! Welcome today. So my name is Serhii. I'm currently an engineer manager at Marks & Spencer. But for the next 30 minutes I will be talking a bit about different topic in my previous company that we were doing AI powered code review.
Background: AI-powered code review
1:00 So it was a legal tech SaaS. And we're going to talk about three different things, like how we decided with the tools, how we build circuit, and how we work with the bronze. And of course, like a lot about context, how we work with the context, what models we choose, how we choose the models and actually talk about the price.
1:21 So where it actually all started and it started with just a message. I think that week I was doing the same code review message like for the fourth time in a row in the week. So I was like, I'm bored to do the same stuff. And we are living in the AI era, so why not to try to use all these tools that we have available?
1:44 And I think at that point of time, Code Rabbit was quite popular on X or X Twitter. And I was like, do you want to give it a try? One of the important things that was considered is we need to have self-managed instance, because we have a lot of different compliances and some different stuff. So
Why they built it: the data behind repeated review comments
2:09 in the months we actually shipped something and I'm going to talk a bit more. So I usually try to stick everything with the data that we have. And I was thinking like, is it only me who have the same issue again and again and again? So I just get the data for the last like a quarter, and there was a lot around like 2000 merge requests.
2:31 And I was trying to find do we have like a lot of repeated stuff what our like engineers. Right. Again and again. And actually it was so I just put like this five, five top categories. So it was about null safety test gaps or crossing concerns and all the things. So I was like, all right, so where are we gonna go from there?
2:58 And how we actually can say how much money and time we waste. So we have 200 developers. We have hundreds of merge requests per week, and we have all these patterns. I don't know the final number, but it's painful. So this is the reason why we just thinking we need to try something that will save money and time.
3:23 And at that point of time, and it was about a year ago maybe when it all started.
Tool selection: CodeRabbit, Greptile, GitLab Duo, PR-Agent
3:29 Claude Code was just starting. It's really good right now. And mostly we had Cursor or just Copilot in VS code and it was like really good for autocomplete but not really good for all this like complicated things. So where what we're going to talk today is about like six things that you can do maybe like in a week and do something similar in your company.
3:59 So I will talk about how we evaluated tools, how we build the architecture, how we worked with prompts, how we work with the context, how we choose the models, and about the cost. So it's just part of the table that I spent some time doing the evaluation. And at that point of time, it was like four tools that get most of the attention.
4:23 So CodeRabbit, the most popular. GitLab is here only because we were using GitLab Enterprise. So it's something that they try to build and try to get working. So Greptile is something that was recommended by our VP of engineering. And something we tried is like to find the best open source model, open source applications that actually doing the same stuff.
4:49 And at that point of time I was told money is not a problem we are paying. So the decision was done to choose Greptile because it's actually like just having a checkbox of all our important things. So it's check and security. It checks and quality because it's like just the tools that we are paying a lot of money for and a lot of operation stuff.
5:15 We should not worry about updates, model scaling. So we just consuming the stuff. But in the end of the day we were like, all right, we need to change some. We need to try something new and something interesting. I was here one month ago on the City of Craft event, and the guy who is the CEO of Tessl was speaking, and I was asking him, what are the best companies that get the most of AI?
5:42 And like he told quite interesting things that I remember is like is not a startup companies, whereas they have just few engineers and not an enterprise company because there is too much bureaucracy. It's something like in the middle. And this actually worked because I was told, you have time, just try it like we give you like two sprints.
6:05 So try this open source tool and we'll see how it will go. So few things. Why we decided to go with open source is because we will own the code. I mean like of course we will fork it so we will own the prompts, we will choose the models and we can we can pretty much implement whatever we want.
6:27 An important part if the company will scale, it doesn't matter how many seats we will need. So. This slide is
Architecture: from GitLab CI job to a Kubernetes listener
6:38 more is more like to show you how easy it is to actually run it in production as the first version. So around six weeks and it's literally set up that we had in our GitLab. Sorry. In GitLab CI, we had a job just to run an image with some comments that will add two different comments like first command is review that will provide pull request review as a senior engineer and improve is a comment where there will be recommendation in the code itself.
7:16 And for six weeks we have it like this. And it was not really scalable because we don't have a lot of control over it, and we need to change configuration every time. Push, push. Another change gets the review. So we're thinking like all right. So we see it works and we have some positive signals what we can do next.
7:41 What we did is like we set up a dedicated service. And instead of just running it on demand as a job in GitLab, we added a Kubernetes pod that was like running 24 over seven and was working as a listener. So in this scenario, we actually give everything under our infrastructure, all the helm charts, all the other DevOps shenanigans that's happening.
8:10 Everything was in our control as well. So how does it work?
How the review pipeline works end to end
8:18 So at first when developer create a merge request in the GitLab, we are listening for the event. So we say they're open up date or comment or like when the event like when the request is going from draft to ready. So we're having a listener that is like always listen for these events from the GitLab. Then we're going for the event router for the slash commands or like after triggers.
8:49 Next step is pretty much like the heart of this review is the context retrieval. And we're going to talk about each step a little bit more afterwards. Then we go into the model chain. So we are choosing which model need to make the review. And we started with a smart first because we really want to have good reviews.
9:08 And then afterwards like the bot just sends a comment in the merge request. And also it's an important part of all of this architecture is having a feedback loop from day one. So we ask our developers just to mark their the comments from the bot with like green, yellow, or red., yellow or red. And then we use the GitLab emoji API.
9:34 We can pull all of this data and like have a build everything in Grafana to have it visible. So we have two entry points. So first one is mostly for power users who really want to have like most of the control, most of the features, every time they create a merge requests, they can manually write down slash command.
9:56 So we had like review improve describe. That is literally going to create pull request description for you when you have it by yourself. And also ask when you just want to ask a question to to the code base or like to the pull request. And you can actually do on someone else pull request if you're not sure about some specific thing and after triggers.
10:18 So when merge request is opened and is going directly to non draft state, we will run a review and the second condition is like if the merge request was in draft and then was moved to ready state, we just like triggers the review again for everyone else. So in this case we have like two categories of people who should be happy.
10:45 So usual users that don't really care much, but really nice to see pull request review and someone who rely on it a lot.
Working with prompts and why context gets hard at scale
10:57 Right. So how we work with prompts and I think this is something that was like started like really, really well but quickly started to become really, really hard because you cannot like when we have only like five rules or ten rules specific to our companies. It was fine. Like we just sent the just one single prompt and it's working as expected.
11:21 Good results. But when we start to scale all these rules, all this documentation, all the things, so it easily started to appear that we have real problem. We have like context, but we have so many things in one prompt and the results getting worse and worse and worse. So then we went to the second version of like what we implemented. So what if we will start to run rules only to specific like pass, maybe like something related to the DB will go only data data rules.
11:57 What if something like with TypeScript will go to the TypeScript rules? Right? Makes sense. But what will happen if there is like some files that is like services DB and it's TypeScript. So every single file explodes, and then the the most important part that was chosen
The AI-powered context retrieval system
12:19 is like how we can use the best of AI to choose the context for AI. So what we did. So we have few different inputs. Right. So first of all it's like pull request metadata title description diff. We have a huge library. There is like different rules documents markdown files. And we actually pull in the JIRA context ticket description all this stuff.
12:52 So first and first we cannot just take all this like library and pull into prompt as we talked before. So what we are doing. So first we are doing heuristic pre filter. So out of all the documentation we build a ranking according to the keywords in the pull request. And even without AI trying to build the top 30 to 50 documents that actually related to the pull request.
13:24 So we filter out even like on like pretty quickly. And then we actually can stop here. Right? We, we know the documents that we're going to use and we, we already have everything ready. But what we've like all these documents have 5000 tokens. 10,000 tokens. So which is already like crazy. So what we did. Next we're using cheaper model.
13:49 So Gemini flash with temperature zero just to hold. We have the same results. And asking AI according to the changes in the diff. According to the context, what actually the documents feed the most for this pull request. And then in every document we had an agreement to always have headers and always have like everything split into sections.
14:17 So then we actually have all these like documents, all the sections, and we are doing second call just to focus on specific sections that we want to have for for this pull request, we also have some kind of like fallback chain just to make sure everything is like working as expected. And in the end, what we have after all of this happens, we have 20 to 40 sections.
14:43 Everyone, every each of them is like up to 300 or maybe less. And just sending it to the LLM. So this is like a real example how it works. So if we will take a look here, we have most likely you don't see it. So I will just say it. So we have like keywords like services getting paid payments invoices stripe.
15:12 And in the end we have a line where we already see what documents were used. And we see that like we use the documents regarding getting paid, we use service stuff, we use stripe. So we only use documents that do matter. And how it looks like in the end, because in the end, like it's a huge, huge problem that they send to LM with all the rules, with all the files changes, with all the gifts, and we want it to be as concise as possible and to include all the critical information as possible.
15:47 So as a part of the prompt, we will just have project documentation, context, what files were used, what sections were used, and what what the what was inside the sections. So we're also thinking about how we can make sure we are not doing too many requests when we don't need it. So we actually implemented like five minute cache per so per merge request where we can just return the same context.
16:17 If let's say we use two different comments like we use review and improve it, it's like given two different results, but the context is the same, right. So why we need to build it twice right. So also what we added is like how we can include JIRA into the context for the merge request and into the PR review.
Integrating Jira tickets for PR compliance checks
16:41 And actually what we noticed this was like 87% of child tickets were empty. Description. So if you will just include the empty description doesn't matter. So we also added the epics and we actually added function to check if ticket. Sorry if merge request is actually compliant with the ticket, because are we actually writing the code that we should write?
17:08 Is it right? Is it something that we were expecting? So at first because we were experimenting with another choice, it's going to work or not.
Model selection: OpenRouter → Vertex AI + Gemini
17:19 We use OpenRouter as a main model provider. And there was like a lot of reasons why to do so. It's like one API surface. Many models, we can choose whatever we want. It was fast to implement. The prices are the same as from the vendor, so why not? But few things that started to appear as the problems is like not all the models were fitting for our security.
17:48 Security like policies. And we need to switch to something that vendor provided. And at that point of time in the company, we already had vertex AI, and it was pretty obvious why we chose Gemini as a main model provider. And we we have much more control over it. And we have like clear monitoring logs, budget security stuff.
18:14 So it was an easy choice for us. So it's important to understand that AI review is more like a suggestions, and it's more like an advisory thing to do because like, I still can hallucinate, I can give bad results. And I think something that Simon mentioned before is like, what will happen if the there is like prompt injection or something like really bad happened.
18:44 So we still having this as a separate line as in the merge gate. Right. So we still make sure that CI/CD is working as expected. All the linters, all the tests, all the stuff is running there as well as Snyk. So and AI is going there more like as an advisor.
Real examples and the self-review flow
19:03 So this is like few real examples. What we what we had in the end. So the bot itself just like write down in line comments. So developer can just apply suggestions or you can literally chat with proposed changes. You can agree. You can disagree. I don't know, you can just ask to do some random things. And also what we built in the in the later stages is if support request is pretty much simple and there is no much of improvements that needs to be done, we implemented new commands that called self review done.
19:42 So we as developers take responsibility for the work we did. And we said, yeah, we applied all the changes that was proposed. And then PR agent is actually approving your pull requests, and it's actually up to you to decide if you would like to add project to your code owners. So if if you edit it there and it approves your pull request, you can actually merge your code without like human interaction.
20:11 And this is literally what happens. So this is like one of your merge requests from our like production GitLab that from the beginning till the end one developer and a bot. And it was like merge without human interaction because it's literally just I think it was pretty simple just removing some JSON file. So why developers need to spend a lot of time reviewing something that is pretty simple. So yeah, what else?
20:45 I think like few things that we build on top, and I shared them before, was how we can have to approve all the work on zero findings, how we can chat with the code base and important part when we ask questions. The context trigger again related to the question itself, and not only to the changes in the pull request.
21:06 And also we build some kind of like stability thing inside of the inside of the bot, just to make sure we're not missing all the commands that are going from merge requests. Because sometimes, let's say in the same time there was like 510 merge requests created and we don't want to miss any request.
Building a feedback loop with Grafana
21:27 So why I mentioned that feedback loop is really important and it's important to build it from day one. So first of all we are trying to understand if it's even like useful if it's even like given some value. And in one day as a part of my routine, I was like looking at the graphs and saw that actually our satisfaction rate is going down.
21:52 And at first thing of course, I was went to the code base trying to understand what was shipped wrong. Is there like a production incident or what's going on? And finally enough, it was already like after we were working in vertex, AI and Gemini, they updated the model. We were using the latest version and the model they shipped was actually given worse results.
22:18 And then when we understand that this is the issue, we quickly rolled back to the previous model. And like satisfaction rate goes up, right. So as a result, after all this like adventure, I would say
Results: 300 reviews/week at MVP scale
22:34 what we had in the end in like a few months is every week we have around like 300 reviews, comments, even at MVP scale. We have people from different teams coming to us and saying, can we get it? Like how much it will cost your time to to do the same for us and cool stuff. It's like maybe our maybe two.
22:59 And of course, it was just a pleasant thing to do to receive a city grant for advancing.
The six-step playbook
23:05 I write so the playbook in five lines that you can, I think, do. So today is Tuesday, maybe like next week on Monday. Evaluate different tools. I think like right now AI is changing so quickly and like there is so many tools appearing every day that is really hard to keep up. But still, as a developers, we are responsible for proper investigation in the tools that we would like to use.
23:35 So please spend as much time as possible in order to choose the right tool in the beginning. Second point I think is like on your own prompts, because if you don't know how it works under the hood, you you don't know what to expect. Builds a feedback loop from day one. It's pretty much what I mentioned before.
23:59 We need to understand if it was useful, if our developer is happy, and if Google again silently update is a model in for CI/CD, not in the prompt. Still, don't forget that CI/CD is important part of our journey. And this is where most of the hours like trust should go. And. Important part that we choose is like road by complexity, fullback by failure is two struggle model.
24:31 First, just start with like a pro model or like Opus model. And then if it's for some reason, just don't give in the good result. Just like fall back to the something like that, this next one in the line. So if we're talking about Gemini it was like pro and then flash right? Thank you.
Q&A
24:49 Thank you everyone. I think we have like a few minutes for live Q&A. But also feel free to catch me after the talk. Yeah. Thank you. Any questions? Yeah. Sorry. Thanks for the talk. I was curious if you could tell us a little bit more about why you chose the, the, the kind of nine step pipeline that you did for retrieval, augmented generation instead of a vector database instead of something that.
25:34 Yeah, I think this was like one of the things that was like just screaming at us like make, make vector databases. But we decided to go with an easy way because vector databases will include additional infrastructure, additional complexity. And at that point of time, we just don't want it to to to have all this conditional complexity. And we kind of see that it was pretty much working.
26:02 And, yeah, we I'm no longer working as a company where we done this stuff. So maybe like stuff is changing right now, but at the moment when I was like doing all these things, I was like, no, it's just too complicated. Thank you. Thanks. Yeah, I have two questions. One would be the PR size, right? Yeah. To set some limits or. Yeah.
26:27 So it's pretty obvious if you will have like pull requests with like 100 files and 200,000 of changes, it will not work. Well, this is the reason why actual development roles in the company was playing well because we had an internal agreement and kind of like hard rules that all the merge requests should be small, right? So something that you can ship in one day.
26:49 So most of the times it was like a few files up to a few hundred lines. So it was like pretty fine. But of course sometimes when you need to just refactor huge thing with like just renaming, it was not working really well. Yeah. So there was a limit of 150,000 tokens per person review. So if it was going more, the boat will just send the message, hey, it's too much.
27:19 I don't want to to handle it. So it's either split or just make sure that as a developer review your pull request okay. Makes sense. And the second question did you look into like trunk based development sorry trunk based development where you don't do burrs basically. It's like option around that or. Yeah, I, we don't invest it too much time into this.
27:44 It was like just usual flow how you like just no create pull requests. Just run through all of this stuff. And I know that right now that company is trying to implement more like spec-driven development in a way that like, all right, we just run the specs and we delegate all this like review to the boats and then see like where the developers actually needed.
28:07 But yeah okay. No worries. Thank you. Anyone else? Yeah. Silly question. I'm quite loud. Is that okay? Yeah, it's actually okay. I lose this. It's fine. It is silly because it's a really good talk. 87% of subtasks had no description what was in them. So I don't know why, but a lot of people was creating stories and creating like sub tickets in every like.
28:39 Yeah, like this is something because I mentioned like we are trying to get like change done in one day. So this is the reason why, let's say story can be like describing all of the stuff that needs to be done. And then sub tickets can be just like implement this. Yeah, just title because it it will kind of make sense.
28:56 But then we just like yeah, if you just pull this information from empty description like will not give you any, any value. So let's try to go level up and then maybe like if, if the story is empty, which usually I seem not not the case, then we just like making sure that we pull at least something to to validate.
29:15 If the merge request align what with what needs to be done. Yeah. I think I think it's it's quite interesting going through all of this and like having to explain it to a robot or let, let's say, you know, and you realize just how bad our processes are dealing with humans and how much we infer and just make stuff up and go wrong.
29:40 And when we're having to go with all of this, just go, oh, this is not very good. Do you find the same? Yeah. Yeah. I, I think this is something that was need to be changed when we started to adopt it. More and more is like the good practices, the like, all these processes in place is like showing the people how good looks like and what we do expect to get better results.
30:02 Unfortunately, I was not part of the company when there was a conversation about like evaluating this, how good the skills are, how good the rules are. But I think it's like pretty interesting topic to think about. And yeah, I really like, the talk from Simon about it, and it's actually just forced you to think how good your rules are, how good your skills are, how you can make sure that all this information is actually useful, and not just creating a more noise and bloat and just like, trying to contradict with each other.
30:41 Yeah. I'm not gonna throw. So when you're doing the reviews, what were you asking the agent to review and kind of saying review my code, tell me if it's good or not. I'm assuming there was probably a set of, like, specific things you wanted to get from that. Yeah. So if you will take a look at the PR agent afterwards, this is like open source software.
31:07 So you will see exactly what the final prompt is. So it's like predefined prompt like you are like AI reviewer, you're doing this this and this and that. So what we did differently is like we injected our personal instructions and our personal rules to align with what is expected from the company because like, the company is like just doing some legal stuff.
31:30 So we need to make sure like all these like requirements are matching, there is no security concerns. And like different I, I can go back to, maybe, the slide. Yeah. So this is what most of the things that was covered by, by all this like political stereo. I would like to mention that you, you should not, like, rely on it too much.
32:00 If we're talking about, like, really complicated work in the pull requests. Right. We still need to have all these, like senior developers with the knowledge of like how what, what good looks like, how it should work, what is the best solution. So I look at this as more like a helper to avoid spending time on this, right?
32:20 I don't want to write down, hey, it's missing tests. It should be obvious and like it's pretty simple for the even. Like the simple like just usual them to say, oh yes, there is a function, but there is no test, so just write it down. Do you think it's do you think it's possible for an agent to recognize that it's there is something that it doesn't know how to do, in the sense that if it's something more complicated, do you think you'd be possible to get an agent to say, I think there's something more complicated here, but I don't know what.
32:52 So. Yeah. I mean, like, we can inject more additional like, again, rules or like specific things saying like, if you're not sure, just let us know. But also part of the functionality of this agent is defining how complicated the request and scope of like, how much effort you will need to spend on the review. And in the end, after the review, it's just like add the label.
33:18 So like the, the pull request is like one out of five in complexity. So should be pretty much simple. But if you see it's a even a like as a 4 or 5 out of five, then it means there is like some additional complexity you need to be careful about. Makes sense. Thanks. All right. Cool. Thank you very much.