S01E01 · Mechanics Behind Generative AI: MEGA lesson video by Theo Browne
Theo Browne explains how language models generate text one token at a time, and how system messages and tools steer their behavior. The lesson demonstrates token probabilities, ChatML, and a model using Bash to inspect a project.
videoTheo Browne · hq.mega.dev · Saved Nov 19, 2026
mega course, tokenization, llm inference, agent prompting, tool calling
Transcript of the lesson video for S01E01 of the MEGA.dev course, by Theo Browne. Agenda: Introduction · Mechanics · Limitations · Providers · Interactions · Steering. The video plays on the lesson page in MEGA Student HQ (needs a login).
Theo (MEGA.dev) Lesson 1
00:00:00,100 --> 00:03:53,560 [Speaker 0]
What's up, nerds? I'm Theo, the first instructor for the MegaDev course. In order for us to get the best out of our agents, we need to understand how the agents themselves work, and that's why we're starting with lesson one, generating text. I know this might seem pretty basic, but hear me out, because there's a lot of misconceptions that I would go as far as to say lies, that we just kind of make assumptions around and believe because it's easier to work that way. But I think understanding the details will make it easier to use these models more effectively. When you send a message to a model, it's important to understand what it gets and how it responds. It's not as simple as you think. You're used to sending a text to a person that they read, they think about, they look into some things, and then they respond. LLMs don't work that way. We make them act that way, but that's not how they work underneath. The simplest way of explaining it is that the model makes informed guesses based on the information in the chat history, and it does this chunk by chunk, not response by response. It's not generating the perfect message based on the history. It's generating the perfect next token based on prior tokens. Let's start with a really simple first message. "Sup, nerds?" This message would be read by a person word by word. LLMs don't see words, though. They see tokens. Tokens are the chunks of text that LLMs use to understand and make suggestions as to what comes next, because in the end, LLMs are effectively just autocomplete. They generate new tokens based on the previous token. For example, let's take the sentence, "The capital of the US is blank." If you hand this to a traditional completion API with an LLM, it would then generate the next tokens, which in this case would be Washington, D.C. Notice how this is breaking things up, though, with the dots, the spaces, and the letters sometimes being included, sometimes not. These are all details of how different tokenizers are implemented, and certain things work well for code, certain ones don't. A ton of iteration and innovation has happened in tokenizers over the last few years. None of that is super important to understand beyond the fact that models aren't seeing the specific paragraphs we send them. They're getting these chunks of text and then guessing what's most likely to be next based on what we've given them. Let's see how this looks in action with a simple demo with autocomplete. I'm intentionally using an old and kind of dumb model here, but thankfully can actually run on my computer to show the insides of how it works. Let's say we send a message like, "Write a JavaScript function that checks if a number is even. Only code." The expectation is that we get back code. So, let's see how the model actually does that. The first important thing to understand is what the model actually gets. It's not getting the text history. It's not getting a bunch of tokens. It's getting a specific text format that describes the state of the conversation and sets the model up to do the next thing. The format for the text here is called ChatML. There's a bunch of new formats since OpenAI, since open-sourced Harmony, which is a pretty good standard. The thing to notice here, though, is these callouts, the IM_start and IM_end. These are special tokens that the models are aware of, 'cause they're used during training, that allow different behaviors to happen at different times. Like, for a user to send a message, the system to send a message, and then for the agent or assistant to respond to these messages. So, while we sent something that looks like traditional JSON wrapped in an array to the model over the API, it doesn't get these JSON blobs. It instead gets this custom format that it's using during training. The format is pretty simple. We have the indicator of a new message start. It immediately opens with the type of message, whether it's system, user, or assistant. We then give more context, in this case, that it is Qwen, it is a helpful assistant. We then have a new message after the end, which is a user message. This indicates that a user is sending this thing. And then the most important part, we hard-code a start. We immediately say, "IM_start assistant." This is important because it's an autocomplete model. It has to generate the next token, so it wouldn't really know what to do if it just ended the user turn. So, we hard-code a start to the next turn so the model can do what it thinks makes the most sense next, which we can see when we press enter.
00:03:55,100 --> 00:10:12,148 [Speaker 0]
Now that we have run this, we can see how the model responded with these percentages. The percentages might be confusing. The simplest way I can explain it is these percentages show how confident the model is with a given pick for the next term. Here, it thinks that after our message asking it to write a JavaScript function that checks if a number's even, that the best next response is the word certainly with 66% confidence. Next, it thinks sure is a good option with 14%. Next, it thinks that three backticks could be a good option with 14% as well. This is the option I would have preferred since we told it to only send code, but it wanted to be a helpful assistant, which steered it towards the word certainly. Next, it was relatively confident, like 92% confident, that an exclamation point was the best next thing to include. Then it suggested below. This little dot is for a space. So next, it thought space below made sense. Then it thought space is made sense, yada, yada, yada. And you can see all the other things it was considering before it responded with these particular words. It then continued generating and finished its response. And at the very end, if we had let it go all the way, it would hopefully have ended with one of these end tokens, thereby signifying to whatever tool is running this that the model can stop. Remember, the model has to stop eventually. It can't just autocomplete forever. So, one of the things they're trained on is this idea of stops. They can respond with a certain phrase to trigger the end. That's not the only thing they can trigger. Remember this. We'll go back to it. One more small thing to showcase the non-deterministic nature of all this with the weird patterns that happen underneath is asking Qwen2-5 to generate a random number. It picks eight with 20% confidence, seven with 18%, nine with 16%, et cetera. So, its definition of random is not quite our definition. One more callout about the generation that it did here. You'll see that when it said console.log is even seven, it responded true. That is not a good response, 'cause as I hope you understand, seven isn't even. The problem with this style of generation is that the model can't go back and correct itself, because it's just outputting one token at a time, appending to this history. It can't go back and say, "That's wrong," and edit it. It has to just keep adding to the existing text corpus. This style of autocomplete responses allows for models to share information that they already know. So, if you ask the model, "Who was president in 1952?" it's able to generate a response based on the information it was trained on. But then if you ask a modern smart model like Opus 5.5, "What's today's date?" you're not gonna get a very good answer. It's actually going to answer hundreds of days behind. If you then ask it what the latest version of a package is, you're never going to get a correct answer, assuming it's been updated anytime recently. All of this is because this data has changed since the model was trained, or maybe it was just trained on old data. But when we use models and tools like Claude Code, we don't see all of these types of issues. The reason is because the model can get that data other ways, not through its training, not through its intelligence, but through tools. In order to understand how agents get data, we have to do something that a lot of us haven't done in a while, read some code. Jokes aside, I think it's important to understand all of this. I have some code here that allows a model response to be parsed and then run as Bash. Remember what I said earlier about the model having to indicate when to stop? One of the reasons it wants to stop isn't just to finish responding, it's to generate a tool call that allows it to get data that it doesn't know. Like, if it wants to know what the current date is, if it wants to know what files are in your code base, if it wants to know what packages are available on npm, the model has tools it can use to do that in most harnesses. And those tools have custom syntaxes the model can use in order to respond and trigger those types of behaviors and actions. In order for this to work, you need to stop generating once one of these tokens comes in. In this case, we have a simple example where stop sequence includes closing tag for Bash. What we're saying here is when we see this in the response, that means stop and we have to go do something. So the model will send something along the lines of bash ls close bash, which allows it to run this command. And our code is pretty simple here. We grab the command out of that syntax. If we don't have a command, we just respond with the text. But if we do have a command, we output the command, and then we run it with this sh helper to get the output of the command. And then importantly, we push this onto the messages. We'll talk about that in just a bit. So let's see this in action with a simple demo that I just threw together. The user sends, "What's in this directory?" And the model responds, "Oh, I can't see your file system." But it absolutely could. If it just responded with that format, it would work fine. So what's up? Why can't it see this data? The reason is actually quite simple. We didn't tell the model that it has access to this syntax. We can't just give it a tool and expect it to know how to use it. We have to tell it what the tool is and that it exists in the first place. Otherwise, it has no idea that certain types of text can trigger certain behaviors. Here, we have done just that. We added to the system prompt this tool prompt, which describes that the model's able to get data by using this format in order to run commands. So if we rerun this with this addition to the system prompt, you'll see the model will run these different commands in order to figure out what's in this project and then respond accordingly. This is a small JavaScript program called Cart, a shopping cart pricing module, yada, yada. You get the idea. It's able to do all of this now because it knows that it has this tool it can call. And the Bash tool is pretty powerful. It gives the model the ability to do a lot of different things. This also showcases the importance of the system prompt and how it steers model behavior. When the model has the ability to trigger certain behaviors or special syntaxes, it needs to know about it and know what it can do. And then once it does those things, it needs to get the results back so that it can continue operating. This all starts with what we usually call a system prompt. In this case, I'll call it system message because there can be more than one. In a simple example, I tell the model that it can use Bash to get info using the following syntax. We then follow up with a simple user message that says, "What files are in this project?" So when it is spun up to respond, it can take advantage of all of this data, which steers it in a certain direction. One of the things the model can do is think, at least if it's a modern reasoning model. So it will think about what it has access to, what is being asked for. And remember earlier when I said the model can't correct itself? It can't if it says something wrong. But if it's talking to itself in the thinking traces and it gets something wrong, it can correct itself and just say, "No, that's not right. I should go this other way." So by the time it gets to the output stage, it's much more likely to be correct. So after it thinks for a bit, it says, "Good question. Let me check what files are here." And then it runs this Bash command. But then I have this line drawn. I have this line drawn for a very important reason. We're done.
00:10:13,188 --> 00:17:42,956 [Speaker 0]
After this response comes through and we hit our end token, which is this close bash, that is the end of the request. The API request that was made is over. The reason is because something has to happen before it can continue. We asked for this command to be run, so now we have to go run said command. Usually what this means is whatever's running this assistant, probably like a Claude Code type tool, will fire off the command, get a response, and then send that response back as an append to the history. Every time an API call is made to the Claude Code servers, Codex, whatever else you're using, it usually will include the entire chat history because the model needs that in order to generate the next tokens. And when a tool call is made, the important thing to understand is that the response is run on whatever computer the harness is running on, and that response is appended to the history so the model can continue generating accordingly. The same way that you send a question to the model, the tool call sends a response to the model, which allows it to continue after. And the model then decides when to stop, whether it's stopping for a tool to be run or it's stopping because it's done doing whatever it has to do and it wants your response. The tool-- or the model chooses when to stop with one of these indicators, and then you act accordingly. Let's talk a bit more about this part quick, the system message. The example I have here is relatively small, but this is not how things will work in the real world. I have another quick demo to show what I mean. I'm going to simply say hi to a handful of different tools, harnesses, whatever you want to call them, to see how they respond, and specifically how many tokens are being used. If I send a simple say hi to Claude, it's not gonna use the three tokens that say hi would be represented by. It's gonna use a hell of a lot more, thousands even. I have tool search on, which means a lot of my tools, MCPs and stuff, aren't included by default, just little bits of info that they exist, so the model knows they exist to go search for them with another tool. But it's still 21,000 tokens for what seems like a very simple request, say hi. Which it responded with, "Hi, Theo. What do you want to work on?" Oh, how does it know it's me? It knows it's me because I have a Claude MD file that describes a handful of things that I want it to know, including who I am. It also will do things like figure out what the current git status is and run a command before it sends the request to include that in the prompt so the model has all of this data before it even starts. Let's pull all this together with the next important thing to understand. Agents. What is an agent? The simplest way to put it is that an agent is some type of tool or system that allows for an LLM to run in a loop and make tool calls to get whatever information it needs, make changes, et cetera. The loop is the important part, though. It doesn't just stop here. So with our example here, it just stops after the message runs. You'd have to go make another API request for this to continue. When I say that agents are loops, I mean it quite literally. Most of the harnesses we use to run agents are literally running in a for loop or a while loop of some form. In this case, we have a simple for loop where I have a limit for turns at 40. I'll explain what that means in a sec. I also think this probably should have been the word step, not turn, but doesn't really matter here. This is the first message in the history, and as you can guess, we'll be appending new messages to this history as we go. So at the start of the loop, we call the Claude API with a request to create a message. We say what model, how many tokens we want it to do at max, the system prompt, thinking, which is us turning on thinking, saying that we want to get summaries of the results, and that it's adaptive, and then we give it tools. In this case, we have a simple bash tool that allows it to run things. We then hand it messages, which is that same array from the top of the messages. It's arguably the only really important part here. Everything else could be kind of changed. This cannot be changed. The response is what we get back from Claude. So at the end here, we push that onto the history. In this case, we push it as role assistant because this is a response from the assistant, which is the model, and the content is whatever it said. We have to decide what to do next. First, we check to see if it is, or in this case, isn't tool use, because if the model's stopping for a reason other than calling a tool, that is because it's done, because we've only given it two reasons to stop, tool calls or being done. So if it is the-- or so if it's not a tool call, that means it's done, so we early return and escape. That means from here on, it has to be a tool call. So now we can assume accordingly. We grab the calls from response content filter, where we filter based on if it is a tool call block or not. This is important because sometimes the model can respond with more than one tool call. Here we have results, which is where we collect all of the results of the things we call, because we're gonna go through and safely parse and then get responses for all the different bash requests that it made. If any of these fail, we append to results with an invalid call saying it's an error and that the content is invalid input. Otherwise, we rip the data, we run it, we get an output, and then we send this as a result. And at the end here, this is where things all come together. We push with role user the contents of those results. This means that we're taking all of the outputs from all of those tool calls and then appending them to the history like a normal message. So what the model would then see is the results of that tool call as a bunch of, like, bash outputs, and it knows to continue working accordingly. And of course, we can wrap this up with one last demo. Pnpm-agent will run the code that we were just looking at. Here we have a really simple broken demo app with some tests that are failing. So let's ask the model to fix it. The tests in this project are failing. Find out why and fix the code. And here we can see the model responding with tool calls that it can run, and then it gets the results of those. It runs another, it gets the results. It then says something, "Two bugs. Discount doesn't divide by 100." It runs another command to edit and fix that. It says, "All five tests passed now. There were two bugs in source cart.js, and I didn't touch the tests." It says what it did, and then when we run the tests after, they are correct, because the model was able to run a command here, this sed command, to actually edit the files to fix the issues. So all this agent had was access to bash, and it ran in a for loop so that every time it did something or requested a command to be run, we would run it, append that to the history, and then let the model generate the next thing. Whether that is another tool call, a explanation, some code that would be executed via a tool call, all of those things, doesn't matter. The model is able to keep going until it tells us to stop or we hit our limit, which in this case was those 40 steps. Let's quickly go through the important takeaways from this lesson. First and foremost, the models aren't generating responses to our messages. They are auto-completing text based on the existing text history. If the thing doesn't exist in the history or the model's weights, it cannot use it. And this history is helpful in steering the model towards the right things in its weights. If the model's trained properly, it can use text outputs to request certain types of data through what we call tools. This allows them to go fetch things that they might not have, to run commands, to edit things, and to do stuff other than just output text. By doing these tool calls, it's able to put things in its own history and make changes in the world around it in order to make the next thing it does more accurate. All of this allows for the model to build its own context and history, and the vast majority of the context models work with are not what we told it. It's all of the data in the system prompt, all the tool calls and things that it's exposed to, and then it builds the context itself by using these tool calls to get the information it needs to make the next things it generates more accurate. And by doing this in a loop, you end up with what we call an agent, a model that's able to use tool calls to build the context it needs and make real changes using all of the things we just discussed. [sighs] That was a lot. And now that we understand it all, we can start taking advantage of it by learning more about the context and how we can manipulate it to make the models more successful in the things that we care about. [upbeat music]