LLMs Demystified
You're here to operate agents. That starts with a grounded model of the engine inside the agent.
You don't need to understand LLMs to use them. You need to understand them to use them well. The next few chapters cover just enough of how they work — and where they fail — so you can pick the right model, stay inside its sweet spot, and catch problems before they cost you.
An LLM is not magic, a coworker, or a mind. It is a token prediction system trained on large-scale patterns in text, code, images, audio, or other data depending on the model. That reductive framing is the key to understanding why LLMs are useful, why they fail, and why there is no such thing as "the best model."
The operator stance is simple:
Treat every LLM as a powerful probabilistic tool with a specific operating envelope. Pick it, route to it, and constrain it like any other production dependency.
What an LLM Actually Is
A Large Language Model is a statistical pattern engine built on transformer architecture. When you send it a prompt, it runs one loop, over and over:
- Encode the input as model-specific tokens
- Use attention to read relationships across the current context
- Estimate likely next tokens
- Pick one of those tokens
- Append it to the context
- Repeat
Here, context is everything currently available to the model: instructions, input, tool results, and output generated so far. Tokens are the units in that sequence.
Inside the model, each token becomes numbers the model can work with, along with information about its order. Attention uses the surrounding tokens to interpret each one—bank carries different information beside river than beside account.
The output is one predicted next token, which immediately becomes part of the next context. This loop is the core mechanism behind every fluent paragraph, code patch, tool call, or chain of steps the model emits.
That works for engineering because most software work is already token-shaped: code, diffs, logs, shell commands, tickets, docs, test failures, review comments, and tool schemas. Multimodal models extend the same mechanism by encoding screenshots, diagrams, audio, PDFs, and charts into token streams the model can condition on.
LLM mechanics
The model predicts the next token
Tokens are the units the model processes and emits — a word, subword, punctuation, image patch, audio frame, or tool-call structure depending on the modality. Rule of thumb for English: 1 token ≈ 0.75 words.
Tokens define the model's input/output budget: providers bill by token, the context window is measured in tokens, and more tokens means higher cost and latency. Output tokens are priced 4–8× higher than input tokens (morphllm), but for coding workloads the volume ratio flips the cost: feeding 50,000 tokens of context for a 200-token edit means input accounts for 70–85% of the bill.
Attention relates tokens inside the current context; it is not persistent memory or retrieval. More context gives the model more material to relate, but it does not guarantee the right evidence will influence the answer. That matters for cost, latency, and long-context reliability.
What Reasoning Actually Is
Every pass through the model is the same fixed pipeline. One token buys a fixed amount of computation — a hard ceiling on how big a problem a single step can solve.
Long division on paper is the same situation. You can't hold the intermediate results in your head, so you write them down: each partial result is read back and used for the next step. Reasoning tokens are that paper. They're appended to the context and read back on the following passes, letting the model carry partial work forward and take on problems no single token could.
Same trade as on paper: more intermediate steps means more places for an error to creep in. A wrong step feeds everything built on it, and nothing in the chain checks itself.1
Why the Same Prompt Gives a Different Answer
Every pass through that loop is a draw, not a lookup. The model scores the tokens that would plausibly come next, then picks one — weighted toward the top of the list, but not always the top. A different pick sends everything after it somewhere else: same prompt, same model, two different answers that are both defensible.
You used to be able to trade predictability for variety with a temperature setting, and the frontier APIs no longer expose one — it survives only on older hosted models and your own open weights. What replaced it is a different dial: how many intermediate steps a request may spend before it answers (reasoning.effort, effort, thinking_level). It is not a cleaner version of the draw. It is a longer chain of the model's own working steps, and every extra step is another chance: more room to find the right route, and more room for one wrong turn to get baked into everything after it. Vendors sell it as quality bought with latency and tokens, and on hard problems that trade usually pays. Usually is the operative word. Accuracy peaks at an intermediate reasoning length and declines past it: the same lesson long context teaches, that past a point more is not better. And while the most common answer keeps improving with more thinking, any single run gets less predictable.2
Stochastic by Design
A stochastic system does not converge on one answer. Deterministic tools codify instances: a compiler, a regex, a linter each encode the exact case, and a new case means new code. Designing a data model, reading a stack trace, triaging a bug report, summarizing a week of notes: none of it reduces to an exact rule. There is a pattern to approximate, not a rule to follow, and two runs approximate it differently.
That is what the randomness is for. Each run draws a plausible generalization, so you can aim the model at a problem instead of enumerating cases. Pin it to one fixed output and you have asked a pattern generalizer to act like a lookup table — same answer every time, right or wrong, with no room to generalize or explore.
Attempts are not answers. A stochastic system has no invariants to check, only a distribution. Reliability comes from the guardrails around it — schemas, tests, diffs, retries, review, the control plane — not from the model. It has no intent behind any of it, which is why it is a tool and never an authority.
Everything in this chapter happens in the moment of answering. Where those probabilities come from is the next chapter: how training bakes patterns into the weights, and how post-training turns that engine into the product behind the API.