LLMs Demystified
You're here to operate agents. That starts with a grounded model of the engine inside the agent.
You don't need to understand LLMs to use them. You need to understand them to use them well. This chapter covers just enough of how they work — and where they fail — so you can pick the right model, stay inside its sweet spot, and catch problems before they cost you.
An LLM is not magic, a coworker, or a mind. It is a token prediction system trained on large-scale patterns in text, code, images, audio, or other data depending on the model. That reductive framing is the key to understanding why LLMs are useful, why they fail, and why there is no such thing as "the best model."
The operator stance is simple:
Treat every LLM as a powerful probabilistic tool with a specific operating envelope. Pick it, route to it, and constrain it like any other production dependency.
What an LLM Actually Is
A Large Language Model is a statistical pattern engine built on transformer architecture. At inference time, it repeatedly runs one loop:
- Encode the input as tokens
- Use attention to read relationships across the current context
- Estimate likely next tokens
- Sample or select one token
- Append it to the context
- Repeat
Tokens are the units. Attention is how the model relates those units inside the current context. The output is one predicted next token, which immediately becomes part of the next context. This loop is the core mechanism behind every fluent paragraph, code patch, tool call, or reasoning trace the model emits.
That works for engineering because most software work is already token-shaped: code, diffs, logs, shell commands, tickets, docs, test failures, review comments, and tool schemas. Multimodal models extend the same mechanism by encoding screenshots, diagrams, audio, PDFs, and charts into token streams the model can condition on.
LLM mechanics
The model predicts the next token
Tokens are the units the model processes and emits — a word, subword, punctuation, image patch, audio frame, or tool-call structure depending on the modality. Rule of thumb for English: 1 token ≈ 0.75 words.
Tokens define the model's input/output budget: providers bill by token, the context window is measured in tokens, and more tokens means higher cost and latency. Output tokens are priced 4–8× higher than input tokens (morphllm), but for coding workloads the volume ratio flips the cost: feeding 50,000 tokens of context for a 200-token edit means input accounts for 70–85% of the bill.
Sampling controls how the next token is selected. Temperature 0 (greedy) always picks the highest-probability token — deterministic and reproducible, which is the default for agent tool calls and evals. Temperature > 0 samples from the distribution, introducing variance useful for drafting and brainstorming. Top-p and top-k further restrict the candidate set.
Attention is not memory, search, awareness, or focus. It is relevance-weighted context mixing inside the model. More context gives the model more tokens to mix, but it does not guarantee the right tokens receive enough weight. The architecture section below explains why that matters for cost, latency, and long-context reliability.
The model has no consciousness, intent, or feelings. It does not "want" to solve your task. It produces continuations that are likely under its training and instruction context.
That does not make it useless. It makes it a tool.
What Training Actually Does
A useful mental model: the architecture is the program shape; the weights are the learned contents of that program.
The architecture defines the fixed computation: layers, attention, token flow, and the mathematical operations that turn input tokens into output probabilities. The weights are billions of numeric parameters inside that computation. During inference, those weights are fixed. During training, they are adjusted.
Training is optimization. The training system shows the model many examples of input context and desired output, measures how wrong the model's prediction was, then nudges the weights so similar contexts produce better probabilities next time.
For a language model, the basic pretraining task is next-token prediction:
Context: "The capital of France is"
Desired next token: " Paris"
If the model assigns low probability to " Paris", the loss is high. If it assigns high probability, the loss is low. Across billions or trillions of examples, training searches for weight values that reduce average loss.
That is the sense in which the model "learns." It does not write facts into named database rows. It tunes a very large mathematical function until its outputs approximate the patterns in the training data.
Training searches through a high-dimensional space where every point is one possible setting of the model's weights. The loss function defines the terrain: lower is better, but the terrain is not a smooth bowl with one obvious bottom.
Modern training algorithms such as stochastic gradient descent and Adam estimate which direction should reduce loss, then take small steps. Because the space is huge and non-convex, training is not guaranteed to find the best weights—or even a useful stable model. Random initialization, batch order, optimizer settings, data mix, and compute details all affect the path.
That stochasticity is part of why model training feels less like compiling a program and more like fitting a massive probabilistic system until it behaves well enough under evaluation.
Post-training then narrows that broad continuation engine into a product behavior profile: follow instructions, prefer useful answers, refuse some requests, call tools, solve checked tasks, or match a domain style.
What Gets Encoded Into the Model
After training, the weights become a compressed map of patterns. They encode statistical structure learned from training data: language, facts, code idioms, document formats, interaction norms, visual patterns, domain terminology, and common ways people respond to situations.
That includes two useful kinds of structure:
- World and domain knowledge: what tends to be true, how concepts relate, what APIs look like, how legal clauses are written, what a support ticket usually contains
- Action patterns: what to do next in a scenario, how to answer a question, how to revise code, when to ask for clarification, how to format a tool call
This knowledge is not stored like rows in a database. It is distributed across parameters.
LLM mechanics
Training compresses examples into model behavior
That is why the model can generalize beyond exact examples, but also why it can be stale, incomplete, biased, or confidently wrong.
A useful mental model: the model has learned patterns for how situations usually continue. Inference gives it a new situation and asks for the next likely continuation.
Base Models Learn Patterns; Post-Training Shapes Behavior
Modern LLM products are not just pretrained models exposed through an API. Vendors usually start with a base model, then post-train it toward a product goal.
| Stage | What it contributes |
|---|---|
| Pretraining | Broad knowledge, language/code/multimodal patterns, raw continuation ability |
| Supervised fine-tuning | Instruction following, task formats, assistant behavior |
| Preference training | Helpfulness, tone, safety behavior, refusal policy, response quality |
| Verifiable reward training | Stronger math, code, retrieval, or tool behavior where outputs can be checked |
| Domain tuning | Specialized patterns for coding, legal, medical, finance, support, or enterprise workflows |
Read the stages as a manufacturing pipeline. Each stage narrows the raw continuation engine into a product behavior profile: what it follows, prefers, refuses, verifies, and specializes for. For agents, this is the difference between a model that can continue code-shaped text and a product model shaped to produce patches, explanations, safe refusals, or tool calls.
LLM mechanics
Post-training shapes product behavior
Reinforcement-style post-training is one way manufacturers shape behavior. InstructGPT popularized the supervised fine-tuning + reward model + reinforcement learning pipeline for instruction following. Later methods such as DPO simplify preference optimization. Verifiable rewards go further for tasks like code, math, retrieval, and tool use because the output can be checked.
For code, the objective may include tests or compiler feedback. For tool use, it may include correct function calls. For regulated domains, it may include compliance, citations, or safe refusal behavior.
That is the product reality behind many model claims: vendors are not selling generic intelligence. They are selling differently trained behavior profiles.
Probability Is Not Logic
LLMs are governed by learned probabilities, not by logical rules.
A logical system derives conclusions from explicit premises. If the premises are true and the rules are valid, the conclusion follows. Correctness is part of the machinery.
An LLM has different machinery. It does not store a set of claims, prove new claims from them, and check whether the result is true. It has learned which tokens, structures, and responses tend to follow other tokens, structures, and responses. At inference time, it generates the continuation that fits the context according to those learned probabilities.
LLM mechanics
Probability imitates logic; it does not become logic
That is why LLM output can read like reasoning without being governed by reasoning. The model can produce a proof-shaped explanation, a confident technical claim, a realistic API call, or a clean architecture argument because those forms are common in the data it learned from. The form can be logical even when the process is not.
Correct answers happen when the probable continuation also matches reality. Wrong answers happen when a plausible continuation diverges from reality. The model itself does not know the difference.
This is the fundamental limitation: probability can imitate correctness, but it does not guarantee correctness.
So an LLM response should be treated as a candidate artifact, not as a verified answer. First judge whether it has the right shape. Then validate whether it is actually correct: run the tests, check the citation, inspect the API, verify the constraint, reproduce the result.
Human Errors Have Reasons; LLM Errors Don't
When a human makes a mistake, there is always a reason: they don't understand the domain, they're tired, they skimmed the input, their agenda pulls a different direction, or a cognitive bias distorts their judgment. Every human error has a causal story. Those reasons create predictable error funnels — you know where to look. A junior engineer struggles with distributed systems. A tired reviewer misses subtle bugs. Competence boundaries are learnable, and trust builds along them.
An LLM has no such reasons. The model that just produced a flawless architecture analysis can, in the next sentence, add 2+2 and get 5. There is no "why" — no misunderstanding, no distraction, no fatigue, no misaligned incentive. The model took a statistically plausible wrong turn in a probability landscape. That is the entire explanation.
Error structure
Human mistakes reveal a cause; LLM mistakes reveal a probability shift
Research confirms this structural difference. LLM errors superficially resemble human errors — models pick the same wrong answers students do1 — but the mechanism is fundamentally different. Human cognitive biases are stable and rooted in evolved heuristics; LLM biases are learned from training data. Scaling and instruction tuning can shift the error distribution — reducing some error types while making others more systematic — but they cannot eliminate errors. The model is a probabilistic local minimum in a high-dimensional weight space, not a reasoning system. Error is mathematically irreducible: hallucination is provably unavoidable,2 and even temperature zero does not guarantee deterministic output.3
| Human mistakes | LLM mistakes | |
|---|---|---|
| Cause | Always a reason (fatigue, knowledge gap, bias, agenda) | No reason — statistical fluctuation |
| Distribution | Concentrated in predictable funnels (expertise gaps, cognitive biases, fatigue) | Driven by step count (p^n), training coverage, and architecture fit. |
| Difficulty correlation | Error rate rises with cognitive difficulty | Error rate depends on step depth, training coverage, and architecture fit — not cognitive difficulty.4 |
| Predictability | Competence boundaries are learnable | Predictable only along the three axes: more steps = less reliable, sparse patterns = less reliable, poor fit = less reliable. Instance-level failure remains unpredictable. |
| Self-awareness | Can catch and correct own errors | Cannot introspect, cannot learn — stateless and fixed. Each error is a fresh event. |
Letter counting and arithmetic are common "LLM tests" but they test the wrong thing. LLMs process tokens, not characters or exact numbers. To count letters correctly, the model must write code, extract the exact input from your prompt, execute it, and reproduce the output — a multi-step pipeline where each step can introduce error. Same for math: the model needs to generate and run code.
These tests measure tool-use orchestration reliability, not understanding. A model that fails at counting the number of r's in "strawberry" can still reason correctly about architecture trade-offs.
The operational implication: you cannot trust an LLM based on human notions of task difficulty. A brilliant design analysis does not make the next token — a trivial constraint check — safe. Reliability follows probability depth: each step multiplies the space, so longer chains decay exponentially regardless of how "easy" each step looks.5 Validate at phase boundaries, where unverified state would propagate.
This also explains why more context does not remove the problem. More context gives the model more material to condition on, but it does not turn probability into logic. The model can still weight the wrong evidence, miss a constraint, or continue from a locally plausible pattern instead of the globally correct one.
Local Prediction, Global Consequences
The previous framing — probability imitates logic but does not guarantee it — captures the philosophical gap. But there is a sharper operational consequence: each token is chosen as the best local continuation, but the overall response needs global coherence. These two objectives can conflict.
LLM mechanics
Local predictions compound into global outcomes
A wrong token early in a response does not stay local. It becomes part of the context for every prediction that follows, shifting what looks probable next. The model was trained on clean human-written text, so when its own output goes off track, recovery is unreliable. Post-training via RLHF partially mitigates this by training on flawed outputs, not just clean text — but prompting the model to "check your work" does not reliably fix errors. Recovery needs external feedback: tool calls, code execution, tests.
One visible side effect is probability wells — a black hole in the model's probability space. Once the model crosses the threshold — repeating a token, doubling down on a reasoning direction, committing to a wrong fact — the self-reinforcement dynamic takes over. Each token deepens the well by making the same path more probable. The most extreme case is repetition loops: a pattern like ab ab ab becomes inescapable because the context now predicts ab with near-certainty. Under greedy decoding (temperature 0), escape is mathematically impossible once the loop forms — the expected escape time is infinite.6 The only reliable way out is external: a context reset that collapses the well by removing the self-reinforcing context. The rare non-deterministic floating-point flips in the inference stack can break a near-tie before the well sets, but once locked in, the model stays trapped until max_tokens or a reset intervenes.
This is the fingerprint of a system with frozen weights — a local minimum found during training, locked at inference. The probability landscape is fixed for the session. Every well that exists in that landscape is permanent. The model cannot adapt its way out, cannot learn mid-conversation, cannot dynamically reshape the terrain. It can only continue generating through the static landscape it was given. Context resets work because they change the input, not the weights — collapsing the well by removing the self-reinforcing context.
Production systems reduce probability well traps with lookahead techniques (Lookahead Decoding, n-gram speculation). Instead of generating one token at a time, the model drafts multiple future tokens in parallel using n-gram caches and mask-predict, then verifies them in a single pass. By considering several future positions at once, lookahead can "see past" an incipient well and accept a token that breaks the cycle — like jumping over a pothole instead of stepping into it.
Lookahead can cut well frequency 3–4× on repetitive workloads without changing the output distribution, but because the landscape is structurally fixed (frozen weights), it cannot eliminate wells entirely. A model already locked in a deep well still needs a context reset.
Context Under Pressure
More context is not automatically better context. For model selection, the context-window number is only an input capacity limit. It does not mean the model becomes more precise as you add tokens; often retrieval gets worse as the prompt grows.
That distinction matters because model marketing often collapses three different capabilities into one number:
- Advertised context is how many tokens the model accepts.
- Effective context is how much of that input the model can still use reliably under load.
- Point drop is absolute percentage-point loss. A fall from 80% to 40% is −40 percentage points, even though relative accuracy was cut in half.
The simplest signal to watch is retrieval decay: as the prompt grows from moderate context toward the advertised maximum, can the model still recover facts buried inside it? A 1M-token window can be real capacity while still producing worse retrieval near 1M than at 128K or 256K.
A model can accept a million tokens and still become less reliable at finding the relevant fact as the prompt approaches that size. Bigger windows expand what you can send; they do not guarantee better retrieval or more precise answers.
For model selection, compare degradation curves, not just maximum window size. Chapter 4: Four-Phase Workflow covers retrieval and RAG as system levers. Chapter 5: Context Engineering covers the attention-shape problem and how to exploit it operationally.
Long-Context Benchmarks Show the Tradeoff
Long-context benchmarks are not coding benchmarks, but they isolate the model-selection question: as context rises, does retrieval stay precise, degrade gradually, or collapse?
MRCR means Multi-Round Co-reference Resolution. The model gets a long prompt with multiple target facts buried inside distractor text, then answers questions that refer to those facts indirectly. The 8-needle variant plants eight facts, so the model has to keep several references straight instead of matching one obvious phrase.
Current long-context results show the pattern: models cluster more tightly at moderate context, then diverge near the advertised maximum. Some products have full published curves; others only publish sparse points.
The benchmark figure plots vendor-reported benchmark points; the table below lists the exact values, mode, evidence density, and source for each selected product.
LLM mechanics
Effective context is smaller than advertised context
The chart plots reported benchmark points from 8K to 1M where available. GPT-5.5, GPT-5.4, and both DeepSeek V4 rows have full MRCR curves; Claude and Gemini rows use sparse published points.
- GPT-5.574% @1M
- Claude Opus 4.678.3% @1M
- Claude Sonnet 4.665.1% @1M
- DeepSeek V4 Pro59% @1M
- GPT-5.436.6% @1M
Native 1M candidates
Validate or chunk near 1M
| Model | Benchmark | Mode | Evidence | Plotted points |
|---|---|---|---|---|
| GPT-5.5Source ↗ | MRCR v2 8-needle | xhigh reasoning effort | Full curve | 4–8K: 98.1%; 8–16K: 93%; 16–32K: 96.5%; 32–64K: 90%; 64–128K: 83.1%; 128–256K: 87.5%; 256–512K: 81.5%; 512K–1M: 74% |
| Claude Opus 4.6Source ↗ | MRCR v2 8-needle | 64K extended thinking | Sparse points | 256K: 91.9%; 1M: 78.3% |
| Claude Sonnet 4.6Source ↗ | MRCR v2 8-needle | 64K extended thinking | Sparse points | 256K: 90.6%; 1M: 65.1% |
| DeepSeek V4 ProSource ↗ | MRCR v2 8-needle | Max | Full curve | 8K: 90%; 16K: 85%; 32K: 94%; 64K: 90%; 128K: 92%; 256K: 82%; 512K: 66%; 1M: 59% |
| GPT-5.4Source ↗ | MRCR v2 8-needle | xhigh reasoning effort | Full curve | 4–8K: 97.3%; 8–16K: 91.4%; 16–32K: 97.2%; 32–64K: 90.5%; 64–128K: 86%; 128–256K: 79.3%; 256–512K: 57.5%; 512K–1M: 36.6% |
The figure tells three stories:
- Same window does not mean same reliability. A 1M-token model can retain usable facts or collapse near the end of its advertised range.
- Moderate context is not the main problem. Many models look strong before the window is saturated.
- The degradation curve matters more than the headline number. If your workload routinely reaches 500K–1M tokens, benchmark shape matters more than maximum capacity.
The verified rows make that distinction immediate. Claude Opus 4.6 reports 76% on the 1M-token MRCR v2 8-needle benchmark, while Claude Sonnet 4.6 reports 65.1% at 1M in the published comparison. GPT-5.4 falls to 36.6% in the 512K–1M bucket, and DeepSeek V4 Pro reaches 59% at 1M. These results come from different providers, modes, and reporting formats; they are workload evidence, not an overall ranking.
Every model has an effective context limit. Some decline gradually, some fall off a cliff, and some recover only with expensive reasoning modes. None make context management obsolete.
Newer Models Can Improve One Dimension and Regress Another
Model releases move along a Pareto frontier. A vendor can improve coding, tool use, vision, latency, safety, or instruction following while making long-context retrieval worse.
Claude Opus 4.6 is a useful example: Anthropic reported 76% on the 8-needle 1M MRCR v2 benchmark, a strong long-context retrieval result for an Opus-class model. Opus 4.7 then improved across agentic coding, visual acuity, tool use, self-verification, instruction following, and long-running task behavior, according to Anthropic's Opus 4.7 release notes. But the long-context retrieval line regressed sharply in the benchmark data above.
That is not a contradiction. It is the point: "better model" is not a scalar property. A model can become better for autonomous coding while becoming worse at needle-in-a-haystack retrieval. Another can improve long-context recall by spending more reasoning tokens, raising cost and latency. A smaller model can be preferable when verification is strong and the loop needs speed.
Use benchmarks to identify which dimension improved, which dimension paid the cost, and whether your workload depends on the dimension that moved. This chapter stops at the model-selection question; Chapter 6 handles the operational attention shape and context-engineering tactics.
Model Cards Are Spec Sheets
A publisher document is useful only when it changes a workload decision. Consider an agent that turns one day of email into a prioritized todo list: it must recover commitments, normalize dates, merge duplicates, rank work, and emit a valid structure. Some emails carry inline images, attachments, or screenshots — those are workload requirements the model's modality field either covers or doesn't. A full inbox pushes the context window — recall benchmarks test whether commitments stay recoverable as the prompt grows. The output must follow a strict schema, deduplicate, rank, and filter spam — instruction-following benchmarks approximate that constraint density.
A modern model card publishes dozens of benchmarks across coding, math, reasoning, vision, safety, and domain-specific tasks. Most are irrelevant to any specific workload. The skill is picking the few fields that map to your actual constraints — price, modality, context-vs-recall, instruction following — and reading across them.
A model card is a spec sheet, not a scorecard. Each field reports a single published measurement under specific conditions — price at listed rates, recall at a given context length, instruction-following on one benchmark. Strong in one field does not predict strong in another. A field without a published result for your workload dimension is simply unknown.
Read across the fields on a card to build a model profile. Then compare profiles across models. The result is a candidate set, not a single winner.
Model selection
A model card is a profile, not a rank
Anthropic
Claude Opus 4.6
Price
$5 input / $25 output per MTok
A high-cost long-context quality envelope.
Modality
Image + text input; text output
Matches the stated email workload without overclaiming attachment support.
Context vs. recall
76% on 1M-token MRCR v2 8-needle
Strongest directly relevant published retrieval signal in this set.
Instruction following
LiveBench Instruction Following: 63.3%
The score is comparable across the cards and provides a broad instruction-following signal.
LiveBench measures compliance with multiple explicit constraints in language tasks. Its score combines strict all-constraints accuracy with partial per-constraint accuracy from automated checks. The reported July 2026 scores are a broad instruction-following signal, not a direct evaluation of email extraction, date normalization, deduplication, ranking, or schema validity. View the methodology and leaderboard.
When comparing cards, "context vs. recall" points back to the retrieval graph above — capacity and recall are separate published signals. The cross-model comparison produces a candidate shortlist, not a production route. Run the labeled inbox set before routing real work, and score commitment recall, date normalization, duplicate handling, ranking, schema validity, latency, and cost.
A model card describes a released checkpoint. A system card describes a provider's assessed safety scope. A release report describes selected published results. Each is release-specific; none replaces a local evaluation of the agent and serving environment.
Marketing Translator
LLM marketing often starts with a familiar launch claim and hides the operational behavior underneath. Translate the claim before evaluating the system.
Marketing Translator
Translate launch claims into operational reality.
“The most capable model we’ve ever trained.”
Larger model: slower inference and higher token cost.
These translations are not cynicism. They are operational clarity. Once you translate the claim, you can evaluate the system like an engineer.
Do not ask, "Is this model smart?" Ask, "Under what workload, constraints, cost profile, and failure tolerance does this model perform well?"
There Is No Best LLM
There is no best LLM in isolation. There is only a model that fits a workload, a constraint set, and a verification strategy.
A model can be stronger at coding and weaker at long-context recall. It can be faster because it spends less compute. It can look better on a benchmark and still be wrong for your system because the benchmark measured the wrong bottleneck. Treat every model claim as a narrow engineering claim: what improved, under what setup, at what cost, and does that matter for this task?
When an output fails, diagnose before switching models. If the model never had the right information, the fix is grounding: retrieval, search, file access, smaller context, or better source selection. If the model had the information and used it badly, the fix is inference support: a stronger model, a narrower task, examples, tests, evals, or review.
That is this chapter's operating model. LLMs are probabilistic engines, not coworkers. They produce candidate work from the context you give them. Production reliability comes from the system around the model: grounding facts, managing context, constraining and routing work, validation, and verifying results before accepting them.
The useful abstraction is:
A probabilistic engine inside a deterministic control plane.
The next chapter covers that control plane: how agents wrap the LLM brain with software that can plan, use tools, observe results, and continue the loop.
Next: Chapter 2: Agents Demystified