LLMs Demystified
You're here to operate agents. That starts with a grounded model of the engine inside the agent.
You don't need to understand LLMs to use them. You need to understand them to use them well. The next few chapters cover just enough of how they work — and where they fail — so you can pick the right model, stay inside its sweet spot, and catch problems before they cost you.
An LLM is not magic, a coworker, or a mind. It is a token prediction system trained on large-scale patterns in text, code, images, audio, or other data depending on the model. That reductive framing is the key to understanding why LLMs are useful, why they fail, and why there is no such thing as "the best model."
The operator stance is simple:
Treat every LLM as a powerful probabilistic tool with a specific operating envelope. Pick it, route to it, and constrain it like any other production dependency.
What an LLM Actually Is
A Large Language Model is a statistical pattern engine built on transformer architecture. At inference time, it repeatedly runs one loop:
- Encode the input as model-specific tokens
- Use attention to read relationships across the current context
- Estimate likely next tokens
- Sample or select one token
- Append it to the context
- Repeat
Here, context is everything currently available to the model: instructions, input, tool results, and output generated so far. Tokens are the units in that sequence.
Inside the model, each token becomes numbers the model can work with, along with information about its order. Attention uses the surrounding tokens to interpret each one—bank carries different information beside river than beside account.
The output is one predicted next token, which immediately becomes part of the next context. This loop is the core mechanism behind every fluent paragraph, code patch, tool call, or reasoning trace the model emits.
That works for engineering because most software work is already token-shaped: code, diffs, logs, shell commands, tickets, docs, test failures, review comments, and tool schemas. Multimodal models extend the same mechanism by encoding screenshots, diagrams, audio, PDFs, and charts into token streams the model can condition on.
LLM mechanics
The model predicts the next token
Tokens are the units the model processes and emits — a word, subword, punctuation, image patch, audio frame, or tool-call structure depending on the modality. Rule of thumb for English: 1 token ≈ 0.75 words.
Tokens define the model's input/output budget: providers bill by token, the context window is measured in tokens, and more tokens means higher cost and latency. Output tokens are priced 4–8× higher than input tokens (morphllm), but for coding workloads the volume ratio flips the cost: feeding 50,000 tokens of context for a 200-token edit means input accounts for 70–85% of the bill.
Sampling controls how the next token is selected. Temperature 0 (greedy) picks the highest-probability token and removes sampling randomness, which is useful for evals and structured tool calls. It does not guarantee correctness or identical runs across the serving stack. Temperature > 0 samples from the distribution, introducing variance useful for drafting and brainstorming. Top-p and top-k further restrict the candidate set.
Attention relates tokens inside the current context; it is not persistent memory or search. More context gives the model more material to relate, but it does not guarantee the right evidence will influence the answer. That matters for cost, latency, and long-context reliability.
The model has no consciousness, intent, or feelings. It does not "want" to solve your task. It produces continuations that are likely under its training and instruction context.
That does not make it useless. It makes it a tool.
The loop explains what the model does at inference time. Training explains where its behavior came from.