Choosing a Production Model
Model Cards Are Spec Sheets
A publisher document is useful only when it changes a workload decision. Consider an agent that turns one day of email into a prioritized todo list: it must recover commitments, normalize dates, merge duplicates, rank work, and emit a valid structure. Some emails carry inline images, attachments, or screenshots — those are workload requirements the model's modality field either covers or doesn't. A full inbox pushes the context window — recall benchmarks test whether commitments stay recoverable as the prompt grows. The output must follow a strict schema, deduplicate, rank, and filter spam — instruction-following benchmarks approximate that constraint density.
A modern model card publishes dozens of benchmarks across coding, math, reasoning, vision, safety, and domain-specific tasks. Most are irrelevant to any specific workload. The skill is picking the few fields that map to your actual constraints — price, modality, context-vs-recall, instruction following — and reading across them.
A model card is a spec sheet, not a scorecard. Each field reports a single published measurement under specific conditions — price at listed rates, recall at a given context length, instruction-following on one benchmark. Strong in one field does not predict strong in another. A field without a published result for your workload dimension is simply unknown.
Read across the fields on a card to build a model profile. Then compare profiles across models. The result is a candidate set, not a single winner.
Model selection
A model card is a profile, not a rank
Anthropic
Claude Opus 4.6
Price
$5 input / $25 output per MTok
A high-cost long-context quality envelope.
Modality
Image + text input; text output
Matches the stated email workload without overclaiming attachment support.
Context vs. recall
Vendor-published: 91.9% @256K, 78.3% @1M · Measured 68.1% at 128K, 54.3% at 512K; no 1M runs
A strong measured curve to its tested endpoint; the published 1M figure has no independent check.
Instruction following
LiveBench Instruction Following: 63.3%
The score is comparable across the cards and provides a broad instruction-following signal.
LiveBench measures compliance with multiple explicit constraints in language tasks. Its score combines strict all-constraints accuracy with partial per-constraint accuracy from automated checks. The reported July 2026 scores are a broad instruction-following signal, not a direct evaluation of email extraction, date normalization, deduplication, ranking, or schema validity. View the methodology and leaderboard.
When comparing cards, "context vs. recall" points back to the retrieval graph — capacity and recall are separate published signals. The cross-model comparison produces a candidate shortlist, not a production route. Run the labeled inbox set before routing real work, and score commitment recall, date normalization, duplicate handling, ranking, schema validity, latency, and cost.
A model card describes a released checkpoint. A system card describes a provider's assessed safety scope. A release report describes selected published results. Each is release-specific; none replaces a local evaluation of the agent and serving environment.
Marketing Translator
LLM marketing often starts with a familiar launch claim and hides the operational behavior underneath. Translate the claim before evaluating the system.
Marketing Translator
Translate launch claims into operational reality.
“The most capable model we’ve ever trained.”
Larger model: slower inference and higher token cost.
These translations are not cynicism. They are operational clarity. Once you translate the claim, you can evaluate the system like an engineer.
Do not ask, "Is this model smart?" Ask, "Under what workload, constraints, cost profile, and failure tolerance does this model perform well?"
There Is No Best LLM
There is no best LLM in isolation. There is only a model that fits a workload, a constraint set, and a verification strategy.
A model can be stronger at coding and weaker at long-context recall. It can be faster because it spends less compute. It can look better on a benchmark and still be wrong for your system because the benchmark measured the wrong bottleneck. Treat every model claim as a narrow engineering claim: what improved, under what setup, at what cost, and does that matter for this task?
When an output fails, diagnose before switching models. If the model never had the right information, the fix is grounding: retrieval, search, file access, smaller context, or better source selection. If the model had the information and used it badly, the fix is inference support: a stronger model, a narrower task, examples, tests, evals, or review.
That is the operating model these chapters build on. LLMs are probabilistic engines, not coworkers. They produce candidate work from the context you give them. Production reliability comes from the system around the model: grounding facts, managing context, constraining and routing work, validation, and verifying results before accepting them.
The useful abstraction is:
A probabilistic engine inside a deterministic control plane.
The next chapter covers that control plane: how agents wrap the LLM brain with software that can plan, use tools, observe results, and continue the loop.
Next: Agents Demystified