How to evaluate smaller language models for real workloads

Anthropic announced Claude Haiku 5.5 on 7 October as a fast, lower-cost model for high-volume work. The release is a timely reminder that model selection is becoming an application architecture decision: a system can route small, repeatable tasks differently from difficult work that needs deeper reasoning.

Anthropic lists Haiku 5.5 input pricing at $0.10 per million tokens for prompts up to 100,000 tokens and $0.50 above that threshold. Its own comparison says this is 90% lower than Haiku 4.5 for prompts up to 100,000 tokens and 50% lower above it. Those numbers come from the vendor; they are not a guarantee that a real workflow will cost 90% less, since models may use different token counts and retries.

Anthropic also says larger models remain better suited to complex agentic coding tasks, while the smaller model fits narrower tasks such as summaries, compaction and subagent work. That distinction is more useful than a single benchmark leaderboard: it tells teams to evaluate each kind of work separately.

Choose a task before choosing a model

Make a short list of the repeated decisions in your application. Examples include classifying incoming requests, extracting fields from a known document format, summarizing a completed conversation, or selecting relevant records. Write down what a correct answer must contain and which mistakes matter most.

Build an evaluation set from representative, permissioned examples. Include ambiguous cases, missing information, long inputs and the formats your system actually receives. Have a human review a sample so that the “correct” answer is not defined only by another model.

Then compare candidate models on the same set. Track task success, unsupported claims, structured-output validity, latency, token use and cost per accepted result. A model that costs less per call can cost more per usable answer if it needs retries or human correction.

Route by confidence and consequence

For low-risk, well-scoped work, a smaller model may be enough. For uncertain or consequential decisions, send the task to a stronger model or a human review queue. The routing rule should be explicit and observable: why did a task go to this model, and what happened next?

A practical pattern is a two-stage flow. The smaller model handles the common path and returns a structured answer with a confidence or validation signal. A deterministic validator checks required fields, allowed values and source references. If the answer fails validation or the task falls outside known boundaries, escalate it instead of accepting a plausible-looking response.

Do not treat a model’s self-reported confidence as proof. Use signals you can test: schema validation, a retrieval match, a known classification threshold, or a reviewer’s decision. Keep sensitive actions—payments, access changes, security findings and messages to customers—behind appropriate authorization and review controls.

Calculate the actual trade-off

A basic cost metric is the total inference spend divided by accepted tasks, not the price of one million tokens. Include retries, tool calls, cache behavior, tokenization differences, infrastructure overhead and review time. Measure latency from the user’s perspective, including routing and validation.

Compare quality at the point where a task is useful. For a classifier, precision and recall may matter more than a general reasoning score. For extraction, measure field-level accuracy and the rate of abstentions. For coding, run the same tests and inspect whether the change is correct, maintainable and safe.

Re-run the evaluation when prompts, models, tools or input distributions change. Model releases move quickly, and a result from last month may not represent the workload you have today.

A small, evidence-led rollout

Start in shadow mode: route a copy of eligible tasks to the smaller model while keeping the existing path authoritative. Compare outputs without letting the experiment trigger external actions. If quality meets the agreed threshold, enable it for a small share, monitor failures and keep a quick rollback.

The lesson from Haiku 5.5 is not that smaller models replace larger ones. It is that lower-cost capability makes routing worth evaluating. Measure the task, set a quality floor, and choose the least costly path that reliably meets it.

Sources: Anthropic · Claude Haiku