Comparing LLM APIs fairly requires more than timing a few requests or choosing the lowest listed token price. This reusable developer test plan shows how to measure latency, cost, reliability, and output quality with the same prompts, workloads, validation rules, and reporting method, so your team can make a defensible model decision and repeat the test when requirements or provider inputs change.
Overview
An LLM API benchmark is a controlled experiment. Its purpose is not to declare one model universally “best,” but to identify which option fits a particular workload. A support assistant, document extraction service, code-generation tool, and real-time chat interface can have very different requirements.
A useful benchmark separates four questions:
- How fast is the API? Measure time to first token, total response time, and latency percentiles rather than relying on one average.
- What does it cost? Calculate input and output token charges using the provider’s current billing units and your expected request mix.
- Does it work reliably? Record timeouts, rate-limit responses, server errors, malformed outputs, and retries.
- Is the result useful? Evaluate correctness, instruction following, completeness, safety requirements, and structured-output validity.
The benchmark should use the same prompt template, input data, generation settings, timeout policy, and validation code for every candidate. Otherwise, you may measure differences in your test harness instead of differences between models. For structured applications, pair the benchmark with a schema validator and explicit retry policy. The guide Reliable Structured Outputs from LLM APIs provides useful context for that part of the workflow.
How to estimate
1. Define the workload
Start with representative requests, not only simple demonstration prompts. Build a test set that reflects production categories, such as short questions, long documents, ambiguous instructions, multilingual inputs, structured extraction, and failure-prone edge cases. Label each prompt by task type so you can compare performance by category as well as overall.
For a first benchmark, a small, curated set can be more useful than a large random sample if each case has a clear expected outcome. Keep the test set versioned. If you change prompts, documents, system instructions, or grading rules, record that as a new benchmark version.
2. Run controlled repetitions
Send each request more than once when practical. A single request can be affected by network congestion, temporary provider load, connection setup, or a transient error. Randomize the order of models and prompts to reduce the chance that one candidate always runs under the same external conditions.
Use a warm-up phase if your client normally maintains connections, then report whether warm-up requests were excluded. Test with production-like concurrency rather than measuring only one sequential request. A model that performs well for one user may behave differently under a realistic request queue.
3. Record the complete request lifecycle
At minimum, log a benchmark row for every attempt. Useful fields include:
- Benchmark version, timestamp, provider, model identifier, and region or endpoint where relevant.
- Prompt category, input size, requested output limit, temperature or equivalent settings, and tool or schema configuration.
- Request start time, time to first token, time to last token, and total elapsed time.
- Reported input tokens, output tokens, cached tokens if applicable, and total tokens.
- HTTP status or provider error type, retry count, timeout status, and final success status.
- Raw response reference, validation result, evaluator score, and notes about unusual behavior.
Keep sensitive prompts and outputs out of general logs unless your data-handling controls permit them. A hash, test-case identifier, or secured object reference can connect a result to the original case without copying confidential content into a metrics system.
4. Calculate latency, cost, and success rates
Report median latency and tail latency, such as the p95 value, instead of only the mean. The median describes a typical request; p95 shows how slow the slowest five percent of measured requests were. For streaming applications, time to first token may matter more than total completion time. For batch extraction, total completion time and throughput may matter more.
Use the provider’s current price units in a transparent formula:
Estimated request cost = (input tokens ÷ billing unit × input rate) + (output tokens ÷ billing unit × output rate) + other applicable request charges.
Then estimate workload cost:
Monthly cost = requests per month × average cost per request.
Keep retries visible. You can report both successful-attempt cost and cost per completed task. The second figure includes failed attempts and retries, making it more useful for production planning.
For reliability, calculate at least two rates:
- Transport success rate: requests that receive a usable API response without an infrastructure or provider error.
- Task success rate: responses that pass application validation and meet the quality threshold.
This distinction prevents a fast API with frequent malformed outputs from appearing better than a slightly slower API that completes tasks correctly.
Inputs and assumptions
Document assumptions before running the test. A compact benchmark specification should include the following:
| Input | What to define | Why it matters |
|---|---|---|
| Request mix | Percentage of short, long, structured, and edge-case requests | Changes average tokens, latency, and cost |
| Traffic pattern | Sequential or concurrent requests, expected peak load, and test duration | Reveals queueing and rate-limit behavior |
| Generation settings | Output limit, sampling settings, stop rules, tools, and response format | Changes output length and consistency |
| Retry policy | Retryable errors, maximum attempts, delay, and timeout | Affects both completion rate and cost |
| Quality rubric | Scoring criteria, pass threshold, and evaluator process | Turns subjective impressions into comparable results |
Quality scoring can combine automated and human review. Automated checks are appropriate for JSON validity, required fields, citation presence, character limits, and exact classifications. Human review is useful for factual accuracy, relevance, tone, reasoning quality, and whether an answer actually solves the user’s task. Define the rubric before examining model results to reduce selection bias.
Do not combine all scores into one number unless the weighting reflects a real product decision. A decision table is often clearer:
| Candidate | p50 latency | p95 latency | Cost per completed task | Task success | Notes |
|---|---|---|---|---|---|
| Model A | Record result | Record result | Calculate result | Record result | Failure patterns and trade-offs |
| Model B | Record result | Record result | Calculate result | Record result | Failure patterns and trade-offs |
Keep provider performance separate from your network and retry behavior. Report raw API latency, client-observed latency, and retry-adjusted completion time where possible. This makes it easier to identify whether a regression came from the model, endpoint, SDK, network path, timeout setting, or application code.
Worked examples
Assume a hypothetical extraction service processes 10,000 requests each month. Its measured average request contains 1,200 input tokens and 300 output tokens. The candidate provider’s billing page lists separate input and output rates, but this example intentionally leaves those rates as variables because prices and billing units change.
Let I represent the input price per billing unit and O represent the output price per billing unit. If the billing unit is 1,000 tokens, the estimated cost per request is:
(1.2 × I) + (0.3 × O)
Before using that estimate, add expected retry overhead. If eight percent of attempts require one additional attempt, a simple planning estimate is:
Adjusted monthly cost = 10,000 × cost per attempt × 1.08.
This is only an assumption-based estimate. Replace the eight percent with your measured retry rate, and model multiple retry attempts if your policy allows them. Also calculate cost per completed task if some failures remain unresolved.
For latency, suppose the benchmark records a median of 1.4 seconds, a p95 of 4.8 seconds, and a p99 of 9.2 seconds. A product team promising a quick interactive experience should examine the tail values, not just the median. It may choose a faster model, reduce prompt size, stream output, or move long extraction jobs to an asynchronous queue.
For quality, suppose two candidates both return valid JSON, but one misses required fields more often. JSON validity alone is therefore insufficient. Track schema validity, field-level accuracy, and complete-task success separately. If the application requires reliable tool calls, also measure invalid arguments, unauthorized actions blocked by validation, and human-approval events. See Secure LLM Tool Calling for related control patterns.
When to recalculate
Revisit the benchmark whenever one of its inputs changes. At a minimum, review it after a provider changes pricing, model names, context limits, rate-limit rules, SDK behavior, response-format support, or billing terminology. A monthly or quarterly review is sensible for actively used systems, even when no visible change has been announced.
Run a new test when your application changes its system prompt, retrieval pipeline, tool definitions, output schema, traffic volume, or retry policy. Recalculate after a latency regression, an unexpected cost increase, a rise in malformed responses, or a change in the type of user input. Save the previous result so you can distinguish a genuine improvement from a changed test set.
To make the process practical, create a small repeatable benchmark command or CI job:
- Pin the benchmark dataset and configuration version.
- Load provider prices and billing units from a dated configuration file rather than hard-coding them in calculations.
- Run the same prompts against each candidate with identical settings.
- Store per-request logs, validation results, latency measurements, token counts, and retry data.
- Generate a table showing p50 and p95 latency, cost per attempt, cost per completed task, error rates, and quality scores.
- Review failures manually, record the decision and assumptions, and schedule the next review date.
Use the result as an engineering decision aid, not a permanent ranking. A provider-neutral, versioned test plan lets you compare LLM APIs fairly, explain trade-offs to stakeholders, and repeat the analysis when prices, prompts, traffic, or model behavior changes.