| Takeaway | Detail |
|---|---|
| Clear a pre-committed p95 TTFT gate of 100 ms | Measure p95 time to first token on your own 512-token prompt and 128-token output mix at 40 req/s before selecting a framework. |
| Meet a tokens-per-second floor at 70% GPU utilization | A framework is defensible only if it satisfies the throughput floor while the GPU is running at 70% utilization on the same workload. |
| Choose the lowest measured p95 TTFT among qualifying frameworks | Rank only the frameworks that clear both the 100 ms p95 TTFT gate and the tokens-per-second floor at 70% GPU utilization. |
| Do not pick a framework if none clears both gates | When no option meets the p95 TTFT and throughput requirements, split the workload or lower the constraint instead of selecting a framework. |
This guide compares 8 LLM inference frameworks to help you choose between sub-100 ms time to first token and sustained throughput.
The right pick changes if no framework clears a 100 ms p95 TTFT gate and your tokens-per-second floor at 70% GPU utilization on a 512-token prompt, 128-token output mix at 40 req/s.
TTFT and throughput share one queue
In 2026, time to first token and throughput are not independent metrics because every request competes for the same scheduler queue. A request arrives, waits for admission, and only then begins prefill and decode. Continuous batching improves device efficiency by packing new arrivals into active batches, and KV-cache reuse reduces repeated prefill work, but both mechanisms change when a request reaches the front of the line. The queueing mechanism is simple: batching raises throughput but adds queue wait that can breach p95 TTFT. Because a small fraction of delayed requests can break an interactive experience, record p95 TTFT rather than mean TTFT when you evaluate an inference framework.
Use a queue-path check before you trust any benchmark. Draw the path from arrival to first token and mark every batching or KV-cache step where p95 TTFT can rise: (1) network and admission into the scheduler queue, (2) wait for a batch slot or scheduling window, (3) prefill of the 512-token prompt, including any KV-cache lookup or allocation, (4) first decode step that emits the first token. At each marked step, log queue depth, batch size, and per-request wait so you can see whether throughput gains are being purchased with tail latency.
Run the measurement as a pass/fail test, not a dashboard glance. Hold the request mix, arrival rate, and 70% GPU utilization target steady, then compare p95 TTFT against the 100 ms gate while confirming the tokens-per-second floor is still met. A framework that wins on throughput but fails the marked queue steps is signaling a scheduling tradeoff you must correct in configuration or reject for this workload. The defensible choice is the one whose queue behavior stays inside the gate under your own traffic shape.
What the available evidence can support
FastCI, fastercapital.com, and growett.com do not contain the measurements required to defend any inference-framework choice against the pre-committed latency and throughput gates. None reports a reproducible time-to-first-token value or a tokens-per-second figure for any named inference framework under a controlled workload. Before treating any framework claim as decision-grade, confirm that the source states the prompt length, output length, request rate, hardware, and percentile method used to produce the reported result. If any of those fields is missing, treat the claim as a hypothesis to test, not a selection.
FastCI, described on arXiv as work on efficient GPU-intensive continuous integration for LLM training frameworks, addresses the cost and operational burden of testing training code. It does not report inference time to first token or tokens-per-second measurements. A reader can use FastCI only as a caution about GPU-intensive validation workflows: budget and schedule the test infrastructure needed to run your own inference benchmarks, but do not treat training-CI evidence as a proxy for serving latency or serving throughput.
The cost-framework pages from fastercapital.com and growett.com discuss continuous improvement, cost modeling, and lean cost management rather than inference latency. According to fastercapital.com, implementing a cost modeling framework is a continuous process that requires monitoring, evaluation, and improvement; growett.com similarly frames lean cost management as a culture of identifying inefficiencies. These sources support a measurement loop—define the gate, run the test, review the result, and repeat—but they do not supply framework rankings or performance data for inference serving.
The convergence check across the supplied material is unambiguous: zero of three named source groups provides a reproducible time-to-first-token or tokens-per-second figure. Under that condition, the correct action is to withhold a framework pick. Do not infer that any candidate clears a sub-100 ms gate from cost guidance, training-CI research, or general management frameworks; require a controlled benchmark on the buyer's own workload before committing.
Apply one winner rule after the scorecard is complete: select the candidate with the lowest measured p95 TTFT that still meets your tokens-per-second floor at 70% GPU utilization. Treat the p95 TTFT gate as mandatory for this workload; if no candidate clears both gates, do not let a throughput-maximizing candidate win—split the workload or lower the admitted concurrency and rerun the test.
Reject any candidate whose throughput advantage disappears after the p95 gate is enforced. A framework that looks efficient on aggregate output but fails the latency gate under the same configuration is not a valid winner for a latency-gated workload. This comparison produces a defensible pick for your measured conditions; it does not establish a universal product winner across models, hardware, or traffic shapes.
Before declaring a verdict, verify that breach count, percentile latency, and throughput were collected from the same run and the same admission settings. If a candidate passes on one metric but the evidence was gathered under different batch or concurrency settings, rerun the test. The scorecard is only decision-grade when every slot is judged against the same gate, the same floor, and the same measurement method.
Token economics: the numbers to price
In 2026, token economics turns a framework benchmark into a defensible budget decision. A cost framework provides a systematic structure for analyzing and managing costs (ScienceDirect, Cost Framework), and a cost modeling framework requires continuous monitoring, evaluation, and improvement rather than a one-time calculation (FasterCapital, Cost Modeling Framework). For inference, that discipline means pricing the output your deployment actually delivers at the utilization level you can sustain while meeting your pre-committed latency and throughput gates.
This section alone defines cost per million output tokens from GPU-hour price and measured tokens/sec. Use the formula: cost per 1M output tokens = (GPU-hour price × 1,000,000) / (measured tokens/sec × 3,600 × utilization). In this formula, GPU-hour price is the amount you pay for one hour of the accelerator, measured tokens/sec is the output rate observed on your own workload, 3,600 converts hours to seconds, and utilization is the fraction of GPU capacity consumed by productive decode work during the measurement window.
Before ranking frameworks, require three price inputs: an on-demand GPU-hour quote, a reserved or committed quote, and an idle-replica cost. Do not treat list price as production cost. Record each quote with its billing unit, commitment term, and region, then run the same formula for each input. This check prevents a framework that looks inexpensive on a headline rate from becoming the most expensive option once reservation discounts, spare capacity, and always-on replicas are included.
Recompute effective output before deciding, because throughput and utilization move together. For the check specified here, compare a 2× throughput gain at 50% utilization with a 1.2× gain at 80% utilization: 2 × 0.50 = 1.00, and 1.2 × 0.80 = 0.96. The effective output factors are 1.00 versus 0.96, so the higher-throughput option delivers only a marginal advantage in this example and can lose once its GPU-hour quote or idle-replica cost is higher. Run the full cost-per-1M-output-tokens calculation on your measured values before you decide.
Apply the result as a pricing gate, not a standalone winner. Compute cost per million output tokens for every candidate that you measure on the same workload, then keep only the options that still clear your tokens-per-second floor at 70% GPU utilization. If no candidate clears both the latency gate and the throughput floor, do not force a framework pick; split the workload or lower the commitment until the economics and service targets align.
Where the
The 100 ms p95 time-to-first-token gate is defensible only when the measurement reflects the conditions your production traffic will actually create. A warmed benchmark can hide three edge cases that invalidate the result: cold start, long prompts, and batch-1 workloads. Each case changes the work performed before the first token is emitted, so a framework that passes a warmed p95 test on a short, steady mix may still miss the pre-committed gate in 2026 production.
| Condition | What changes | Rule status | Required check |
|---|---|---|---|
| Cold start vs. warmed KV cache | Model load, graph compilation, and cache population add work before prefill. | Breaks if the SLA includes first-request or scale-out latency. | Measure cold-start p95 separately and report its breach rate. |
| 128-token vs. 8k-token prompts | Prefill cost grows with prompt length, increasing time before the first output token. | Breaks for long-context contracts; holds when the SLA is production-mix p95 TTFT. | Test at the prompt lengths your traffic actually sends. |
| Batch-1 latency vs. saturated throughput | Single-request runs remove admission delay but do not represent shared-capacity behavior. | Breaks if batch-1 results are used as a capacity claim. | Evaluate at the target request rate and utilization defined in the gate. |
| p99 or long-context contract | Tail latency and extreme prompt sizes tighten the effective budget. | Breaks under p99 or long-context SLAs; still wins when production-mix p95 TTFT is the SLA. | Match the percentile and context profile to the contract you will enforce. |
Do not accept a warmed-cache p95 as the service-level agreement. Run a separate cold-start test that captures process restarts, autoscaling events, and cache-miss paths, then report the breach rate against the 100 ms gate. If cold-start breaches are frequent in your deployment pattern, treat the warmed result as incomplete evidence rather than a pass.
If prompt length p95 is undocumented, the 100 ms gate is untestable. Require a token-length histogram from production logs or a representative trace before shortlisting any framework. The histogram lets you define the prompt mix, verify that the 512-token test case is relevant, and identify whether long-context requests need a separate workload split.
Apply these checks before committing to a framework choice: confirm the percentile, confirm cache state, and confirm prompt-length distribution. A framework that clears the gate only under warmed, short-prompt, batch-1 conditions has not cleared the buyer's production contract.
Worked run: 512-in, 128-out at 40 req/s
This section alone gives the worked measurement: a fixed 512-token prompt and 128-token output mix at 40 req/s evaluated against a 100 ms p95 TTFT gate. In 2026, run the test on the buyer's own hardware and serving stack with four locked inputs: 512 prompt tokens, 128 output tokens, a sustained arrival rate of 40 req/s, and a 70% GPU utilization cap. Treat the 100 ms p95 TTFT gate as pre-committed before the run starts so the result is a pass-fail decision rather than a post hoc interpretation.
Choose and document a warm-up period, and discard those samples from the reported distribution. Then hold the workload at steady state for a documented duration and record p50, p95, and p99 TTFT alongside aggregate tokens/sec. The utilization cap matters as much as the latency gate: a configuration that clears 100 ms p95 TTFT only by running the device well below 70% GPU utilization has not met the operating condition this measurement is designed to test.
After the steady-state window, apply a documented overload spike at a multiple of the 40 req/s baseline and record the breach count against the 100 ms p95 TTFT gate. Capture the breach count separately from the steady-state percentiles so a short overload does not hide inside the steady-state average. If the serving stack exposes per-request timestamps, keep the raw log so the p95 calculation and breach count can be recomputed during review.
The cost of being wrong runs in both directions. A throughput-max choice that breaches the 100 ms p95 TTFT gate creates exposure to an SLA penalty, while a latency-only choice that misses the tokens/sec floor creates queue growth under sustained load. No observed benchmark result is asserted here; the worked run produces the buyer's own measurements, and any framework that fails either checkpoint should be rejected for this workload rather than tuned after the fact.
Document the full run as a repeatable artifact: input shape, arrival rate, warm-up count, steady-state duration, spike duration, utilization cap, and the recorded p50/p95/p99 TTFT, tokens/sec, and breach count. Consistent with the continuous monitoring discipline described in cost and operations frameworks such as FasterCapital's guidance on implementing a cost modeling framework, the same measurement should be rerun whenever the model, prompt mix, batch policy, or hardware changes.
Decision rules for the 2026 shortlist
Rule 1 — Contractual latency: If p95 TTFT at or below 100 ms is the pre-committed gate for the workload, select the candidate with the lowest measured p95 TTFT that still clears the tokens-per-second floor. If no candidate clears both gates, do not substitute a highest-throughput pick; split the workload or lower the admitted concurrency and rerun the test.
Rule 2 — Prompt length transfer: If the production p95 prompt length exceeds 2,000 tokens, rerun the full two-gate test before deciding. A result from the 512-token prompt mix does not transfer to longer prompts, so treat the original measurement as invalid for that workload and require a new pass/fail record at the observed prompt distribution.
Rule 3 — Cold start versus warmed state: If cold-start p95 TTFT breaches 100 ms but warmed p95 TTFT passes, require a documented pre-warming procedure as a deployment condition or reject the candidate for any interactive service-level agreement. The acceptance check is simple: the state your users will actually hit must clear the 100 ms p95 gate, not only the warmed steady state.
Rule 4 — No candidate clears both gates: If no candidate clears the 100 ms p95 TTFT gate and the tokens-per-second floor at 70% GPU utilization, do not pick a framework. Split the workload into latency-sensitive and throughput-sensitive paths, or lower the admitted concurrency until a candidate passes both gates on the revised test. Document the change and rerun the measurement before selection.
Rule 5 — Baseline drift: If your target concurrency, prompt/output mix, or operating utilization differs from 40 requests per second, the 512-token/128-token mix, or 70% GPU utilization, rerun the test at the new baseline before deciding. A defensible 2026 choice is tied to the exact conditions you pre-committed to; any material change to those conditions requires a new two-gate measurement and a fresh application of Rules 1 through 4.
What to do next
| Step | Action | Why it matters |
|---|---|---|
| 1 | Build a fixed benchmark for all 8 LLM inference frameworks using your own 512-token prompt and 128-token output mix at 40 req/s. | A consistent workload ensures every framework is evaluated under the same conditions before any selection decision. |
| 2 | Measure p95 time to first token (TTFT) for each framework on that workload and record which ones clear the pre-committed 100 ms gate. | Only frameworks that meet the p95 TTFT gate remain eligible for comparison. |
| 3 | On the same workload, measure tokens per second for each framework while holding GPU utilization at the 70% level named above, and check each result against your throughput floor. | A framework is defensible only when it meets the throughput floor at the target GPU utilization, not just on latency. |
| 4 | Rank only the frameworks that clear both gates, ordering them by lowest measured p95 TTFT. | This applies the canonical decision rule directly and prevents a fast-but-underpowered or high-throughput-but-slow option from being chosen. |
| 5 | If no framework clears both the 100 ms p95 TTFT gate and the tokens-per-second floor at the 70% GPU utilization level, do not pick a framework. | Forcing a choice when no option qualifies leads to missed latency or throughput targets in production. |
| 6 | When no option qualifies, split the workload across separate serving paths or lower concurrency, then rerun the same benchmark before deciding. | Adjusting the workload or constraint creates a valid path to a framework that can meet both gates. |
Frequently Asked Questions
What workload parameters should be used when measuring p95 time to first token for framework selection?
Measure p95 time to first token on a 512-token prompt and 128-token output mix at 40 req/s before selecting a framework.
What is the p95 time to first token threshold a framework must clear in this test plan?
A framework must clear a pre-committed p95 TTFT gate of 100 ms to qualify.
Under what condition is an inference framework considered defensible on throughput?
A framework is defensible only if it satisfies the tokens-per-second floor while the GPU is running at the stated utilization target on the same workload.
How should qualifying inference frameworks be ranked after testing?
Among frameworks that clear both the 100 ms p95 TTFT gate and the tokens-per-second floor at the stated GPU utilization target, choose the one with the lowest measured p95 TTFT.
What action should be taken if no inference framework meets both performance gates?
If no option meets the p95 TTFT and throughput requirements, split the workload or lower the constraint instead of selecting a framework.
Why do time to first token and throughput affect each other during inference?
Time to first token and throughput share one scheduler queue, so every request competes for admission before it begins prefill and decode.
Quick answers
| What workload should be used to measure the p95 TTFT gate before selecting a framework? | Measure p95 time to first token on your own 512-token prompt and 128-token output mix at 40 req/s before selecting a framework. |
| When is an LLM inference framework considered defensible? | A framework is defensible only if it satisfies the throughput floor while the GPU is running at the utilization threshold mentioned above on the same workload. |
| How should you choose among frameworks that meet the test requirements? | Choose the lowest measured p95 TTFT among qualifying frameworks that clear both the 100 ms p95 TTFT gate and the tokens-per-second floor at the specified GPU utilization. |
| What should you do if no framework clears both the p95 TTFT and throughput gates? | Do not pick a framework; instead, split the workload or lower the constraint. |
| Why are time to first token and throughput not independent metrics? | Time to first token and throughput are not independent because every request competes for the same scheduler queue. |
Also worth reading: 2026 B2B Sensitive Exports: Real-Time vs Batch Timing Pitfalls: 2026 B2B Sensitive Exports: Real-Time · Data Retention: 3 Governance Models vs. Time-to-Market: Data Retention: 3 Governance Models · GDPR Audit Prep: Centralized vs Federated May Cut Time 40%: GDPR Audit Prep: Centralized vs