Batch-1 interactive decode

Local LLM throughput calculator

Check whether a model fits in one accelerator’s memory, then estimate its ideal output-token ceiling from memory bandwidth and peak compute.

Hardware

One discrete GPU or one unified-memory system.

Accelerator preset

Total parameters determine fit; active parameters determine decode traffic.

Model preset

Weight format Context in use tokens

Quick context:

Compare streamed output speed; first-token latency is excluded.

Reference Observed speed tok/s

Advanced assumptions

KV cache precision Actual model file size optional override GB Decode compute precision Memory reserve OS + runtime GB

Peak compute is a second theoretical ceiling. Preset marketing figures that explicitly use structured sparsity are divided by two before use.

Ideal output speed

Fits in memory

Choose hardware and a model to see an estimate.

Can it fit? — —

—

Memory speed — —

—

Compute speed — —

—

Memory fit —

Estimated allocation inside usable accelerator memory.

Weights — KV cache — Headroom —

Compare with cloud

Local ceiling Cloud reference

Versus cloud — —

Show the math

This is an upper bound, not a benchmark.

Assumes one accelerator, batch size 1, weights resident in accelerator-accessible memory, and ideal sustained bandwidth.
Bandwidth traffic uses active—not total—parameters for MoE decode, plus the readable KV cache at the selected context.
Compute uses approximately two operations per active parameter. Kernel, framework, routing, sampling, and recurrent-state overheads are omitted.
Native ternary is modeled as 1.585 bits per weight only when purpose-built storage and kernels actually exist.

How to read the answer: Capacity determines what fits. RAM bandwidth and compute determine the ideal speed limit. Real software will be slower.