How much model fits on my GPU?

Every fit and speed on this page is an estimate, calculated from published specs and the rules below. Real speeds depend on the engine, its settings and the prompt, and are often lower.

Memory

A model fits when its weights, its KV cache and an overhead for working space fit in usable memory:

weights   = parameters x bits per weight / 8
KV cache  = 2 bytes x tokens x layers that keep every token x 2 x KV heads x head size
            (+ the same for sliding-window layers, up to the window)
overhead  = 1 GiB + 5% of the weights
needed    = weights + KV cache + overhead

• A model fits when it needs at most 90% of usable memory, which leaves room for the system and longer prompts. It’s tight at 90% to 100% and won’t fit above that.
• Bits per weight are llama.cpp’s measured averages for its quantization types: Q8_0 8.50, Q6_K 6.56, Q5_K_M 5.70, Q4_K_M 4.89 and Q3_K_M 4.00 (llama.cpp quantize README). gpt-oss ships only in MXFP4. Its bits per weight are its checkpoint size divided by its parameter count.
• “Best that fits” picks the highest of those types that fits, or else the highest that’s tight. It skips 16-bit, which needs twice the memory of 8-bit for little gain.
• Layer counts, KV heads and head sizes come from each model’s config.json on Hugging Face. Layers with linear attention or a state-space design keep no KV cache, and models with latent attention cache one compressed vector per token.
• The context is capped at the model’s own maximum.

Usable memory

Usable memory is the part of a computer’s memory a model can use:

For a custom machine, you enter this figure.

Speed

Generating a token reads every active weight from memory once, plus the KV cache so far, so memory bandwidth sets the speed:

tokens per second                 = bandwidth x 0.6 / active weight bytes
tokens per second, window full    = bandwidth x 0.6 / (active weight bytes + KV cache)

• 0.6 is the assumed share of the stated bandwidth that an engine reaches. It’s a round figure, not a measurement.
• A mixture-of-experts model reads only its active parameters for each token, so a 35B model with 3B active can be faster than a dense 8B model.
• The first figure is for an empty cache, at the start of a chat. The second is for the whole chosen context in the cache, the slowest case.
• Both are speeds for generating the answer. Reading the prompt isn’t counted.

What’s left out

• Running part of a model from system memory when it doesn’t fit on the graphics card.
• Prompt processing, batching and several users at once.
• Differences between engines (llama.cpp, MLX, vLLM and others), speculative decoding, and quantization formats other than the llama.cpp types above.
• Vision encoders loaded beside a model, unless the model’s stated size includes them.

For memory, GB means GiB (2³⁰ bytes), as vendors use it for RAM and VRAM. For bandwidth, GB/s means 10⁹ bytes per second.