How to Know What Model Fits Your GPU: The Math, Memory, and Method Behind Local LLM Inference

Introduction: The Hard Limit You Can't Ignore

Every enthusiast who has tried to run a large language model locally hits the same wall: VRAM. Unlike system RAM, video memory cannot be easily expanded, swapped, or compressed without severe performance penalties. If your model's weights, KV cache, and activation buffers exceed your GPU's VRAM, inference either fails outright or falls back to painfully slow CPU offloading.

Understanding exactly how much memory a model needs — and why — transforms local LLM deployment from trial-and-error into an engineering decision. This article gives you the formulas, the intuition, and the practical rules of thumb to answer the only question that matters: "Will this model fit on my GPU?"

The Three Memory Consumers

Before diving into math, distinguish the three distinct memory consumers during inference:

• Model Weights (Static): The parameter tensors themselves. This is the dominant term and scales linearly with parameter count and quantization precision.
• KV Cache (Dynamic, per token): Key-Value attention caches that grow with context length, batch size, and number of layers. For long contexts, this can rival or exceed weight memory.
• Activation Buffers (Transient): Intermediate tensors during the forward pass. Usually small for inference (batch=1), but non-trivial for large models or batch processing.

"VRAM is the budget; weights are the fixed cost, KV cache is the variable cost, and activations are the transaction fee. Optimize in that order."

— Hesham Kholef

The Core Formula: Weight Memory

The baseline memory for model weights is straightforward:

weight_memory_bytes = parameter_count * bytes_per_parameter

Bytes per parameter by quantization:

Example: A 7B parameter model at INT4 (4-bit):

7_000_000_000 * 0.5  # = 3.5 GB (weights only)

The Hidden Giant: KV Cache Memory

The KV cache stores Key and Value projections for every layer, every head, every token in the context. Its formula:

kv_cache_bytes = 2 * num_layers * num_kv_heads * head_dim * context_length * batch_size * bytes_per_element

Where bytes_per_element is typically 2 (FP16/BF16) even for quantized models, because attention computation usually runs in higher precision.

Simplified for common architectures (Llama, Mistral, Qwen):

# Approximation: 2 * hidden_size * context_length * bytes_per_element
# hidden_size = num_layers * num_kv_heads * head_dim (roughly model_dim)
kv_cache_gb ≈ (2 * hidden_size * context_length * 2) / 1e9

Practical examples (FP16 KV cache, batch=1):

• Llama-3-8B (hidden=4096): 32K context ≈ 0.52 GB, 128K context ≈ 2.1 GB
• Qwen2.5-32B (hidden=5120): 32K context ≈ 0.66 GB, 128K context ≈ 2.6 GB
• Llama-3.1-70B (hidden=8192): 32K context ≈ 1.05 GB, 128K context ≈ 4.2 GB

This is why long-context inference on consumer GPUs often requires KV cache quantization (FP8, INT8, or K-quants) or sliding-window attention.

Total VRAM Estimation Formula

Combine everything into a single estimator:

def estimate_vram_GB(params_B, quant_bits, context_len, hidden_size, 
                     kv_quant_bits=16, batch_size=1, overhead_factor=1.15):
    """
    Returns estimated VRAM in GB for inference.
    params_B: model parameters in billions (e.g., 7, 8, 70)
    quant_bits: weight quantization bits (4, 8, 16)
    context_len: max context length (tokens)
    hidden_size: model hidden dimension (e.g., 4096, 5120, 8192)
    kv_quant_bits: KV cache precision (16 for FP16, 8 for INT8, 4 for INT4)
    batch_size: concurrent sequences
    overhead_factor: safety margin for activations, fragmentation, OS (1.1–1.2)
    """
    # Weights
    weight_gb = params_B * 1e9 * (quant_bits / 8) / 1e9
    
    # KV Cache (2 for K and V)
    kv_bytes_per_token = 2 * hidden_size * (kv_quant_bits / 8)
    kv_gb = kv_bytes_per_token * context_len * batch_size / 1e9
    
    # Activations (~2-5% of weights for batch=1)
    activation_gb = weight_gb * 0.03
    
    total = (weight_gb + kv_gb + activation_gb) * overhead_factor
    return round(total, 2)

# Examples:
print(f"Llama-3-8B INT4, 32K ctx: {estimate_vram_GB(8, 4, 32768, 4096)} GB")
print(f"Qwen2.5-32B INT4, 32K ctx: {estimate_vram_GB(32, 4, 32768, 5120)} GB")
print(f"Llama-3.1-70B INT4, 8K ctx: {estimate_vram_GB(70, 4, 8192, 8192)} GB")

Quick Reference Cheat Sheet (INT4 Weights, FP16 KV, 4K–8K Context)

Add 2–4 GB for 32K+ context windows. For Apple Silicon (Unified Memory), total system RAM replaces VRAM — but reserve 8–16 GB for OS and apps.

Quantization: The Quality/Size Trade-off

Quantization reduces weight precision. The standard hierarchy for local inference:

• Q4_K_M / Q4_K_S (GGUF) — Sweet spot. Near-FP16 quality for most tasks, 4.5–5 bits/param effective.
• Q5_K_M — Noticeable quality gain on coding/reasoning, ~25% larger than Q4.
• Q6_K — Diminishing returns; only for quality-critical workloads.
• Q3_K_M / Q2_K — Avoid unless VRAM is severely constrained; coherence degrades.
• GPTQ / AWQ / EXL2 (4-bit, 128-group) — GPU-native formats; faster than GGUF on NVIDIA, similar quality.

Context Length vs. VRAM: The Linear Trap

KV cache scales linearly with context length. Doubling context doubles KV memory. For a 70B model at INT4:

• 4K context: ~36 GB (weights) + 0.3 GB (KV) = ~37 GB
• 32K context: ~36 GB + 2.5 GB = ~39 GB
• 128K context: ~36 GB + 10 GB = ~47 GB

If your GPU has 24 GB VRAM, you cannot run 70B at 128K context — regardless of quantization — unless you use:

• KV Cache Quantization (FP8/INT8): Cuts KV memory by 2–4× (supported in vLLM, TensorRT-LLM, llama.cpp recent builds).
• Sliding Window / Attention Sinks: Keeps only recent tokens (e.g., mistral, gemma2).
• CPU Offloading (llama.cpp -ngl): Offloads layers to system RAM; slow but works.

Real-World GPU Mapping (2024–2025 Consumer Cards)

Tools That Do the Math for You

• Hugging Face VRAM Calculator — Interactive, accounts for KV cache, quantization, FlashAttention.
• llama.cpp Memory Docs — Authoritative formulas for GGUF.
• vLLM Profiling Guide — Production-grade KV cache estimation.
• nvidia-smi -q -d MEMORY — Check actual free VRAM before loading.

Practical Decision Workflow

1. Measure free VRAM: nvidia-smi after boot, no other GPU apps. Note Free memory.
2. Pick target context: 4K (chat), 8K (RAG), 32K+ (long docs).
3. Choose quantization: Start at Q4_K_M (GGUF) or 4-bit EXL2/AWQ.
4. Run the estimator: Use the Python function above or HF calculator.
5. Apply 1.15× safety margin: If estimate × 1.15 < free VRAM → green light.
6. Test load: llama-cli -m model.gguf -n 1 -p "test" -ngl 99. Watch for CUDA out of memory or sudden CPU fallback (token/s drops < 1).

Conclusion: Math Beats Guesswork

The difference between a frustrating weekend of OOM errors and a smooth local LLM setup is 15 minutes of arithmetic. Weights are predictable; KV cache is the variable you control via context length and quantization. Master the formula Total ≈ Weights + KV_Cache + Overhead, and you'll never again download a 40 GB model for a 24 GB card.

Your GPU has a hard ceiling. The model you choose should live comfortably below it — with headroom for the context your actual use case demands.