Llama 3.1 8B
8 B parameters · 128K context · Llama 3.1 Community License · released 2024-07-23 by Meta.
At INT4 Llama 3.1 8B needs about 4 GB and runs on almost any modern GPU including consumer cards like the RTX 4090. A single H100 handles FP16 (16 GB weights) at short context. Long-context production serving wants an L40S or H100.
Where this model sits
The default small open-weight LLM. Instruct variant clears most everyday tasks (chat, coding, tool use) with quality competitive with GPT-3.5 while running on a single consumer GPU.
What to run it on
From cheapest workable option to production-grade. Bigger memory = longer context + more concurrent users, not a bigger model.
Consumer / desktop
- NVIDIA RTX 409024 GB · $1,800street, 24 GB GDDR6X
- NVIDIA RTX 309024 GB · $800used market, 24 GB GDDR6X
- AMD Radeon RX 7900 XTX24 GB · $1,000new, ROCm/HIP support in llama.cpp + Ollama
- NVIDIA RTX 509032 GB · $2,500MSRP, 32 GB GDDR7
- Apple M4 Mac Mini (32 GB)32 GB · $1,600cheapest Mac Silicon with meaningful LLM memory
- NVIDIA RTX 6000 Ada48 GB · $7,000workstation card, new
All above hold Llama 3.1 8B at INT4 or better. INT8 needs ≥ 8 GB; FP16 needs ≥ 16 GB. Local tools: Ollama, LM Studio, llama.cpp (Linux/Win/Mac); MLX (Apple Silicon); vLLM for higher-throughput serving. Most set up in under 10 minutes.
Datacenter (FP16, production serving)
Memory at different context lengths
Weights are fixed. The KV cache grows linearly with context length and can rival the weights at full window. Most guides only quote weights, then hand-wave a 15% overhead, which is only true at short context.
| Context | FP16 total | INT8 total | INT4 total |
|---|---|---|---|
| 4K (short chat) | 16 + 0.5 = 16.5 GB | 8 + 0.3 = 8.3 GB | 4 + 0.3 = 4.3 GB |
| 32K (typical RAG) | 16 + 4.0 = 20.0 GB | 8 + 2.0 = 10.0 GB | 4 + 2.0 = 6.0 GB |
| 128K (full window) | 16 + 16.0 = 32.0 GB | 8 + 8.0 = 16.0 GB | 4 + 8.0 = 12.0 GB |
KV bytes per token = 2 × 32 layers × 8 KV heads × 128 head-dim × precision bytes. INT4 rows keep the KV cache at INT8 (typical serving default).
Quality benchmarks
Vendor-reported scores from the release blog or tech report. Every row cites its source.
| Benchmark | Score | Source | Date |
|---|---|---|---|
| MMLU (5-shot) | 73.0 | Meta Llama 3.1 blog | 2024-07-23 |
| HumanEval | 72.6 | Meta Llama 3.1 blog | 2024-07-23 |
| MATH (0-shot) | 51.9 | Meta Llama 3.1 blog | 2024-07-23 |
| GPQA | 32.8 | Meta Llama 3.1 blog | 2024-07-23 |
Speed and cost to serve
What cloud providers currently charge to serve Llama 3.1 8B. Tokens per second is what the provider publishes on their own page. Batch size, prompt length, and inference engine can shift these numbers a lot; treat as directional.
| Provider | Hardware | Speed | $/M in | $/M out | As of |
|---|---|---|---|---|---|
| Groq Instant | Groq LPU | 750 tok/s | $0.05 | $0.08 | 2025-08-01 |
| Together AI Instruct Turbo | H100 | — | $0.18 | $0.18 | 2025-08-01 |
| AWS Bedrock Instruct | — | — | $0.22 | $0.22 | 2025-08-01 |
| Fireworks AI | H100 | — | $0.20 | $0.20 | 2025-08-01 |
Cheapest option in this list for output-heavy workloads: Groq at $0.08/M out. Fastest published: Groq at 750 tok/s (Groq LPU).
Serving scale on rack designs
On a rack-scale AI system, Llama 3.1 8B isn't a fit question: everything fits many times over. The real number is how many independent replicas the rack can host at typical context (32K, INT8). Ceiling: ignores rack overhead, so real deployments hit ~60-80% of this.
| Rack design | Total memory | Replicas (INT8, 32K) | Per replica |
|---|---|---|---|
| Google TPU v5p Pod (8,960 chips) | 851,200 GB | 85,120 | 10.0 GB |
| Huawei Atlas 900 A3 SuperCluster (Ascend 910B) | 524,288 GB | 52,428 | 10.0 GB |
| NVIDIA DGX SuperPOD H100 (32-node reference) | 20,480 GB | 2,048 | 10.0 GB |
| NVIDIA GB200 NVL72 | 13,824 GB | 1,382 | 10.0 GB |
| Google TPU v6e (Trillium) Pod-256 | 8,192 GB | 819 | 10.0 GB |
| AWS Trainium 2 UltraServer | 6,144 GB | 614 | 10.0 GB |
Multiply by the per-replica throughput above (Groq / Together / Fireworks tokens/sec) to estimate aggregate throughput. That figure varies by engine (vLLM, TensorRT-LLM, SGLang) and batch policy, so we don't compute it here.
License and variants
- License
- Llama 3.1 Community License
- The gotcha
- Free for commercial use unless your product has 700M+ monthly active users at launch, in which case you need a separate Meta license.
- Variants
- Base · Instruct · Guard
- Context window
- 131,072 tokens (128K)
- Architecture
- dense
Compare with
Model card: https://huggingface.co/meta-llama/Llama-3.1-8B · Release: Meta announcement
See also: every open-source LLM we track · all AI chips · rack designs.