Qwen 2.5 72B
72 B parameters · 128K context · Qwen License (72B only) · released 2024-09-19 by Alibaba.
Qwen 2.5 72B needs about 144 GB at FP16: no single GPU can hold it. Serving requires an 8-GPU HGX baseboard (INT8/INT4) or a full rack (NVL72, TPU pod) at FP16.
Where this model sits
Trades Llama 3.1 70B on most benchmarks. Coder + Math variants beat closed-source rivals on code/reasoning specifically.
What runs it (production serving)
Datacenter GPUs and rack designs sized for Qwen 2.5 72B. Consumer cards are not practical at this size.
Smallest datacenter GPU that fits (INT4)
Multi-GPU / rack-scale (FP16)
Memory at different context lengths
Weights are fixed. The KV cache grows linearly with context length and can rival the weights at full window. Most guides only quote weights, then hand-wave a 15% overhead, which is only true at short context.
| Context | FP16 total | INT8 total | INT4 total |
|---|---|---|---|
| 4K (short chat) | 144 + 1.3 = 145.3 GB | 72 + 0.6 = 72.6 GB | 36 + 0.6 = 36.6 GB |
| 32K (typical RAG) | 144 + 10.0 = 154.0 GB | 72 + 5.0 = 77.0 GB | 36 + 5.0 = 41.0 GB |
| 128K (full window) | 144 + 40.0 = 184.0 GB | 72 + 20.0 = 92.0 GB | 36 + 20.0 = 56.0 GB |
KV bytes per token = 2 × 80 layers × 8 KV heads × 128 head-dim × precision bytes. INT4 rows keep the KV cache at INT8 (typical serving default).
Quality benchmarks
Vendor-reported scores from the release blog or tech report. Every row cites its source.
| Benchmark | Score | Source | Date |
|---|---|---|---|
| MMLU (5-shot) | 86.1 | Qwen 2.5 blog | 2024-09-19 |
| HumanEval | 86.6 | Qwen 2.5 blog | 2024-09-19 |
| MATH (0-shot) | 83.1 | Qwen 2.5 blog | 2024-09-19 |
| GPQA | 49.0 | Qwen 2.5 blog | 2024-09-19 |
Speed and cost to serve
What cloud providers currently charge to serve Qwen 2.5 72B. Tokens per second is what the provider publishes on their own page. Batch size, prompt length, and inference engine can shift these numbers a lot; treat as directional.
| Provider | Hardware | Speed | $/M in | $/M out | As of |
|---|---|---|---|---|---|
| Together AI Instruct Turbo | H100 | — | $1.20 | $1.20 | 2025-08-01 |
| Fireworks AI | — | — | $0.90 | $0.90 | 2025-08-01 |
| Alibaba Cloud (Model Studio) | Ascend | — | $0.55 | $1.65 | 2025-08-01 |
Cheapest option in this list for output-heavy workloads: Fireworks AI at $0.90/M out. Fastest published: not enough data.
License and variants
- License
- Qwen License (72B only)
- The gotcha
- The 72B is under Qwen License (not Apache 2.0 like the smaller Qwen sizes), free for commercial use unless MAU exceeds 100M.
- Variants
- Base · Instruct
- Context window
- 131,072 tokens (128K)
- Architecture
- dense
Compare with
Model card: https://huggingface.co/Qwen/Qwen2.5-72B · Release: Alibaba announcement
See also: every open-source LLM we track · all AI chips · rack designs.