Qwen 2.5 7B
7 B parameters · 128K context · Apache 2.0 · released 2024-09-19 by Alibaba.
At INT4 Qwen 2.5 7B needs about 4 GB and runs on almost any modern GPU including consumer cards like the RTX 4090. A single H100 handles FP16 (14 GB weights) at short context. Long-context production serving wants an L40S or H100.
Where this model sits
Consistently strong on multilingual + math. Apache 2.0 makes it the default choice when the Llama license terms are a blocker.
What to run it on
From cheapest workable option to production-grade. Bigger memory = longer context + more concurrent users, not a bigger model.
Consumer / desktop
- NVIDIA RTX 409024 GB · $1,800street, 24 GB GDDR6X
- NVIDIA RTX 309024 GB · $800used market, 24 GB GDDR6X
- AMD Radeon RX 7900 XTX24 GB · $1,000new, ROCm/HIP support in llama.cpp + Ollama
- NVIDIA RTX 509032 GB · $2,500MSRP, 32 GB GDDR7
- Apple M4 Mac Mini (32 GB)32 GB · $1,600cheapest Mac Silicon with meaningful LLM memory
- NVIDIA RTX 6000 Ada48 GB · $7,000workstation card, new
All above hold Qwen 2.5 7B at INT4 or better. INT8 needs ≥ 7 GB; FP16 needs ≥ 14 GB. Local tools: Ollama, LM Studio, llama.cpp (Linux/Win/Mac); MLX (Apple Silicon); vLLM for higher-throughput serving. Most set up in under 10 minutes.
Datacenter (FP16, production serving)
Memory at different context lengths
Weights are fixed. The KV cache grows linearly with context length and can rival the weights at full window. Most guides only quote weights, then hand-wave a 15% overhead, which is only true at short context.
| Context | FP16 total | INT8 total | INT4 total |
|---|---|---|---|
| 4K (short chat) | 14 + 0.2 = 14.2 GB | 7 + 0.1 = 7.1 GB | 4 + 0.1 = 3.6 GB |
| 32K (typical RAG) | 14 + 1.8 = 15.8 GB | 7 + 0.9 = 7.9 GB | 4 + 0.9 = 4.4 GB |
| 128K (full window) | 14 + 7.0 = 21.0 GB | 7 + 3.5 = 10.5 GB | 4 + 3.5 = 7.0 GB |
KV bytes per token = 2 × 28 layers × 4 KV heads × 128 head-dim × precision bytes. INT4 rows keep the KV cache at INT8 (typical serving default).
Quality benchmarks
Vendor-reported scores from the release blog or tech report. Every row cites its source.
| Benchmark | Score | Source | Date |
|---|---|---|---|
| MMLU (5-shot) | 74.2 | Qwen 2.5 blog | 2024-09-19 |
| HumanEval | 84.8 | Qwen 2.5 blog | 2024-09-19 |
| MATH (0-shot) | 75.5 | Qwen 2.5 blog | 2024-09-19 |
Speed and cost to serve
What cloud providers currently charge to serve Qwen 2.5 7B. Tokens per second is what the provider publishes on their own page. Batch size, prompt length, and inference engine can shift these numbers a lot; treat as directional.
| Provider | Hardware | Speed | $/M in | $/M out | As of |
|---|---|---|---|---|---|
| Together AI Instruct Turbo | H100 | — | $0.30 | $0.30 | 2025-08-01 |
| Alibaba Cloud (Model Studio) | Ascend | — | $0.06 | $0.18 | 2025-08-01 |
Cheapest option in this list for output-heavy workloads: Alibaba Cloud (Model Studio) at $0.18/M out. Fastest published: not enough data.
Serving scale on rack designs
On a rack-scale AI system, Qwen 2.5 7B isn't a fit question: everything fits many times over. The real number is how many independent replicas the rack can host at typical context (32K, INT8). Ceiling: ignores rack overhead, so real deployments hit ~60-80% of this.
| Rack design | Total memory | Replicas (INT8, 32K) | Per replica |
|---|---|---|---|
| Google TPU v5p Pod (8,960 chips) | 851,200 GB | 108,088 | 7.9 GB |
| Huawei Atlas 900 A3 SuperCluster (Ascend 910B) | 524,288 GB | 66,576 | 7.9 GB |
| NVIDIA DGX SuperPOD H100 (32-node reference) | 20,480 GB | 2,600 | 7.9 GB |
| NVIDIA GB200 NVL72 | 13,824 GB | 1,755 | 7.9 GB |
| Google TPU v6e (Trillium) Pod-256 | 8,192 GB | 1,040 | 7.9 GB |
| AWS Trainium 2 UltraServer | 6,144 GB | 780 | 7.9 GB |
Multiply by the per-replica throughput above (Groq / Together / Fireworks tokens/sec) to estimate aggregate throughput. That figure varies by engine (vLLM, TensorRT-LLM, SGLang) and batch policy, so we don't compute it here.
License and variants
- License
- Apache 2.0
- Variants
- Base · Instruct · Coder · Math
- Context window
- 131,072 tokens (128K)
- Architecture
- dense
Compare with
Model card: https://huggingface.co/Qwen/Qwen2.5-7B · Release: Alibaba announcement
See also: every open-source LLM we track · all AI chips · rack designs.