Open-source LLM · Mistral AI
Mixtral 8x7B
46.7 B parameters (12.9 B active per token) · 32K context · Apache 2.0 · released 2023-12-11 by Mistral AI.
Mixtral 8x7B needs about 93 GB at FP16: no single GPU can hold it. Serving requires an 8-GPU HGX baseboard (INT8/INT4) or a full rack (NVL72, TPU pod) at FP16.
Where this model sits
The first open MoE that mattered. Now behind Llama 3.1 70B on most benchmarks but still serves well when Apache 2.0 is required and cost is tight.
What runs it (production serving)
Datacenter GPUs and rack designs sized for Mixtral 8x7B. Consumer cards are not practical at this size.
Smallest datacenter GPU that fits (INT4)
Multi-GPU / rack-scale (FP16)
Memory at different context lengths
Weights are fixed. The KV cache grows linearly with context length and can rival the weights at full window. Most guides only quote weights, then hand-wave a 15% overhead, which is only true at short context.
| Context | FP16 total | INT8 total | INT4 total |
|---|---|---|---|
| 4K (short chat) | 93 + 0.5 = 93.9 GB | 47 + 0.3 = 47.0 GB | 23 + 0.3 = 23.6 GB |
| 32K (typical RAG) | 93 + 4.0 = 97.4 GB | 47 + 2.0 = 48.7 GB | 23 + 2.0 = 25.4 GB |
KV bytes per token = 2 × 32 layers × 8 KV heads × 128 head-dim × precision bytes. INT4 rows keep the KV cache at INT8 (typical serving default).
Quality benchmarks
Vendor-reported scores from the release blog or tech report. Every row cites its source.
| Benchmark | Score | Source | Date |
|---|---|---|---|
| MMLU (5-shot) | 70.6 | Mixtral 8x7B blog | 2023-12-11 |
| HumanEval | 40.2 | Mixtral 8x7B blog | 2023-12-11 |
| MATH (maj@4) | 28.4 | Mixtral 8x7B blog | 2023-12-11 |
Speed and cost to serve
What cloud providers currently charge to serve Mixtral 8x7B. Tokens per second is what the provider publishes on their own page. Batch size, prompt length, and inference engine can shift these numbers a lot; treat as directional.
| Provider | Hardware | Speed | $/M in | $/M out | As of |
|---|---|---|---|---|---|
| Groq | Groq LPU | 500 tok/s | $0.24 | $0.24 | 2025-08-01 |
| Together AI | H100 | — | $0.60 | $0.60 | 2025-08-01 |
| AWS Bedrock | — | — | $0.45 | $0.70 | 2025-08-01 |
Cheapest option in this list for output-heavy workloads: Groq at $0.24/M out. Fastest published: Groq at 500 tok/s (Groq LPU).
License and variants
- License
- Apache 2.0
- Variants
- Base · Instruct
- Context window
- 32,768 tokens (32K)
- Architecture
- MoE (46.7B total, 12.9B active per token)
Compare with
Model card: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 · Release: Mistral AI announcement
See also: every open-source LLM we track · all AI chips · rack designs.