Gemma 2 27B
27 B parameters · 8K context · Gemma Terms of Use · released 2024-06-27 by Google.
Gemma 2 27B needs about 54 GB at FP16, 27 GB at INT8, or 14 GB at INT4. Runs on one H100 (80 GB) at INT8/INT4; needs H200 (141 GB) or an 8-GPU baseboard for FP16.
Where this model sits
Punchy for its size but the 8K context window is a serious limitation vs. Llama 3.1 8B's 128K. Best for fine-tuning to a specific narrow task.
What runs it (production serving)
Datacenter GPUs and rack designs sized for Gemma 2 27B. Consumer cards are not practical at this size.
Smallest datacenter GPU that fits (INT4)
Memory at different context lengths
Weights are fixed. The KV cache grows linearly with context length and can rival the weights at full window. Most guides only quote weights, then hand-wave a 15% overhead, which is only true at short context.
| Context | FP16 total | INT8 total | INT4 total |
|---|---|---|---|
| 4K (short chat) | 54 + 1.4 = 55.4 GB | 27 + 0.7 = 27.7 GB | 14 + 0.7 = 14.2 GB |
| 32K (typical RAG) | 54 + 11.5 = 65.5 GB | 27 + 5.8 = 32.8 GB | 14 + 5.8 = 19.3 GB |
KV bytes per token = 2 × 46 layers × 16 KV heads × 128 head-dim × precision bytes. INT4 rows keep the KV cache at INT8 (typical serving default).
Quality benchmarks
Vendor-reported scores from the release blog or tech report. Every row cites its source.
| Benchmark | Score | Source | Date |
|---|---|---|---|
| MMLU (5-shot) | 75.2 | Google Gemma 2 tech report | 2024-06-27 |
| HumanEval | 51.8 | Google Gemma 2 tech report | 2024-06-27 |
| MATH (0-shot) | 42.3 | Google Gemma 2 tech report | 2024-06-27 |
| GPQA | 25.3 | Google Gemma 2 tech report | 2024-06-27 |
Speed and cost to serve
What cloud providers currently charge to serve Gemma 2 27B. Tokens per second is what the provider publishes on their own page. Batch size, prompt length, and inference engine can shift these numbers a lot; treat as directional.
| Provider | Hardware | Speed | $/M in | $/M out | As of |
|---|---|---|---|---|---|
| Together AI | H100 | — | $0.80 | $0.80 | 2025-08-01 |
| Google Vertex AI | TPU v5e | — | $0.35 | $0.53 | 2025-08-01 |
Cheapest option in this list for output-heavy workloads: Google Vertex AI at $0.53/M out. Fastest published: not enough data.
Serving scale on rack designs
On a rack-scale AI system, Gemma 2 27B isn't a fit question: everything fits many times over. The real number is how many independent replicas the rack can host at typical context (32K, INT8). Ceiling: ignores rack overhead, so real deployments hit ~60-80% of this.
| Rack design | Total memory | Replicas (INT8, 32K) | Per replica |
|---|---|---|---|
| Google TPU v5p Pod (8,960 chips) | 851,200 GB | 25,990 | 32.8 GB |
| Huawei Atlas 900 A3 SuperCluster (Ascend 910B) | 524,288 GB | 16,008 | 32.8 GB |
| NVIDIA DGX SuperPOD H100 (32-node reference) | 20,480 GB | 625 | 32.8 GB |
| NVIDIA GB200 NVL72 | 13,824 GB | 422 | 32.8 GB |
| Google TPU v6e (Trillium) Pod-256 | 8,192 GB | 250 | 32.8 GB |
| AWS Trainium 2 UltraServer | 6,144 GB | 187 | 32.8 GB |
Multiply by the per-replica throughput above (Groq / Together / Fireworks tokens/sec) to estimate aggregate throughput. That figure varies by engine (vLLM, TensorRT-LLM, SGLang) and batch policy, so we don't compute it here.
License and variants
- License
- Gemma Terms of Use
- The gotcha
- Custom Google terms with a Prohibited Use Policy that is narrower than Apache 2.0 and updated at Google's discretion.
- Variants
- Base · Instruct (IT)
- Context window
- 8,192 tokens (8K)
- Architecture
- dense
Compare with
Model card: https://huggingface.co/google/gemma-2-27b · Release: Google announcement
See also: every open-source LLM we track · all AI chips · rack designs.