Can the NVIDIA RTX 4060 run Mixtral 8x7B?
No
No. Even at Q4_K_M, Mixtral 8x7B needs about 30 GB, but the NVIDIA RTX 4060 has 8 GB usable. You could offload some layers to system RAM, but it will be very slow.
Try other context lengths in the calculator →Run it in the cloud
Too big for the NVIDIA RTX 4060? Run Mixtral 8x7B on a rented GPU instead.
Rent a cloud GPU by the hour. No upfront cost, cancel any time.
Memory needed by quantization
At 8,192 tokens of context.
| Quant | Quality | Memory | Fits? | Speed |
|---|---|---|---|---|
| FP16 | Full quality | 95.1 GB | Too big | — |
| Q8_0 | Near-lossless | 51.3 GB | Too big | — |
| Q6_K | Excellent | 40 GB | Too big | — |
| Q5_K_M | Very good | 34.9 GB | Too big | — |
| Q4_K_M | Good, most popular | 30 GB | Too big | — |
| Q3_K_M | Noticeably degraded | 24.5 GB | Too big | — |
| Q2_K | Heavily degraded | 19.2 GB | Too big | — |
How to run Mixtral 8x7B anyway
On a rented cloud GPU with enough memory, the same command you'd use locally works:
ollama run mixtral:8x7bAt Q4_K_M it needs about 30 GB, so pick a GPU with at least 32 GB.
GPUs that run Mixtral 8x7B comfortably
- NVIDIA RTX 5090 (32 GB)
- Apple M4 Pro (48GB)
- Apple M3 Max (64GB)
- Apple M4 Max (128GB)
- Apple M2 Ultra (192GB)