Can the NVIDIA RTX 3080 10GB run Mistral Small 24B?

No

No. Even at Q4_K_M, Mistral Small 24B needs about 16.2 GB, but the NVIDIA RTX 3080 10GB has 10 GB usable. You could offload some layers to system RAM, but it will be very slow.

Try other context lengths in the calculator →
Run it in the cloud

Too big for the NVIDIA RTX 3080 10GB? Run Mistral Small 24B on a rented GPU instead.

Rent a cloud GPU by the hour. No upfront cost, cancel any time.

RunPodOn-demand GPUs with ready-made templates for Ollama and vLLM.Rent a GPU on RunPod → Vast.aiA marketplace of rentable GPUs, from RTX cards to H100s.Rent a GPU on Vast.ai →

Memory needed by quantization

At 8,192 tokens of context.

QuantQualityMemoryFits?Speed
FP16Full quality49.1 GBToo big—
Q8_0Near-lossless27 GBToo big—
Q6_KExcellent21.3 GBToo big—
Q5_K_MVery good18.7 GBToo big—
Q4_K_MGood, most popular16.2 GBToo big—
Q3_K_MNoticeably degraded13.5 GBToo big—
Q2_KHeavily degraded10.8 GBToo big—

How to run Mistral Small 24B anyway

On a rented cloud GPU with enough memory, the same command you'd use locally works:

ollama run mistral-small:24b

At Q4_K_M it needs about 16.2 GB, so pick a GPU with at least 24 GB.

GPUs that run Mistral Small 24B comfortably

Check NVIDIA RTX 3090 prices →

Models that run well on the NVIDIA RTX 3080 10GB