Rent a GPU

Llama 3.3 70B hardware requirements

70.6B parameters. Needs about 46.1 GB at Q4_K_M with 8K context.

Parameters71BDense
Q4_K_M · 8K46.1 GBthe usual download
FP16 · 8K144.5 GBfull precision
Runs well on7/51GPUs and Macs we track

Memory needed at each quantization

Weights, KV cache and runtime overhead, at 8K and 32K tokens of context.

QuantQuality8K context32K context
FP16Full quality144.5 GB152.5 GB
Q8_0Near-lossless78.3 GB86.3 GB
Q6_KExcellent61.2 GB69.2 GB
Q5_K_MVery good53.5 GB61.6 GB
Q4_K_MGood, most popular46.1 GB54.1 GB
Q3_K_MNoticeably degraded37.8 GB45.8 GB
Q2_KHeavily degraded29.8 GB37.8 GB

Run it with Ollama

Downloads Q4_K_M by default.

ollama run llama3.3:70b

Model card

meta-llama/Llama-3.3-70B-Instruct on Hugging Face: licence, prompt format and GGUF builds.

Run it in the cloud

Run Llama 3.3 70B on a rented GPU today.

It needs about 46.1 GB at Q4_K_M: a rented RTX A6000 / L40S runs it for about $0.59 to $1.09 an hour.

RunPodRent a GPU on RunPodVast.aiRent a GPU on Vast.ai

Every GPU and Mac checked

Sorted by memory. Bars show how much of each card Llama 3.3 70B fills at the best quant that fits.

7 Runs well

NameMemory usedBest quantSpeed

3 Fits with heavy quantization

NameMemory usedBest quantSpeed

41 Too big

NameMemory usedBest quantSpeed

Other Llama models