Local LLM "out of memory" or won't load?

Sounds like this?

• "Failed to load model" in LM Studio
• "CUDA error: out of memory" or "model requires more system memory" in Ollama
• The app freezes or the computer slows to a crawl while loading

1. Pick a smaller download

Every model comes in several sizes ("quantizations"). Q4_K_M is the usual sweet spot. If you downloaded Q8 or Q6, try Q4_K_M instead. If Q4_K_M still doesn't fit, try a model with fewer parameters (the "B" number: 8B is smaller than 14B). The calculator shows exactly what fits your GPU.

2. Lower the context length

Context is how much text the model can keep in mind at once, and it uses memory on top of the model itself. If you raised it, lower it again to 4,096 or 8,192 tokens.

• LM Studio: when you load a model, open the load settings and reduce Context Length.
• Ollama: inside a chat, type /set parameter num_ctx 4096.

3. Close other apps that use the GPU

Games, video editors, and browsers with many tabs all use graphics memory. On Windows, open Task Manager, go to Performance > GPU and look at Dedicated GPU memory to see how much is already in use.

Also unload models you are no longer using: ollama ps lists loaded models and ollama stop MODEL unloads one. In LM Studio, click Eject next to the loaded model.

4. Let part of the model run on the CPU

In LM Studio's load settings, move the GPU Offload slider down so some layers run from system RAM. The model will load, but it will be noticeably slower. Ollama does this automatically when a model is too big.

5. On a Mac: give the GPU more memory (advanced)

macOS lets the GPU use about two thirds to three quarters of unified memory. You can raise that limit until the next restart with sudo sysctl iogpu.wired_limit_mb=NUMBER (for example 24576 for 24 GB on a 32 GB Mac). Leave several GB for macOS itself, or the whole system can freeze.