Local LLM giving wrong, weird or repeating answers?
Local models can be excellent, but small or badly configured ones make mistakes that paid assistants don't. Here is what to check.
Sounds like this?
- Made-up facts, names or links
- The same sentence repeated over and over, or gibberish
- Strange tags, or the model's "thinking" shown in the reply
1. Small models make things up
Models under about 8B parameters are fine for simple jobs but confidently invent facts. Don't trust them for anything important, and move up to a bigger model if your hardware allows. Start here picks the biggest model that runs well on your machine.
2. Avoid heavily squeezed versions
Q2 and Q3 downloads save memory but noticeably damage quality. Use Q4_K_M or higher when you can.
3. Repeating or gibberish: use a well-known download
This usually means the model's chat template or settings are wrong, which happens with random uploads. Download models from LM Studio's staff picks, lmstudio-community, or the Ollama library, which are set up correctly. If you changed settings such as temperature, reset them to the defaults.
4. It forgot what we said earlier
Once a conversation is longer than the model's context length, the start drops out of its memory. Start a new chat, or raise the context length if you have spare graphics memory.
5. Thinking text in the answer
Reasoning models write out their working before answering, sometimes between <think> tags. That is normal. LM Studio and Ollama usually fold it away; updating the app fixes it if they don't.
Check what fits first
Most problems come from a model that's too big. See what fits your GPU or Mac, and how fast it will run.
Run it in the cloud
Still too big or too slow? Run it on a rented GPU.
A rented GPU with 24 to 80 GB runs models your computer can't, from well under a dollar an hour.