Why run models locally?
Nothing leaves your machine, it works offline, and there's no per-token bill. The trade-off is that small local models are weaker than the big hosted ones, so use them for private drafting, summarising and classification rather than your hardest reasoning.
Install and run
Download Ollama from ollama.com for macOS, Windows or Linux, then in a terminal:
bash
ollama pull llama3.2 # download a small general model
ollama run llama3.2 # chat in the terminal (/bye to exit)
ollama list # see what's installed
ollama rm llama3.2 # free up disk spaceTip: Rough sizing: a 4-bit quantised 7–8B model needs about 5–6GB of memory. Start small and step up if the answers are too weak.
Call it from code
Ollama serves a local HTTP API on port 11434 while it's running.
bash
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"messages": [{ "role": "user", "content": "Summarise GDPR in 3 bullets" }],
"stream": false
}'Customise with a Modelfile
A Modelfile bakes in a system prompt and settings so you can reuse a configured assistant by name.
text
FROM llama3.2
SYSTEM "You are a careful UK English proofreader. Return the corrected text only."
PARAMETER temperature 0.2
# then: ollama create proofreader -f Modelfile && ollama run proofreader