You’re paying $20 a month for ChatGPT. Maybe another $20 for Claude Pro. Possibly a third subscription for something else you forgot to cancel. And every single prompt you type gets sent to someone else’s servers, processed on someone else’s hardware, and stored in someone else’s database.
What if you could run AI locally, on a computer sitting in your house? No monthly fees. No internet required. No one reading your prompts. And honestly, for a lot of everyday tasks, the results are surprisingly good.
That’s what Ollama makes possible. It’s a free, open-source tool that lets you download and run large language models locally. If you followed along with the home server guide we published a few weeks back, you already have hardware that can handle this. And if you didn’t, a regular desktop or laptop with decent specs will work just fine. (Wondering how local AI stacks up against just paying for a subscription? The mid-2026 state of self-hosting survey covers that question alongside every other major self-hosting use case.)
This guide walks you through the whole thing, from installation to running your first model to connecting a proper chat interface. No prior AI experience needed.
Why Run AI Locally?
Before we get into the how, let me make the case for the why. Because “it’s cool” is true but not sufficient.
Privacy is the big one. When you use ChatGPT or any cloud AI service, your prompts travel across the internet to a data center. That’s fine for “write me a funny limerick,” but maybe not for “review this contract” or “help me draft this sensitive email.” With a local model, your data never leaves your machine. It doesn’t touch the internet at all.
No subscription fees. Ollama is free. The models are free. Once you have the hardware, your ongoing cost is electricity. That’s it. Given that AI subscriptions can easily run $40-60/month if you’re using multiple services, the math gets compelling fast.
It works offline. On a plane. During an internet outage. In a cabin in the woods. Wherever. The model lives on your disk and runs on your hardware. No connection needed.
You learn how this stuff actually works. Using cloud AI is like driving a car. Running local models is like opening the hood. You start understanding model sizes, quantization, context windows, and GPU acceleration. That knowledge makes you a better user of all AI tools, including the cloud ones.
Nobody can change the rules on you. Cloud providers update their models, adjust their content policies, and sometimes remove capabilities without warning. Your local model stays exactly as it is until you decide to change it.
What Ollama Actually Is
Ollama is a tool that handles the messy parts of running AI models locally. Without it, you’d need to manually download model weights, install Python dependencies, configure GPU drivers, and write scripts to load and serve the model. Ollama wraps all of that into a single command-line tool.
You install Ollama. You tell it which model you want. It downloads the model, optimizes it for your hardware, and serves it through a local API. The whole process takes about five minutes for your first model.
As of July 2026, Ollama has reached 8.9 million active developers per month and just closed a $65M Series B. Over 176,000 GitHub stars. It’s not a niche experiment anymore. It’s become the standard way people run local AI.
Hardware: What You Actually Need
Here’s the honest breakdown. Local AI is more demanding than running a web server or a Docker container, but it’s not as scary as some guides make it sound.
The minimum: Any modern computer with 8GB of RAM can run small models (7 billion parameters). You’ll get usable results for basic tasks like summarization, writing assistance, and simple Q&A. Response speed won’t blow you away, but it works.
The sweet spot: 16-32GB of RAM and a CPU from the last five years. This lets you run mid-size models (13-14B parameters) comfortably, which are genuinely capable for most tasks. If your machine has a dedicated GPU with 8GB+ of VRAM (like an NVIDIA RTX 3060 or better), response times get dramatically faster because the model runs on the GPU instead of the CPU.
Ollama requires NVIDIA GPUs with compute capability 5.0+ and driver version 550 or newer (older Maxwell and Pascal GPUs, GTX 750 Ti through GTX 1080, require driver 570+). AMD GPUs work via ROCm but setup is rougher, mostly on Linux.
The enthusiast setup: 32-64GB of RAM and/or a GPU with 16GB+ VRAM. This opens up larger models (30B+ parameters) that rival cloud AI quality for many tasks. At this level, you’re getting seriously impressive results.
If you built that Linux home server from the earlier guide, those mini PCs (Dell OptiPlex, Lenovo ThinkCentre) work for small models. For a dedicated AI box, you’ll want something with more RAM and ideally a GPU. More on specific hardware recommendations at the end of this post.
A Quick Note on Apple Silicon
If you’re on a Mac with an M1, M2, M3, or M4 chip, you’re in great shape. Apple Silicon’s unified memory architecture means the GPU and CPU share the same RAM pool, and Ollama takes full advantage of it. A MacBook Pro with 16GB of unified memory can run 13B models faster than many desktop setups with dedicated GPUs. Apple Silicon is genuinely one of the best platforms for local AI right now.
Installing Ollama
The installation is almost anticlimactically simple.
Linux
curl -fsSL https://ollama.com/install.sh | shThat’s the whole thing. One command. It installs Ollama, sets it up as a system service, and starts it automatically. If you have an NVIDIA GPU, it detects it and configures CUDA support.
macOS
Download the app from ollama.com/download. Open the DMG, drag to Applications, launch it. Done. It runs as a menu bar app.
Windows
Download the installer from ollama.com/download and run it. Windows support used to be rough, but it’s been solid since late 2025. WSL2 is no longer required.
Verify It’s Working
Open a terminal and run:
ollama --versionIf you get a version number back, you’re good.
Running Your First Model
This is the fun part. Let’s pull down a model and start talking to it.
ollama run llama3.2That single command does three things: downloads the Llama 3.2 model (about 2GB for the 3B version), loads it into memory, and drops you into an interactive chat session. The first run takes a few minutes depending on your internet speed. After that, it’s instant since the model is cached locally.
You’ll see a prompt. Type something. Ask it a question. Tell it to write something. It’s running entirely on your hardware. No internet involved.
To exit the chat, type /bye.
Which Model Should You Start With?
Ollama’s model library has hundreds of options. Here’s where to start:
Phi-4-mini (3.8B) is the new ultra-low-RAM option from Microsoft. At only 2.5GB on disk and 4GB of RAM to run, it works on machines that can’t meet the 8GB minimum for Llama 3.2. Surprisingly capable at math, logic, and multilingual tasks for its size. Pull it with ollama pull phi4-mini. If you’re on a laptop with 8GB of RAM total or a Raspberry Pi 5, start here instead of Llama 3.2.
Llama 3.2 (3B) is the best entry point. Small enough to run on almost anything, capable enough to be useful. Good for quick questions, basic writing help, and getting a feel for how local AI works.
Llama 3.1 (8B) is the step up. Noticeably smarter, handles more complex tasks, and still runs comfortably on 16GB of RAM. This is probably where most people will settle for daily use.
Mistral (7B) is another strong option at the 7B size. Some users prefer its writing style over Llama’s. Worth trying both and seeing which clicks for you.
Gemma 4 (12B) from Google landed in April 2026 and is a substantial upgrade over Gemma 2. The 12B version is 7.6GB on disk, runs on 16GB of RAM, adds native multimodal support (text and images), and outperforms models twice its size on instruction-following and coding benchmarks. Pull it with ollama run gemma4:12b.
Qwen 3 (8B/14B) is the 2026 successor to Qwen 2.5 and a meaningful step up. The 8B version (5.2GB, 16GB RAM) is the easy entry point via ollama run qwen3:8b; the 14B version (9.3GB, 32GB RAM) gets close to cloud-tier quality on writing and reasoning tasks.
DeepSeek-R1 (various sizes) is worth looking at if you need strong reasoning and code generation. The distilled versions run locally and handle technical tasks well.
gpt-oss (20B) is OpenAI’s open-weight model, released under Apache 2.0. Pull it with ollama pull gpt-oss:20b. Needs 16GB of RAM and about 14GB of disk space. Strong at agentic tasks and function calling. Early adopters have run into pull errors intermittently, so if the initial pull fails, check Ollama’s GitHub issues for the current status before troubleshooting locally.
Qwen 3 (30B MoE) is the step up from the 8B-14B Qwen 3 entries above. The 30B mixture-of-experts version is 19GB on disk and runs on 24GB of RAM. Pull it with ollama pull qwen3:30b. The headline feature is the 256,000-token context window, if you’re feeding it long documents, this is the local model that competes with cloud APIs on context length alone.
Llama 4 Scout (April 2026) is Meta’s latest open model and the most capable option available locally right now. It’s a mixture-of-experts model with 17 billion active parameters and a 10-million-token context window. Hardware caveat: the model file is 67GB and you need at least 64GB of Apple Silicon unified memory (or ~60GB+ of VRAM across multiple GPUs). This one is genuinely enthusiast-only hardware. If your setup qualifies, ollama run llama4:scout is where the local-AI ceiling currently sits. If you want an independent take on which models the r/LocalLLaMA community has actually rallied behind for different hardware setups, the synthesis of 311 r/LocalLLaMA replies is worth reading before you commit.
| Model | Size (B params) | Disk Size | Min RAM (Q4_K_M) | Best For |
|---|---|---|---|---|
| Phi-4-mini (Microsoft) | 3.8B | ~2.5 GB | 4 GB | Lowest-RAM option; fast reasoning, math, multilingual |
| Llama 3.2 | 3B | ~2 GB | 8 GB | Entry point, quick Q&A |
| Llama 3.1 | 8B | ~4.9 GB | 16 GB | Daily driver, balanced |
| Mistral | 7B | ~4.4 GB | 16 GB | Writing style alternative |
| Gemma 4 | 12B | ~7.6 GB | 16 GB | Multimodal, instruction-following |
| Qwen 3 | 8B-14B | ~5.2-9.3 GB | 16-32 GB | Near cloud-tier quality |
| DeepSeek-R1 (distilled) | 7B-14B | ~4.7-9 GB | 16-32 GB | Reasoning and code |
| gpt-oss (OpenAI open-weight) | 20B | ~14 GB | 16 GB | Agentic tasks, function calling; ChatGPT-comparable output |
| Qwen 3 (30B MoE) | 30B | ~19 GB | 24 GB | Best overall for 24+ GB RAM; 256K context, strong reasoning |
| Llama 4 Scout | 17B active (MoE) | ~67 GB | 64 GB+ unified mem. (Apple Silicon) or 60 GB+ VRAM | Best local quality (enthusiast) |
To pull a model without starting a chat:
ollama pull llama3.1To see what you have installed:
ollama listTo remove a model you’re done with:
ollama rm llama3.2Adding a Web Interface (Open WebUI)
The terminal chat works, but let’s be real: most people want something that looks like ChatGPT. That’s where Open WebUI comes in.
Open WebUI is a free, self-hosted web interface that connects to your local Ollama instance. It gives you a familiar chat interface with conversation history, model switching, file uploads, and even image generation if you configure it. It runs as a Docker container, which makes setup trivial.
If you already have Docker installed (and you do if you followed the home server guide):
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:mainOpen your browser to http://localhost:3000, create an account (it’s local only, this isn’t going to the cloud), and you’ll see all your Ollama models ready to use.
The interface is polished. You can switch between models mid-conversation, upload documents for analysis, adjust parameters like temperature and context length, and organize chats into folders. It feels like a premium product, and it’s completely free.
For machines on your home network, replace localhost with your server’s IP address and you can access it from any device.
Want to take it further? My full guide to setting up Open WebUI and RAG on top of Ollama shows how to connect your own documents so you can ask questions against your files, not just chat with a base model.
The Ollama API: For the Tinkerers
Ollama exposes a local REST API on port 11434. This is what makes it genuinely useful beyond just chatting. Any application that supports the OpenAI API format can usually be pointed at Ollama with minimal changes, because Ollama offers an OpenAI-compatible endpoint.
curl http://localhost:11434/api/generate -d '{"model": "llama3.1", "prompt": "Explain Docker in two sentences."}'If you’ve been following our series on using Claude Code to automate this blog, you can probably see where this gets interesting. Local models can handle certain tasks in automation pipelines where you don’t want to send data to an external API, or where you want to avoid per-token costs on high-volume, lower-complexity tasks.
The API supports streaming responses, conversation context, model management, and embeddings. Full documentation lives at Ollama’s GitHub.
Practical Use Cases (What This Is Actually Good For)
Let me set expectations honestly. Local models, especially the smaller ones, are not going to replace GPT-4 or Claude for complex reasoning tasks. But they’re genuinely useful for plenty of everyday work.
Writing assistance. Drafting emails, rewording paragraphs, brainstorming ideas, generating outlines. A 7-8B model handles this well.
Code help. Explaining code, generating boilerplate, debugging simple issues. DeepSeek-R1 and CodeLlama are solid picks here.
Summarization. Feed it a long document and get a concise summary. Works great with the right context window settings.
Private document analysis. Tax documents, medical records, legal contracts. Anything you don’t want leaving your network.
Learning and experimentation. Understanding how different model sizes and architectures behave. Trying fine-tuning. Building local RAG (retrieval-augmented generation) pipelines.
Offline access. Traveling without reliable internet? Your AI assistant travels with you.
Where local models still fall short: complex multi-step reasoning, very long document processing (context windows are smaller than cloud models), and tasks that benefit from the massive training compute behind frontier models. Use the right tool for the job.
Tips for Getting the Most Out of Local AI
A few things that aren’t obvious when you’re starting out.
Model size vs. quality isn’t linear. A well-made 8B model often outperforms a mediocre 13B model. Pay attention to benchmarks like the HuggingFace Open LLM Leaderboard and community feedback, not just parameter counts.
Quantization matters. Most models on Ollama are quantized (compressed) to reduce memory usage. The default Q4 quantization is a good balance of quality and speed. If you have extra RAM, look for Q5 or Q6 versions for slightly better output.
Context window is king. The context window determines how much text the model can consider at once. Ollama defaults to 2048 tokens for most models. For longer conversations or document analysis, increase it:
ollama run llama3.1 --num-ctx 8192More context uses more memory, so find the balance that works for your hardware.
Keep your models updated. Model creators release improvements regularly. Run ollama pull modelname periodically to grab the latest version.
GPU offloading is the single biggest speed improvement. If you have an NVIDIA GPU, make sure Ollama is using it. You can check with:
ollama psThis shows which models are loaded and whether they’re using GPU or CPU.
What You’ll Need (Gear Recommendations)
If you’re setting up a dedicated machine for local AI (or upgrading an existing one), here’s what to look at.
Mini PCs for a Dedicated AI Box
These are great if you want a small, quiet machine that sits on a shelf and runs models for your whole household.
Beelink SER5 Max (AMD Ryzen 7 5800H, 32GB RAM) ($280-350) The Ryzen 7 5800H is an 8-core chip with solid integrated graphics. Paired with 32GB of RAM, this handles 7-13B models comfortably. Quiet, compact, and energy efficient. A strong pick for a local AI server that doubles as a general-purpose home server.
Beelink SER7 (AMD Ryzen 7 7840HS, 32GB RAM) ($350-450) The newer Ryzen 7 7840HS includes a Radeon 780M iGPU with ROCm support, which means Ollama can actually offload to the integrated GPU for meaningful speedups. 32GB of DDR5 RAM gives you room for larger models. If you’re buying new specifically for AI workloads, this is the one to get.
ASUS NUC 14 Pro ($400-550) Intel exited the NUC business in July 2023 and sold the product line to ASUS. The ASUS NUC 14 Pro carries on the same compact NUC form factor with strong single-threaded performance, Thunderbolt 4 for eGPU expansion later, and solid Linux support. No meaningful GPU offloading without an external GPU, but CPU inference is respectable for models up to 13B. (If you find an Intel NUC 13 Pro on the used or refurb market, it will work, but verify current pricing before buying.)
RAM Upgrade Kits
If you already have a mini PC or laptop and just need more memory, DDR5 SODIMM kits are the move.
Crucial 32GB DDR5-4800 SODIMM Kit (2x16GB) ($70-90) Compatible with most DDR5 laptops and mini PCs. Going from 16GB to 32GB is often the single cheapest upgrade that opens up a whole new tier of models.
Crucial 64GB DDR5-5600 SODIMM Kit (2x32GB) ($150-200) For running 30B+ parameter models or keeping multiple models loaded simultaneously. Overkill for most people, but if you catch the local AI bug, you’ll get here eventually.
External SSDs for Model Storage
AI models eat disk space. Llama 3.1 8B is about 4.7GB. Larger models can be 20-40GB each. An external SSD keeps your boot drive from filling up.
Samsung T7 1TB Portable SSD ($80-100) Fast USB 3.2 Gen 2 speeds (1,050 MB/s), compact, and reliable. 1TB holds a solid collection of models with room to spare.
SanDisk Extreme 2TB Portable SSD ($120-150) If you want more breathing room. The 2TB model is sweet spot pricing for storage-per-dollar and gives you space to experiment with lots of models without worrying about cleanup.
To tell Ollama to store models on an external drive, set the OLLAMA_MODELS environment variable:
export OLLAMA_MODELS=/mnt/external-ssd/ollama-modelsAdd that to your .bashrc or .zshrc to make it permanent.
Sources
- Ollama Official Documentation
- Ollama GitHub Repository
- Open WebUI Documentation
- Run Your Own AI Model Locally: A Practical Ollama Setup Guide 2026 (DEV.to)
- Local AI in 2026: Running Production LLMs on Your Own Hardware with Ollama (DEV.to)
- Run AI Models Locally: Offline Privacy Guide (AI Thinker Lab)
- Local AI Guide (yuv.ai)
- Ollama Model Library
- Open WebUI GitHub Repository
- Ollama raises $65M, grows to nearly 9M active developers (TechCrunch, July 2026)
Running local AI already, or thinking about giving it a shot? I want to hear what models you’re running and what you’re using them for. Drop a comment below. And if you set this up after reading this guide, come back and tell me how it went. I’m especially curious what hardware you’re running it on and whether the speed is good enough for your workflow. 🖥️
Pingback: Open WebUI Ollama: RAG Setup No Guide Explains (2026)