Local and Private AI Setup: Run a Personal AI With No Cloud (2026)

A fully local personal AI is four pieces: a chat model, an embedding model, a vector store, and a small script that glues them together. None of it requires a cloud account. This guide builds that stack and is honest about where it still loses to a hosted service.

What you actually need

16 GB is the floor because the chat model and your index compete for the same memory. A 7B model quantised to 4-bit needs roughly 4–5 GB; the rest is your OS and the vector store.

Step 1: install the model server

Ollama gives you a local HTTP API on port 11434 and handles model downloads.

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# then pull two models: one to write, one to embed
ollama pull llama3.1:8b
ollama pull nomic-embed-text

On Windows, install the desktop build and the same commands work in PowerShell. Verify with ollama list.

Step 2: embed your notes locally

Two calls matter. Embeddings turn text into vectors; generation turns vectors + question into an answer.

curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "The pricing decision from March was to keep the free tier"
}'

Batch your ingestion: read each file, split into passages of roughly 350 tokens on paragraph boundaries, embed each passage, and store the vector next to the raw text. Keep the filename and passage index — you will need them for citations.

Step 3: store the vectors on disk

Two sane choices:

• sqlite-vec — a SQLite extension. One file, no server, easy to back up, SQL for filtering.
• Chroma — a small local server with a Python client. More features, one more process.

pip install chromadb
python - <<'PY'
import chromadb
client = chromadb.PersistentClient(path="./brain")
collection = client.get_or_create_collection("notes")
print(collection.count())
PY

If you prefer the single-file route, sqlite-vec inside a plain SQLite database is the least moving parts and the easiest thing to encrypt or delete later.

Step 4: wire retrieval into a chat interface

The loop is: embed the question → fetch the top five passages → put them in the prompt with the citation instruction → stream the answer → show which passages were used.

Open WebUI can front Ollama for chat, but you will need a small script for retrieval-augmented answers. The prompt that makes this trustworthy is short, and it is the same one we publish on the templates page: answer only from context, cite passage numbers, and refuse when the answer is absent.

Where local still loses

• Cross-document synthesis. Ask an 8B model to reconcile four contradictory meeting notes and it will flatten the disagreement instead of surfacing it.
• Structured output. Tables and long JSON from local models need more repair than hosted ones.
• Speed on big indexes. Retrieval is fast, but a 20-second generation makes exploratory questioning painful.

The honest framing: local is a privacy decision with a quality tax, not a free upgrade. Most people who love it use it for a sensitive subset — health, legal, client archives — and keep a hosted tool for general research.

Keep it actually private

• Put the vault outside any cloud-synced folder, or accept that your "local" brain is in someone's backup.
• Encrypt the disk. A local index on an unencrypted laptop is one stolen bag away from being public.
• Pin model versions. A silent upgrade can change the embedding model and invalidate your index.
• Test deletion: remove one passage and confirm the vector is gone, not just the file.

Next steps

If chunking is unclear, read how to train AI on your notes — the parameters there apply identically to a local stack. If you are still deciding between local and hosted, the main build guide compares the two on privacy, cost and effort.