AI/ML Ops

How to Run Local LLMs with Ollama and Open WebUI on Your Homelab (Complete Guide)

If you've ever wanted to run AI models locally without sending your data to the cloud, this guide is for you. By the end, you'll have a production-ready local AI stack running on your homelab with a polished web interface, GPU acceleration, and enterprise-grade security.

Word count: ~2,663 Reading time: 13 min Category: AI/ML Ops, Self-hosting, Docker, Homelab, Linux, Automation Last reviewed: August 2026

Guide Summary

This complete guide walks you through deploying a local LLM stack using Ollama as the model runtime and Open WebUI as the chat interface. Covers hardware selection, Docker Compose deployment with GPU support, model management, RAG configuration, reverse proxy with TLS, security hardening, backups, and monitoring. Tested on Ubuntu 22.04/24.04, Proxmox VE 8.x, Docker 27.x, Ollama 0.3.x, Open WebUI 0.3.x.

Why Run LLMs Locally on Your Homelab?

Privacy and Data Sovereignty

When you chat with cloud AI services, every prompt, document upload, and conversation travels over the internet to someone else's servers. For personal projects this might be fine, but for sensitive work — financial data, medical records, proprietary code, or private conversations — local inference keeps everything on your hardware. Your data never leaves your network.

No API Costs or Rate Limits

Cloud APIs charge per token. A serious coding assistant workflow can easily burn through $50-100/month in API costs. Local models are free after the initial hardware investment. No rate limits, no billing surprises, no "you've exceeded your quota" messages during critical work.

Offline Capability

Your homelab doesn't need internet access to run inference. Whether you're on a plane, in a secure facility, or your ISP goes down, your AI assistant keeps working. This is critical for air-gapped environments and disaster recovery scenarios.

Custom Model Fine-Tuning

Want a model that speaks your company's internal DSL? Need a model trained on your specific documentation? Local inference lets you fine-tune models with LoRA adapters, create custom Modelfiles, and experiment without asking permission or paying for fine-tuning APIs.

Learning and Experimentation

Running models locally teaches you how they actually work — tokenization, quantization, context windows, KV caches, GPU memory management. This knowledge transfers directly to cloud deployments and makes you a more effective AI engineer.

Prerequisites: Hardware and Software Requirements

Minimum Hardware Specs

Component	Minimum	Recommended	Notes
CPU	4 cores (AVX2)	8+ cores (AVX-512)	Apple Silicon uses unified memory
RAM	16 GB	32-64 GB	Model weights + KV cache + OS
GPU	None (CPU-only)	NVIDIA 12 GB+ VRAM	See GPU section below
Storage	50 GB free	200+ GB NVMe	Models are 2-50 GB each

GPU Options

NVIDIA (Best Support)

• RTX 3060 12 GB → Excellent entry point (~$300 used)
• RTX 3090/4090 24 GB → Runs 70B models at 4-bit
• RTX 6000 Ada 48 GB → Runs 70B at 8-bit, 120B at 4-bit
• Data center: A100 40/80 GB, H100 80 GB

AMD (Improving via ROCm)

• RX 7900 XTX 24 GB → Good value, ROCm 6.0+ support
• MI300X 192 GB → Enterprise grade

Apple Silicon (Unified Memory)

• M1/M2/M3 Max/Ultra 64-192 GB → Best price/performance for large models
• No separate VRAM — system RAM is shared

Software Prerequisites

# Ubuntu/Debian/Proxmox VE sudo apt update && sudo apt install -y docker.io docker-compose-plugin git curl # Verify Docker works without sudo sudo usermod -aG docker $USER newgrp docker docker run --rm hello-world

Network Considerations

• Ollama API defaults to 127.0.0.1:11434 (localhost only)
• Open WebUI defaults to port 3000
• For remote access, use a reverse proxy (Traefik/Nginx/Caddy) with TLS — never expose ports directly

Installing Ollama: The Model Runtime

Method 1: Native Installation

# Linux (single command) curl -fsSL https://ollama.com/install.sh | sh # macOS brew install ollama # Windows winget install Ollama.Ollama

Start the service:

# Linux (systemd) sudo systemctl enable --now ollama # macOS (launchd) brew services start ollama

Method 2: Docker Container (Recommended for Homelabs)

# CPU-only docker run -d \ --name ollama \ --restart unless-stopped \ -p 11434:11434 \ -v ollama:/root/.ollama \ ollama/ollama:latest # With NVIDIA GPU (requires nvidia-container-toolkit) docker run -d \ --name ollama \ --restart unless-stopped \ --gpus all \ -p 11434:11434 \ -v ollama:/root/.ollama \ ollama/ollama:latest

Method 3: Docker Compose with GPU Support

Create docker-compose.yml:

version: '3.8' services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] environment: - OLLAMA_HOST=0.0.0.0 - OLLAMA_ORIGINS=* - OLLAMA_MAX_LOADED_MODELS=3 - OLLAMA_NUM_PARALLEL=2 open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" volumes: - open-webui:/app/backend/data environment: - OLLAMA_BASE_URL=http://ollama:11434 - WEBUI_SECRET_KEY=your-secret-key-change-this - ENABLE_SIGNUP=true - DEFAULT_MODELS=llama3.1:8b-instruct-q4_k_m depends_on: - ollama volumes: ollama: open-webui:

Deploy:

docker compose up -d docker compose logs -f

Verifying Ollama Is Running

# Health check curl http://localhost:11434/api/tags # Should return: {"models":[]}

Basic Ollama Commands Cheat Sheet

Command	Purpose
ollama list	List downloaded models
ollama pull llama3.1:8b-instruct-q4_k_m	Download a model
ollama run llama3.1:8b-instruct-q4_k_m	Interactive chat
ollama rm model-name	Remove a model
ollama show model-name	Model details (Modelfile, template)
ollama ps	Currently loaded models
ollama serve	Start server manually (foreground)

Pulling and Managing Models

Understanding Model Naming and Tags

Format: model-name:tag

• Model name: llama3.1, mistral, qwen2.5, phi3, codellama
• Size suffix: :8b, :70b, :120b (parameters in billions)
• Quantization tag: :q4_k_m, :q8_0, :fp16, :q4_0

Quantization Explained

Quantization	Size (8B)	Quality	Speed	VRAM (8B)
fp16 / bf16	~16 GB	Best	Slow	16 GB
q8_0	~9 GB	Near-fp16	Fast	9 GB
q6_k	~7 GB	Excellent	Fast	7 GB
q5_k_m	~5.5 GB	Very good	Fast	5.5 GB
q4_k_m	~4.5 GB	Great	Fastest	4.5 GB
q4_0	~4 GB	Good	Fastest	4 GB
q3_k_m	~3.5 GB	Decent	Fastest	3.5 GB

Recommendation: Start with q4_k_m — best balance of quality, speed, and memory.

Popular Models for Different Use Cases

# General chat / reasoning (start here) ollama pull llama3.1:8b-instruct-q4_k_m ollama pull qwen2.5:7b-instruct-q4_k_m # Coding ollama pull codellama:7b-instruct-q4_k_m ollama pull qwen2.5-coder:7b-instruct-q4_k_m # Reasoning / Math ollama pull deepseek-r1:8b-q4_k_m ollama pull phi3:14b-q4_k_m # Multilingual ollama pull aya-expanse:8b-q4_k_m # Large models (need 24+ GB VRAM) ollama pull llama3.1:70b-instruct-q4_k_m ollama pull qwen2.5:72b-instruct-q4_k_m

Model Management Commands

# List installed models with sizes ollama list # Show model details (template, parameters, license) ollama show llama3.1:8b-instruct-q4_k_m # Remove unused models ollama rm old-model-name # Update a model to latest tag ollama pull llama3.1:8b-instruct-q4_k_m # Run with custom parameters ollama run llama3.1:8b-instruct-q4_k_m --temperature 0.7 --num_ctx 8192

Custom Modelfiles for Fine-Tuning

Create Modelfile:

FROM llama3.1:8b-instruct-q4_k_m # System prompt SYSTEM """You are a senior DevOps engineer specializing in Kubernetes, Terraform, and GitOps. Provide concise, production-ready answers. Always include error handling and validation steps.""" # Parameters PARAMETER temperature 0.3 PARAMETER num_ctx 8192 PARAMETER num_predict 2048 PARAMETER stop "<|eot_id|>" PARAMETER stop "<|end_of_text|>" # Template (usually inherited from base) TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|> {{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|> {{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|> {{ .Response }}<|eot_id|>"""

Build and use:

ollama create devops-assistant -f Modelfile ollama run devops-assistant

Setting Up Open WebUI: The Chat Interface

Why Open WebUI Over Other Interfaces?

Feature	Open WebUI	LibreChat	AnythingLLM	Chatbox
Self-hosted	Yes	Yes	Yes	No (cloud sync)
RAG (Documents)	Native	Yes	Yes	Limited
Pipelines/Tools	Native	No	Yes	No
Multi-user/Auth	Built-in	Yes	Yes	No
Model Management	From UI	Yes	No	Manual
Code Execution	Pipelines	Yes	Yes	No
Mobile PWA	Yes	Yes	Yes	No
Active Development	Very active	Active	Active	Slow

Open WebUI is the most feature-complete, actively maintained option with native Ollama integration.

Docker Compose Setup with Ollama

Use the compose file from Method 3 above — it includes both services with proper networking.

Key environment variables for Open WebUI:

environment: - OLLAMA_BASE_URL=http://ollama:11434 # Internal Docker network - WEBUI_SECRET_KEY=generate-with-openssl-rand-base64-32 - ENABLE_SIGNUP=true # Set false after admin created - DEFAULT_MODELS=llama3.1:8b-instruct-q4_k_m - ENABLE_RAG=true - RAG_TEMPLATE=... - ENABLE_WEB_SEARCH=true - WEB_SEARCH_ENGINE=duckduckgo - FILES_UPLOAD_MAX_SIZE=50 - AUDIO_TTS_ENGINE=elevenlabs - AUDIO_STT_ENGINE=openai

Generate a secure secret:

openssl rand -base64 32

Configuration Options

Volumes persist:

• /app/backend/data — SQLite database, uploaded files, vector DB, user settings

Key settings (adjust via UI after first login):

• Admin Panel → Settings → General — Site title, default model, signup
• Admin Panel → Settings → Ollama — Multiple Ollama endpoints
• Admin Panel → Settings → RAG — Chunk size, overlap, embedding model
• Admin Panel → Settings → Web Search — Engine, API keys

Enabling GPU Acceleration in Open WebUI

Open WebUI itself doesn't need GPU — it's a frontend. Ollama does the inference. Ensure Ollama container has GPU access (see Docker Compose deploy.resources.reservations.devices).

Verify GPU works:

# Inside Ollama container docker exec ollama nvidia-smi # Or check logs for "GPU" detection docker compose logs ollama | grep -i gpu

First Login and Admin Setup

1. Open http://your-homelab-ip:3000
2. First account created = Admin — use a strong password
3. Immediately: Admin Panel → Settings → General → Disable Signup
4. Configure your preferences in Settings (gear icon)

Key Features Walkthrough

RAG (Retrieval-Augmented Generation)

• Upload PDFs, Markdown, text files in chat via + button
• Documents are chunked, embedded, stored in local vector DB (ChromaDB)
• Queries retrieve relevant chunks → injected into context

Pipelines (Web Search, Tools)

• Admin Panel → Pipelines → Enable "Web Search"
• Add API keys for Serper, Tavily, or use DuckDuckGo (free)
• Model can now search the web during conversation

Model Management (from UI)

• Admin Panel → Models → Pull/Delete/Tag models
• See model sizes, parameters, last used
• Set default model per user

Connecting Open WebUI to Ollama

Default Connection (Same Host)

With Docker Compose, services communicate via service names:

• Open WebUI → http://ollama:11434 (Docker DNS)
• No configuration needed — works out of the box

Connecting to Remote Ollama Instance

If Ollama runs on a different machine:

# In open-webui service environment - OLLAMA_BASE_URL=http://192.168.1.50:11434 # Remote Ollama IP

On the remote Ollama host, bind to all interfaces:

# Native install: edit /etc/systemd/system/ollama.service Environment="OLLAMA_HOST=0.0.0.0" # Then: sudo systemctl daemon-reload && sudo systemctl restart ollama # Docker: already handled by OLLAMA_HOST=0.0.0.0

Troubleshooting Connection Issues

Symptom	Cause	Fix
"Connection refused"	Ollama not running / wrong URL	Check docker compose ps, verify OLLAMA_BASE_URL
"Model not found"	Model not pulled on that Ollama	ollama pull model-name on the Ollama host
Slow first response	Model loading into VRAM	First load is slow; subsequent are fast
CORS errors	Browser blocking	Set OLLAMA_ORIGINS=* on Ollama

Configuring Multiple Ollama Backends

Open WebUI supports multiple Ollama endpoints (Admin → Settings → Ollama → Add Connection). Useful for:

• GPU server for large models + CPU server for small models
• Different model collections per team
• Failover between instances

Model Sync Between Ollama and Open WebUI

Models pulled via CLI (ollama pull) appear automatically in Open WebUI. Models pulled via Open WebUI UI are stored in the same Ollama volume. They're the same model store.

Optimizing Performance: GPU Acceleration and Tuning

NVIDIA GPU Setup with nvidia-container-toolkit

On host (Ubuntu/Proxmox/Debian):

# Add NVIDIA container toolkit repo curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \ sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt update && sudo apt install -y nvidia-container-toolkit # Configure Docker sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker # Test docker run --rm --gpus all nvidia/cuda:12.4-base-ubuntu22.04 nvidia-smi

In docker-compose.yml (already shown in Method 3):

deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu]

AMD ROCm Support

# Requires ROCm 6.0+ and supported GPU (RX 7000 series, MI200/300) # Ollama ROCm images: ollama/ollama:rocm # In docker-compose.yml: image: ollama/ollama:rocm # Remove nvidia deploy section; add: devices: - /dev/kfd - /dev/dri group_add: - video - render

Apple Metal (MPS) on macOS

Native Ollama on macOS uses Metal automatically. No Docker needed — native install is faster on Apple Silicon.

# Verify Metal is used ollama run llama3.1:8b-instruct-q4_k_m "hello" # Check Activity Monitor → GPU tab → ollama process

CPU-Only Optimization Tips

If no GPU, optimize for CPU inference:

# In docker-compose.yml for ollama service: environment: - OLLAMA_NUM_PARALLEL=1 # One model at a time - OLLAMA_MAX_LOADED_MODELS=1 # Keep one in RAM - OLLAMA_FLASH_ATTENTION=1 # Enable flash attention - OLLAMA_KV_CACHE_TYPE=f16 # Half-precision KV cache

Model selection for CPU:

• Smaller models: phi3:3.8b, gemma2:2b, qwen2.5:1.5b
• 4-bit quantization essential (q4_k_m or q3_k_m)
• Expect 2-10 tokens/second depending on CPU

Context Window and Batch Size Tuning

# Increase context window (default 2048-4096) ollama run llama3.1:8b-instruct-q4_k_m --num_ctx 8192 # In Modelfile for persistent setting: PARAMETER num_ctx 8192 PARAMETER num_batch 512

Trade-offs:

• Larger context → more VRAM/RAM, slower first token
• Larger batch → faster prompt processing, more VRAM

Monitoring GPU Usage

# Real-time GPU monitoring watch -n 1 nvidia-smi # Or use nvtop (better UI) docker run --rm -it --gpus all -v /proc:/host/proc:ro -v /sys:/host/sys:ro ghcr.io/sachaos/nvtop # For AMD rocm-smi

Advanced Features: RAG, Pipelines, and More

Setting Up RAG (Retrieval-Augmented Generation)

In Open WebUI:

1. Click + in chat → Upload document (PDF, MD, TXT, DOCX)
2. Document is processed → chunked → embedded → stored
3. Chat with the document: "Summarize this PDF" or "What does section 3 say?"

Configuration (Admin → Settings → RAG):

• Embedding Model: nomic-embed-text (pull first: ollama pull nomic-embed-text)
• Chunk Size: 500-1000 tokens
• Chunk Overlap: 100-200 tokens
• Top K Results: 3-5

For better RAG, use a dedicated embedding model:

ollama pull nomic-embed-text ollama pull mxbai-embed-large

Document Ingestion and Vector Databases

Open WebUI uses ChromaDB (embedded SQLite) by default. For production scale, configure external ChromaDB or Qdrant:

# In docker-compose.yml, add to open-webui service: environment: - CHROMA_HOST=chromadb - CHROMA_PORT=8000 # Add chromadb service: chromadb: image: chromadb/chroma:latest volumes: - chromadb:/chroma/data

Open WebUI Pipelines for Web Search and Tools

Admin Panel → Pipelines → Add Pipeline

1. Web Search (built-in):
   • Engine: DuckDuckGo (free), Serper, Tavily, Google
   • Model decides when to search, retrieves results, synthesizes answer
2. Custom Function Pipelines (Python):
   • Create functions for: API calls, database queries, file ops, shell commands
   • Model calls functions → gets results → continues reasoning

Example pipeline structure:

# pipelines/web_search.py from open_webui.pipelines import Pipeline, PipelineResult class WebSearchPipeline(Pipeline): async def run(self, query: str) -> PipelineResult: # Your search logic here return PipelineResult(content=results)

Custom System Prompts and Model Parameters

Per-model parameters (Admin → Models → Edit):

{ "temperature": 0.7, "top_p": 0.9, "top_k": 40, "num_ctx": 8192, "num_predict": 2048, "repeat_penalty": 1.1, "seed": -1 }

Per-chat system prompt (chat settings → System Prompt):

You are a Kubernetes expert. Always provide: 1. YAML manifests with comments 2. kubectl commands to verify 3. Common troubleshooting steps 4. Links to official docs

User Management and Access Control

Admin Panel → Users:

• Create users, assign roles (Admin, User)
• Set per-user model access
• View usage statistics

For teams:

• Disable public signup
• Create accounts manually
• Use groups for model access control (Enterprise feature)

Backups and Persistence

What to back up:

# Open WebUI data (SQLite, uploads, vector DB) docker compose cp open-webui:/app/backend/data ./backup/open-webui-$(date +%F) # Ollama models docker compose cp ollama:/root/.ollama ./backup/ollama-$(date +%F) # Or back up Docker volumes directly: docker run --rm -v ollama:/source -v $(pwd)/backup:/backup alpine tar czf /backup/ollama.tar.gz -C /source . docker run --rm -v open-webui:/source -v $(pwd)/backup:/backup alpine tar czf /backup/open-webui.tar.gz -C /source .

Automated backup script:

#!/bin/bash # /usr/local/bin/backup-homelab-ai.sh DATE=$(date +%F) BACKUP_DIR="/mnt/backups/homelab-ai" mkdir -p "$BACKUP_DIR" docker run --rm -v ollama:/source -v "$BACKUP_DIR":/backup alpine \ tar czf "/backup/ollama-$DATE.tar.gz" -C /source . docker run --rm -v open-webui:/source -v "$BACKUP_DIR":/backup alpine \ tar czf "/backup/open-webui-$DATE.tar.gz" -C /source . # Keep last 7 days find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete

Add to crontab: 0 2 * * * /usr/local/bin/backup-homelab-ai.sh

Troubleshooting Common Issues

Open WebUI Can't Connect to Ollama

# 1. Check Ollama is running docker compose ps ollama docker compose logs ollama --tail 50 # 2. Test connectivity from Open WebUI container docker exec open-webui wget -qO- http://ollama:11434/api/tags # 3. Check OLLAMA_BASE_URL in Open WebUI env docker compose exec open-webui env | grep OLLAMA # 4. Common fix: Ollama binding to localhost only # Ensure OLLAMA_HOST=0.0.0.0 in ollama service environment

Out of Memory Errors

Error: CUDA out of memory / llama.cpp: failed to allocate

Fixes:

# Use smaller quantization ollama pull llama3.1:8b-instruct-q3_k_m # Reduce context window ollama run model --num_ctx 4096 # Limit loaded models # In docker-compose.yml: OLLAMA_MAX_LOADED_MODELS=1 # Offload layers to CPU (if partially fit) # In Modelfile: PARAMETER num_gpu 35 # Adjust based on VRAM

Slow Inference Speeds

Issue	Fix
CPU-only on large model	Use smaller model (3B-7B) + q4_k_m
GPU not detected	Install nvidia-container-toolkit, check nvidia-smi in container
Swap thrashing	Add more RAM or reduce model size
First token slow	Normal — model loading. Keep model loaded: OLLAMA_KEEP_ALIVE=10m

# Keep model in VRAM longer # In docker-compose.yml: environment: - OLLAMA_KEEP_ALIVE=10m # Default 5m

Model Not Found / Pull Failures

# Check exact model name ollama search llama3.1 # Pull with explicit tag ollama pull llama3.1:8b-instruct-q4_k_m # Registry issues — try mirror OLLAMA_HOST=https://ollama.com ollama pull llama3.1:8b-instruct-q4_k_m # Clear cache and retry rm -rf ~/.ollama/models/manifests/registry.ollama.ai/library/llama3.1 ollama pull llama3.1:8b-instruct-q4_k_m

GPU Not Detected

# 1. Host driver installed? nvidia-smi # 2. nvidia-container-toolkit installed? docker run --rm --gpus all nvidia/cuda:12.4-base-ubuntu22.04 nvidia-smi # 3. Compose file has GPU reservation? # Check deploy.resources.reservations.devices in docker-compose.yml # 4. Ollama logs show GPU? docker compose logs ollama | grep -i "gpu\|cuda\|metal\|rocm"

Docker Permission Issues

# Add user to docker group sudo usermod -aG docker $USER newgrp docker # Or fix volume permissions sudo chown -R 1000:1000 /path/to/ollama/data sudo chown -R 1000:1000 /path/to/open-webui/data

Security Hardening for Production Homelabs

Reverse Proxy with TLS (Nginx/Traefik/Caddy)

Never expose port 3000 directly. Use a reverse proxy with automatic HTTPS.

Caddy (Simplest — Auto HTTPS via Let's Encrypt):

# /etc/caddy/Caddyfile ai.yourdomain.com { reverse_proxy localhost:3000 header { # Security headers Strict-Transport-Security "max-age=31536000" X-Content-Type-Options "nosniff" X-Frame-Options "DENY" Referrer-Policy "strict-origin-when-cross-origin" } }

# Run Caddy docker run -d --name caddy --restart unless-stopped \ --network host \ -v /etc/caddy/Caddyfile:/etc/caddy/Caddyfile \ -v caddy_data:/data -v caddy_config:/config \ caddy:latest

Traefik (If Already Using for Other Services):

# Add to open-webui service labels: labels: - "traefik.enable=true" - "traefik.http.routers.open-webui.rule=Host(`ai.yourdomain.com`)" - "traefik.http.routers.open-webui.tls=true" - "traefik.http.routers.open-webui.tls.certresolver=letsencrypt" - "traefik.http.services.open-webui.loadbalancer.server.port=8080"

Nginx (Manual Certs):

server { listen 443 ssl http2; server_name ai.yourdomain.com; ssl_certificate /etc/letsencrypt/live/ai.yourdomain.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ai.yourdomain.com/privkey.pem; location / { proxy_pass http://localhost:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # WebSocket support proxy_read_timeout 86400; } }

Authentication and Authorization

Open WebUI built-in:

• Admin creates all accounts (disable signup)
• Per-user model permissions
• API keys for programmatic access

Add SSO (OIDC/SAML) — Enterprise:

• Admin Panel → Settings → Authentication
• Configure Keycloak, Authentik, Authelia, Google, GitHub, Microsoft

Network-Level (Defense in Depth):

# Tailscale Funnel for secure remote access (no port forwarding) tailscale funnel 443 # Or Tailscale Serve for internal-only tailscale serve https / http://localhost:3000

Network Isolation with Docker Networks

# In docker-compose.yml networks: ai-internal: driver: bridge internal: true # No outbound internet ai-external: driver: bridge # For reverse proxy only services: ollama: networks: - ai-internal \n open-webui: networks: - ai-internal - ai-external # Only service touching reverse proxy

Rate Limiting and Abuse Prevention

At reverse proxy level (Caddy):

ai.yourdomain.com { rate_limit { zone ai_limit 10r/s key {remote_host} } reverse_proxy localhost:3000 }

At Open WebUI level (Admin → Settings):

• Max message length
• Max file upload size
• Requests per minute per user

Regular Updates and Vulnerability Scanning

# Weekly update routine cd /opt/homelab-ai docker compose pull docker compose up -d --remove-orphans docker image prune -f # Scan images for vulnerabilities docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ aquasec/trivy:latest image ollama/ollama:latest ghcr.io/open-webui/open-webui:main # Or use Trivy in CI/CD pipeline

What's Next: Expanding Your Local AI Stack

Adding More Models for Different Tasks

Build a model zoo for specialized tasks:

# Reasoning ollama pull deepseek-r1:32b-q4_k_m # Code review ollama pull qwen2.5-coder:32b-instruct-q4_k_m # SQL generation ollama pull defog/sqlcoder:7b-q4_k_m # Multimodal (vision) ollama pull llava:13b-q4_k_m ollama pull llava-llama3:8b-q4_k_m # Embeddings (for RAG) ollama pull nomic-embed-text ollama pull mxbai-embed-large # Reranking ollama pull bge-reranker-v2-m3

Integrating with Automation (n8n, Home Assistant)

n8n Workflow Example:

1. Webhook receives GitHub PR event
2. Ollama analyzes code changes via Open WebUI API
3. Posts summary as PR comment

Home Assistant:

• Voice assistant with local LLM (Wyoming + Open WebUI)
• Automation: "When motion detected, describe scene with LLaVA"

Building Custom Agents with Function Calling

Open WebUI Pipelines + Ollama tools = agents:

# Example: Kubernetes troubleshooting agent functions = [ { "name": "kubectl_get_pods", "description": "Get pods in namespace", "parameters": {"namespace": {"type": "string"}} }, { "name": "kubectl_logs", "description": "Get logs for pod", "parameters": {"pod": {"type": "string"}, "namespace": {"type": "string"}} } ]

Fine-Tuning Models with LoRA

# Using unsloth (fast LoRA fine-tuning) pip install unsloth # Or use Ollama's native fine-tuning (experimental) # Create training data in JSONL format # ollama create fine-tuned-model -f Modelfile with FROM base + ADAPTER

Monitoring and Observability

Prometheus + Grafana Stack:

# Add to docker-compose.yml services: prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - prometheus:/prometheus ports: - "9090:9090" grafana: image: grafana/grafana:latest volumes: - grafana:/var/lib/grafana ports: - "3001:3000" environment: - GF_SECURITY_ADMIN_PASSWORD=admin node-exporter: image: prom/node-exporter:latest pid: host network_mode: host volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro

Key metrics to alert on:

• GPU memory utilization > 90%
• Inference latency p99 > 30s
• Disk space < 10% free
• Container restart loops

Quick Reference: Complete Docker Compose

version: '3.8' services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - "11434:11434" volumes: - ollama:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] environment: - OLLAMA_HOST=0.0.0.0 - OLLAMA_ORIGINS=* - OLLAMA_MAX_LOADED_MODELS=3 - OLLAMA_NUM_PARALLEL=2 - OLLAMA_KEEP_ALIVE=10m networks: - ai-internal open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped ports: - "3000:8080" volumes: - open-webui:/app/backend/data environment: - OLLAMA_BASE_URL=http://ollama:11434 - WEBUI_SECRET_KEY=CHANGE_THIS_TO_SECURE_RANDOM_STRING - ENABLE_SIGNUP=false - DEFAULT_MODELS=llama3.1:8b-instruct-q4_k_m - ENABLE_RAG=true - ENABLE_WEB_SEARCH=true - WEB_SEARCH_ENGINE=duckduckgo depends_on: - ollama networks: - ai-internal - ai-external networks: ai-internal: driver: bridge internal: true ai-external: driver: bridge volumes: ollama: open-webui:

Deploy:

mkdir -p /opt/homelab-ai && cd /opt/homelab-ai # Save compose file, generate secret, deploy docker compose up -d

Verification Checklist

After deployment, verify each component:

• [ ] Ollama API responds: curl http://localhost:11434/api/tags
• [ ] Model pulls work: docker exec ollama ollama pull llama3.1:8b-instruct-q4_k_m
• [ ] Open WebUI loads: http://your-ip:3000 (or via reverse proxy)
• [ ] Admin account created and signup disabled
• [ ] GPU detected: docker exec ollama nvidia-smi (or rocm-smi)
• [ ] Chat works: Send message, receive response
• [ ] RAG works: Upload PDF, ask question about it
• [ ] Web search works: Ask "What's the weather in Tokyo?"
• [ ] Reverse proxy works: HTTPS, valid cert, security headers
• [ ] Backups run: Test restore from backup
• [ ] Updates work: docker compose pull && docker compose up -d

Summary

You now have a complete, production-ready local AI stack:

Component	Purpose	Access
Ollama	Model runtime, API server	http://localhost:11434 (internal)
Open WebUI	Chat interface, RAG, pipelines	https://ai.yourdomain.com
Reverse Proxy	TLS, auth, rate limiting	Port 443
Monitoring	Observability, alerts	Grafana dashboards

Start small: One model (Llama 3.1 8B q4_k_m), CPU or single GPU, basic auth. Scale up: Add models, GPUs, RAG, pipelines, SSO, monitoring as needs grow.

The homelab AI journey is iterative. Each model you pull, each pipeline you build, each integration you automate teaches you more about running AI on your own terms — no API keys, no rate limits, no data leaving your network.

Current Status

Production Ready

This stack has been tested on Ubuntu 22.04/24.04, Proxmox VE 8.x, Docker 27.x, Ollama 0.3.x, and Open WebUI 0.3.x. For production homelabs, add a reverse proxy with TLS, enable backups, and configure monitoring before exposing to the internet.

Last reviewed: August 2026 | Tested on: Ubuntu 22.04/24.04, Proxmox VE 8.x, Docker 27.x, Ollama 0.3.x, Open WebUI 0.3.x