Local Deployment of Open Source AI Models

Running large language models on your own hardware has gone from hobby experiment to serious infrastructure choice. Teams and makers deploy locally for data sovereignty, air-gapped environments, predictable per-token economics, and resilience when cloud APIs change price or availability overnight.

The stack has matured quickly: Ollama for fast startup and OpenAI-compatible API, llama.cpp for efficient inference with GGUF quantization on CPU/GPU/Apple Silicon, vLLM for production throughput on multi-GPU servers. Models like Llama 3.x, Mistral, and Chinese frontier releases like GLM-5.2 appear on Hugging Face — but catalog presence does not equal "fits on my laptop."

Why Deploy Locally

• Privacy: Prompts, documents, and code never leave the network — critical for healthcare, legal, defense
• Offline: Construction sites, ships, factories, secure labs without always-on internet
• Cost control: High-volume inference on owned GPUs can beat API bills if usage is constant
• Customization: Fine-tuning, quantization, LoRA adapters without vendor approval workflows
• Latency: Local GPU can beat round-trip to distant cloud regions for interactive apps
• Geopolitical independence: Open-weight models not subject to foreign API kill switches (with license caveats)

Local Stack Architecture

A typical local deployment consists of stacked layers:

The golden rule: each layer adds flexibility and complexity. To start, stay on Ollama until you have a measurable reason to drop down to raw llama.cpp or move up to vLLM.

Ollama — The Entry Point

Ollama wraps model pull, GPU detection, and HTTP server in a single binary. Ideal for prototyping on macOS, Linux, Windows.

# Installation (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run Llama 3.2 8B
ollama pull llama3.2
ollama run llama3.2

# OpenAI-compatible API
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Explain MVCC in PostgreSQL"}]
  }'

Mistral models via Ollama

ollama pull mistral
ollama pull mixtral   # MoE 8x7B — requires ~26GB+ RAM for Q4

ollama run mixtral "Write an Arduino sketch to read DHT22"

Ollama uses llama.cpp under the hood for many models. Trade-off: less fine tuning than direct llama.cpp, drastically lower friction.

llama.cpp — Efficiency and Control

Reference implementation for GGUF quantized models. Runs on CPU-only (slow but possible), Apple Silicon, NVIDIA CUDA, AMD ROCm, Intel.

# Build from source (Linux/macOS)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release

# Download GGUF from Hugging Face
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
  mistral-7b-instruct-v0.2.Q4_K_M.gguf --local-dir ./models

# Inference
./build/bin/llama-cli -m ./models/mistral-7b-instruct-v0.2.Q4_K_M.gguf \
  -p "Explain PID control for a mobile robot" -n 512 -c 4096

Common quantizations

vLLM — Production Serving

When you need concurrent users, continuous batching, and tensor parallelism across multiple GPUs, vLLM is the de facto standard.

# Installation
pip install vllm

# OpenAI-compatible API server
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192

# Multi-GPU (e.g. 70B sharded)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2

vLLM targets the datacenter, not the casual laptop. PagedAttention reduces KV cache memory waste — critical at high batch sizes.

Models: Llama, Mistral, GLM-5.2

Llama 3.x (Meta)

Open-weight family from 1B to 405B. Local workhorses are 8B and 70B instruct. Llama 3.2 1B/3B for tight edge; 70B requires 40–48GB VRAM at Q4 or multi-GPU setup.

ollama pull llama3.1:70b    # Check RAM/VRAM first!
ollama pull llama3.2:3b     # Edge / Raspberry Pi 5 with accelerators

Mistral / Mixtral

Mistral 7B remains a solid baseline on a single GPU. Mixtral 8x7B (MoE, ~47B total / ~13B active per token) offers superior quality at moderate VRAM when quantized — typically 24–32GB for Q4.

GLM-5.2 — read the specs carefully

GLM-5.2 from Z.ai is a frontier Mixture-of-Experts model with approximately 744 billion total parameters. It is not a 7B model. It is not something you casually install in Ollama on a gaming PC.

Even with aggressive 2-bit quantization, credible estimates place minimum system RAM in the 256GB+ range for the full MoE — assuming expert offloading strategies, not a single consumer GPU with 6GB or 24GB holding the entire model resident in VRAM.

For realistic local use with the GLM ecosystem, target:

• GLM-4-9B-Chat — instruct chat, quantizable, Ollama-compatible via community
• GLM-Z1 series — specialized reasoning, manageable sizes
• GLM-5.2 — only for labs with enterprise hardware or multi-node GPU clusters

Honest Hardware Table

Empirical rule for dense models: Q4 requires roughly 0.5–0.7 GB per billion parameters of weights in memory, plus KV cache overhead (grows with context length and batch size). MoE adds routing complexity — total parameter count overestimates active compute but underestimates storage because many experts must remain addressable.

Lifecycle Tools — From Prototype to Production

1. Define the task: coding assistant, RAG on internal documents, batch summarization — the task determines minimum model size
2. Measure hardware: nvidia-smi, Activity Monitor, ollama ps during candidate run
3. Prototype with Ollama: validate quality before optimizing
4. Quantize if needed: step down Q8 → Q6 → Q4 until it fits; compare output on real prompts
5. Integrate RAG: local embeddings (nomic-embed, bge) + vector store (Chroma, pgvector)
6. Promote to vLLM when multiple clients hit the same endpoint concurrently
7. Monitor: tokens/sec, VRAM headroom, context length — OOM means quantize more or reduce context
8. Backup models: GGUF on NAS; avoid re-downloading from Hugging Face on every reinstall

Complementary tools

• LM Studio — GUI for desktop model exploration
• text-generation-webui — Gradio interface for testing and comparison
• llamafile — single executable with embedded model
• MLX (Apple) — optimized inference on Apple Silicon
• Open WebUI — self-hosted chat frontend over Ollama/vLLM

Common Mistakes

• Confusing total with active parameters on MoE in online VRAM calculators
• Ignoring KV cache: model that "fits" at idle can go OOM at 32k context
• Chasing frontier models on inadequate hardware — a fast 8B beats a crawling 405B
• Skipping license review: some "open" weights limit commercial use or deployment scale
• Believing "6GB for GLM-5.2": invented numbers or referring to other models

Local RAG — Complementary Pattern

Local model deployment often accompanies Retrieval-Augmented Generation: embedding internal documents, querying a vector store, injecting context into the prompt. Typical maker/enterprise stack:

# Local embedding with Ollama
ollama pull nomic-embed-text

# Vector store: Chroma, pgvector, or SQLite-vec
# Python app: LangChain / LlamaIndex calling localhost:11434

Advantage: sensitive documents never leave the network. Disadvantage: retrieval quality and chunking remain human responsibility — a local model does not compensate for misconfigured RAG.

Conclusions

Local AI deployment is mature enough for daily work on realistic hardware — 7B–70B models on prosumer GPUs, smaller models on laptops. Ollama, llama.cpp, and vLLM cover the path from experiment to production.

Frontier MoE releases like GLM-5.2 belong in the same conversation as frontier cloud APIs: impressive, strategic, hungry for hardware. Plan for hundreds of gigabytes of memory, not a single consumer graphics card. Matching model to machine turns local inference into a lasting advantage instead of frustration.

Sources

• Ollama — ollama.com
• llama.cpp — github.com/ggerganov/llama.cpp
• vLLM — github.com/vllm-project/vllm
• Meta Llama — llama.meta.com
• Mistral AI — mistral.ai
• Z.ai GLM Models — huggingface.co/zai-org