Self-Hosting Open Models for Cyber Defense

The Data Governance Dilemma

Sending firewall logs, phishing emails, or breach postmortems to a commercial LLM API creates a data governance problem. Even with enterprise DPAs, many CISOs in regulated sectors — finance, healthcare, defense suppliers — prefer models that run entirely on controlled infrastructure.

Open-weight language models, served locally with Ollama or vLLM, keep sensitive text on-prem. They will not match the largest closed models on every task, but for structured security workflows with human review they are already useful. This guide covers why to self-host, realistic use cases, stack choices, minimal Ollama setup, and the hard limits not to ignore.

Why Local LLMs for Security

• Data residency: incident narratives, PII in logs, and proprietary architecture diagrams stay in the VLAN.
• Air-gapped SOC: classified or isolated environments run inference without outbound HTTPS to model vendors.
• Predictable cost: CapEx on GPU servers vs unlimited per-token API bills during a major incident.
• Customization: RAG on internal runbooks, past tickets, and MITRE mappings without uploading them to third parties.
• Supply-chain control: you choose when to update weights; no silent model version change that breaks prompts.

Self-hosting is a privacy and control decision, not a magical accuracy upgrade. A 7B model in your rack is still a 7B — it just never phones home.

Practical Use Cases (With Human in the Loop)

Log triage and summarization

Feed anonymized snippets from web server, authentication, or EDR logs. Ask the model to group repeated error patterns, summarize a 500-line trace into bullet points, or suggest which fields indicate lateral movement. The analyst validates before escalation.

Phishing and BEC analysis

Paste headers and body (redact user identifiers first). Request explanation of SPF/DKIM failures, suspicious URLs, and social engineering tactics. Useful for tier-1 triage — not for final verdict without sandbox detonation and threat intel feeds.

Playbook and runbook drafting

Generate first drafts of containment steps for ransomware or credential stuffing, anchored via RAG to organization-approved templates. A senior engineer edits before publication on Confluence or a SOAR platform.

Policy and control mapping

Map observed behaviors to MITRE ATT&CK techniques or internal control frameworks. Treat suggestions as initial checklists; automated compliance claims from LLMs are unreliable.

Recommended Stack

Inference runtime

• Ollama — Fastest path for teams without deep MLOps. Single binary, REST API, excellent for 7B–14B quantized models on one GPU.
• vLLM — Higher throughput for shared SOC services; better when many analysts hit the same endpoint concurrently.

Model selection

• Security-tuned models — SecGPT-style checkpoints (community forks vary; evaluate on your own holdout prompts).
• General models + RAG — Llama 3.x, Mistral, or Qwen variants with a vector store on internal docs often beat small domain models on org-specific vocabulary.

RAG layer

Index runbooks, past incident reports (sanitized), asset inventories, and vendor advisories. Tools like LlamaIndex, LangChain, or pgvector + cron jobs work. Never index live secrets or complete credential dumps.

Minimal Ollama Setup

On a Linux SOC utility host with NVIDIA GPU (or Apple Silicon Mac for lab):

# Ollama installation (see ollama.com for your OS)
curl -fsSL https://ollama.com/install.sh | sh

# Pull general-purpose model (adapt to RAM/GPU)
ollama pull llama3.1:8b

# Optional: security-oriented community variant if evaluated
# ollama pull <your-vetted-sec-model>

# Interactive test
ollama run llama3.1:8b

Call the local API from a triage Python script (no data leaves localhost):

import requests

prompt = (
    "Summarize this auth log snippet. Flag anomalies only.\n"
    "Log:\n"
    "2026-07-27T08:12:01 failed login user=admin src=203.0.113.44\n"
    "2026-07-27T08:12:02 failed login user=admin src=203.0.113.44\n"
    "2026-07-27T08:12:05 successful login user=svc_backup src=10.0.0.12\n"
)

resp = requests.post(
    "http://127.0.0.1:11434/api/generate",
    json={"model": "llama3.1:8b", "prompt": prompt, "stream": False},
    timeout=120,
)
print(resp.json()["response"])

Example: phishing analysis with redaction

import re
import requests

def redact_emails(text):
    return re.sub(r'[\w.-]+@[\w.-]+', '[EMAIL_REDACTED]', text)

email_body = redact_emails(open("sample_phish.eml").read())
prompt = (
    "Analyze this email for phishing indicators.\n"
    "Do not click links. List: suspicious sender, urgency tactics, URLs.\n"
    "---\n" + email_body[:4000]
)

r = requests.post(
    "http://127.0.0.1:11434/api/generate",
    json={"model": "llama3.1:8b", "prompt": prompt, "stream": False},
    timeout=180,
)
print(r.json()["response"])

Place Ollama behind an internal reverse proxy with mTLS, rate limiting, and audit logging. Do not expose port 11434 to the internet.

RAG-Assisted Playbooks

A productive pattern: index approved runbook PDFs, then query with retrieval-augmented context.

# Conceptual example with Ollama + Chroma (simplified)
import chromadb
from chromadb.utils import embedding_functions

ef = embedding_functions.OllamaEmbeddingFunction(
    model_name="nomic-embed-text",
    url="http://127.0.0.1:11434",
)
client = chromadb.PersistentClient(path="./soc_rag")
collection = client.get_or_create_collection("runbooks", embedding_function=ef)

# Ingest (one-time)
collection.add(
    documents=[open("runbook_ransomware.md").read()],
    ids=["rb-ransomware-v3"],
)

# Incident query
results = collection.query(
    query_texts=["HR file share encryption, ransom .txt note"],
    n_results=2,
)
context = results["documents"][0][0]

prompt = f"Using ONLY this runbook:\n{context}\n\nSuggest first 5 containment steps."
# ... POST to /api/generate

The senior analyst validates every step before execution on SOAR. The model accelerates search, it does not replace judgment.

Honest Limits vs SIEM

• Does not replace a SIEM: LLMs do not correlate billions of events in real time with deterministic rules.
• Hallucinations: models invent CVE numbers, IoCs, and legal citations. Verify every factual claim.
• Prompt injection: an attacker controlling email body can manipulate analysis. Sanitize and isolate untrusted content.
• No autonomous containment: do not connect model output directly to firewall block APIs without analyst approval.
• Model theft and GPU: self-hosted weights on a compromised server become attacker assets. Harden the inference host like any crown-jewel system.

Conclusions

Self-hosted open models belong in the modern SOC toolbox as analyst accelerators — summarization, drafting, and RAG lookup — not as autonomous defenders. Pair Ollama or vLLM with vetted weights, internal documentation indexes, and mandatory human review.

Start with one workflow (phishing triage or log summarization), measure time saved and error rates, then expand. The win is keeping secrets secret while shaving minutes off every ticket — not replacing the SIEM that actually catches the breach.

Sources

• Ollama — Local model serving
• vLLM — High-throughput LLM inference
• MITRE ATT&CK — Adversary tactics and techniques knowledge base
• SecGPT (community) — Security-oriented fine-tune; evaluate before production use
• NIST AI RMF — Risk management framework for enterprise AI deployment