Fine-Tuning Open-Source LLMs on Enterprise Data with PyTorch

1. Introduction

Open-source large language models — Llama, Qwen, Mistral, Gemma — have closed the gap with proprietary systems on virtually every benchmark. Yet a gap remains between what a general-purpose model can do and what your enterprise needs it to do. That gap is filled by fine-tuning: adapting a pre-trained foundation model to the specific language, terminology, and reasoning patterns of your organization's data.

This article walks you through the complete technical workflow — from loading a base model in Transformers to running parameter-efficient fine-tuning with PyTorch — and compares the trade-offs against Retrieval-Augmented Generation (RAG), the alternative approach that has become the default for many enterprises.

Assumptions: You are comfortable with Python, basic deep learning concepts (tensors, backpropagation, optimizers), and the transformer architecture. You know what an LLM is and how it generates text. What you need are the concrete, production-grade code patterns that connect the pieces together.

2. RAG vs. Fine-Tuning: Understanding the Fundamental Difference

Before writing any code, it is critical to understand what RAG and fine-tuning actually do at the level of model architecture and data flow.

2.1 How RAG Works

RAG (Retrieval-Augmented Generation) does not modify the model's weights. Instead, it inserts a retrieval step before every inference call:

1. The user's query is embedded using a vector encoder (e.g., sentence-transformers/all-MiniLM-L6-v2).
2. Similar chunks from your enterprise knowledge base are looked up in a vector database (Milvus, Weaviate, Chroma, Pinecone).
3. The retrieved chunks are injected into the prompt's context window as supplementary text.
4. The frozen LLM conditions its generation on both the original query and the retrieved passages.

The model's parameters remain untouched. All the enterprise-specific knowledge lives in the external vector store. The model simply learns to attend to whatever text is placed in its prompt.

2.2 How Fine-Tuning Works

Fine-tuning alters the model's parameters by continuing training on a domain-specific dataset. The model internalizes the patterns, vocabulary, and reasoning style of your data:

1. A labeled dataset is prepared (e.g., question-answer pairs in your company's domain).
2. The model performs forward passes on this data, computing loss against ground-truth labels.
3. Gradients are backpropagated and optimizer steps update the weights.
4. The adapted model now generates responses that reflect the domain without needing retrieval at inference time.

2.3 Side-by-Side Comparison

3. The Fine-Tuning Workflow: Six Stages

The fine-tuning pipeline consists of six distinct stages. Each is shown with production-grade code.

Stage 1: Load the pre-trained base model.
Stage 2: Prepare and tokenize the domain dataset.
Stage 3: Apply parameter-efficient adapters (LoRA / QLoRA).
Stage 4: Configure the training arguments.
Stage 5: Run the training loop.
Stage 6: Merge and export the adapted weights for inference.

4. Stage 1 — Loading the Pre-Trained Base Model

The entry point is the transformers library. We load a causal language model in its native precision (or quantized, for QLoRA):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "meta-llama/Llama-3.1-8B"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token  # Required for padding

# Load model in fp16 on GPU
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
    attn_implementation="sdpa",
)

For 70B+ models, or when GPU memory is constrained, load with 4-bit quantization using bitsandbytes:

# 4-bit quantized loading (reduces VRAM ~4x)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)

5. Stage 2 — Preparing the Domain Dataset

Fine-tuning requires a dataset in the model's expected format. For causal language modeling, each sample is a sequence where the model predicts the next token. The standard format uses the instruction-following structure:

from datasets import Dataset, DatasetDict
import json

# Load your enterprise data: list of {"instruction": ..., "output": ...}
with open("enterprise_data.json") as f:
    raw_data = json.load(f)

# Format as chat-style sequences
def format_example(example):
    prompt = (
        "<|begin_of_text|><|start_header_id|>user<|end_header_id|><br>"
        f"{example['instruction']}<|eot_id|>"
        "<|start_header_id|>assistant<|end_header_id|><br>"
        f"{example['output']}<|eot_id|>"
    )
    return {"text": prompt}

formatted = [format_example(d) for d in raw_data]
dataset = Dataset.from_dict({"text": [d["text"] for d in formatted]})
dataset = dataset.train_test_split(test_size=0.1, seed=42)

# Tokenize
def tokenize(batch):
    return tokenizer(
        batch["text"],
        truncation=True,
        max_length=512,
        padding=False,
    )

tokenized_ds = dataset.map(tokenize, batched=True, remove_columns=["text"])

6. Stage 3 — Applying Parameter-Efficient Fine-Tuning (LoRA)

Fine-tuning all 8 billion parameters of Llama-3.1-8B requires ~64 GB of GPU memory even with batch size 1. LoRA (Low-Rank Adaptation) avoids this by freezing the base weights and injecting trainable rank-decomposition matrices into select attention layers:

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# For QLoRA: prepare the model for k-bit training
model = prepare_model_for_kbit_training(model)

# Configure LoRA
lora_config = LoraConfig(
    r=16,                          # Rank of the update matrices
    lora_alpha=32,                 # Scaling factor: alpha / r
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# Wrap the model -- injects LoRA adapters
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# → trainable: 41,943,040 | all: 8,431,641,600 | 0.497%

Architecturally: instead of updating 8.4B parameters, you update only ~42M (0.5%). The LoRA adapters are low-rank matrices A ∈ R^(r, d) and B ∈ R^(d, r). The effective weight update is ΔW = B @ A, a rank-16 matrix. At inference, these can be merged into the base weights for zero-overhead generation.

7. Stage 4 — Configuring the Training Loop

The Hugging Face Trainer API wraps PyTorch's training machinery. Here we configure hyperparameters, optimization, and evaluation:

from transformers import TrainingArguments, Trainer, DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

training_args = TrainingArguments(
    output_dir="./outputs/llama3-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,   # Effective batch size = 4 x 8 = 32
    optim="paged_adamw_8bit",        # 8-bit Adam saves VRAM
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    weight_decay=0.01,
    fp16=True,
    bf16=False,
    logging_steps=10,
    logging_dir="./outputs/logs",
    save_strategy="epoch",
    eval_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    report_to="tensorboard",
    seed=42,
)

# Loss default: causal LM loss (cross-entropy on next-token prediction)

Key hyperparameter decisions:

• learning_rate = 2e-4: Higher than full-finetuning because LoRA adapters learn faster. Full fine-tuning typically uses 1e-5–5e-5.
• optim = "paged_adamw_8bit": Uses 8-bit optimizer states with VRAM paging. Essential for fitting large models on consumer GPUs.
• gradient_accumulation_steps = 8: Simulates a batch size of 32 without loading 32 examples in memory simultaneously.
• lr_scheduler_type = "cosine": Cosine annealing reduces the learning rate smoothly to zero, helping converge on the fine-grained domain signal.

8. Stage 5 — Executing the Training Loop

Instantiate the Trainer and call train(). The Trainer handles the PyTorch training loop internally — forward pass, loss computation, backward pass, optimizer step, gradient scaling, and logging:

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_ds["train"],
    eval_dataset=tokenized_ds["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()

Under the hood, each training step executes this PyTorch sequence:

# What happens inside trainer.train() per batch:
optimizer.zero_grad(set_to_none=True)          # Zero gradients
with torch.amp.autocast(device_type="cuda", dtype=torch.float16):
    outputs = model(**batch)                    # Forward pass
    loss = outputs.loss                           # Cross-entropy loss
scaled_loss = scaler.scale(loss)               # AMP scaling
scaled_loss.backward()                         # Backward pass (autograd)
scaler.step(optimizer)                          # Unscale + optimizer step
scaler.update()                                # Update scale factor

9. Stage 6 — Merging Adapters and Exporting

After training completes, merge the LoRA weights back into the base model. This produces a single standard model file that can be deployed without the PEFT dependency:

# Merge LoRA adapters into base weights
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./outputs/merged-model")
tokenizer.save_pretrained("./outputs/merged-model")

# Adapter weights also saved separately for dynamic loading:
model.save_pretrained("./outputs/lora-adapter")

After merging, inference is identical to running any Hugging Face causal LM:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the merged model -- no PEFT needed at inference
model = AutoModelForCausalLM.from_pretrained(
    "./outputs/merged-model",
    device_map="auto",
    torch_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained("./outputs/merged-model")

# Prompt the model
prompt = ("<|begin_of_text|><|start_header_id|>user<|end_header_id|><br>"
          "What is our company’s return policy?<|eot_id|>"
          "<|start_header_id|>assistant<|end_header_id|><br>")
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

10. Alternative: Inference with Unmerged Adapters

You can keep the base model and adapter separate — useful for dynamic adapter swapping across multiple domains:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", device_map="auto"
)
adapter_model = PeftModel.from_pretrained(base_model, "./outputs/lora-adapter")

# adapter_model now behaves like a regular causal LM
outputs = adapter_model.generate(**inputs, max_new_tokens=256)

11. Going Deeper: The Manual PyTorch Training Loop

The Trainer API abstracts the training loop away. But understanding the manual loop is essential for debugging and for custom training logic — gradient clipping, curriculum scheduling, or custom loss terms. Here is the minimal loop:

import torch.nn.functional as F
from torch.utils.data import DataLoader

optimizer = torch.optim.AdamW(
    filter(lambda p: p.requires_grad, model.parameters()),
    lr=2e-4,
    weight_decay=0.01,
)
scaler = torch.cuda.amp.GradScaler()

for epoch in range(num_epochs):
    model.train()
    for batch in DataLoader(train_dataset, batch_size=4):
        batch = {k: v.to(model.device) for k, v in batch.items()}

        optimizer.zero_grad(set_to_none=True)

        with torch.cuda.amp.autocast(dtype=torch.float16):
            outputs = model(**batch)
            loss = outputs.loss

        scaler.scale(loss).backward()

        # Gradient clipping to prevent explosion
        scaler.unscale_(optimizer)
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        scaler.step(optimizer)
        scaler.update()

12. Infrastructure and Hardware Considerations

Choosing the right hardware is as important as choosing the right hyperparameters. Here are the practical constraints:

# Configure distributed training
accelerate config

# Run with Accelerate (handles DDP/FSDP automatically)
accelerate launch train.py

13. Practical Decision Guide: When to Fine-Tune vs. When to RAG

Choosing between fine-tuning and RAG is not a binary decision. It is a spectrum of architectural choices that depends on your data characteristics, latency requirements, and operational constraints. Both approaches modify how a model interacts with enterprise information, but they do so at fundamentally different layers of the stack.

At a deep level, RAG and fine-tuning are two strategies for injecting domain knowledge into a language model. RAG does so externally: the weights stay frozen, and context from your knowledge base is appended to each prompt at inference time. The model never sees your data during training; it only conditions on whatever retrieval surfaces. Fine-tuning does so internally: the model's parameters are updated to encode domain patterns directly. After fine-tuning, the adapted model produces responses shaped by your data without any retrieval step.

The table in Section 2 already compares the two across ten dimensions. The guidance below translates those technical trade-offs into practical decision criteria for your engineering team.

Fine-tune when:

• Domain language matters — internal acronyms, product naming conventions, regulatory frameworks that appear in every answer.
• Consistent response format is required — structured outputs, specific JSON schemas, standardized templates.
• Low-latency inference is needed — no retrieval step means faster response times.
• Domain reasoning is required — diagnosing technical issues by combining internal knowledge in novel ways.
• Stable dataset — the training data does not change every day.

Use RAG when:

• Knowledge base changes frequently — pricing, inventory, org charts, policies.
• Citations and evidence are required — every answer must be grounded in retrievable source material.
• Data access controls matter — RAG can filter retrieved chunks by user permissions.
• Massive, growing dataset — a vector DB scales more cheaply than retraining.
• GPU training infrastructure is not available — RAG runs on CPU for embedding + inference on a single GPU.

The hybrid approach (recommended):

14. Common Pitfalls to Avoid

Even with the correct code, fine-tuning can fail silently. Here are the most common issues:

1. Catastrophic forgetting. Fine-tuning on narrow domain data can degrade the model's general capabilities. Mitigation: use a small portion of general-purpose data (e.g., Wikipedia excerpts) mixed with your domain data.

2. Overfitting to training format. If your training data has a rigid template, the model may lose flexibility. Use data augmentation — paraphrase variations of the same instruction.

3. Not evaluating on out-of-distribution queries. Report metrics not just on your test set but on queries that slightly diverge from your training distribution. This reveals whether the model truly learned or just memorized.

4. Ignoring the tokenizer. If your domain uses specialized terminology (chemical formulas, medical codes, internal SKUs), extend the tokenizer's vocabulary before training:

# Add domain-specific tokens before training
additional_tokens = ["[[INTERNAL_CODE]]", "[[POLICY_REF]]", "SKU-XXXX"]
tokenizer.add_tokens(additional_tokens)
model.resize_token_embeddings(len(tokenizer))  # Resize embedding layer

5. Training for too few epochs on too little data. A good rule of thumb: epochs = max(3, floor(3000 / train_samples)). With fewer than 1,000 samples, train at least 3–5 epochs. With more than 3,000, 1–2 epochs suffice.

15. Conclusion

Fine-tuning an open-source LLM on enterprise data is no longer the exclusive domain of teams with massive GPU clusters. With QLoRA and the Hugging Face PEFT library, a single GPU can adapt an 8B-parameter model to your company's domain in hours, not weeks. The code patterns above — from model loading through LoRA configuration to the PyTorch training loop — form the foundation for any fine-tuning effort.

But the decision to fine-tune should always be preceded by a clear understanding of what fine-tuning can and cannot do. It internalizes patterns and style but does not provide live factual grounding. RAG does the latter but not the former. The most effective enterprise AI systems use both: a fine-tuned model that speaks your language, augmented by RAG that grounds every answer in your live data.

The question is no longer whether you can fine-tune an LLM. It is whether you should — and if so, how to design a system that combines fine-tuning and RAG for maximum impact.