Chapter 1: Exporting and Serving Models
Preparing Models for Deployment
Export formats and serving options for production deployment.
Export Options
| Format | Use Case | Tools |
|---|---|---|
| Hugging Face | Standard serving | transformers |
| GGUF | CPU inference | llama.cpp |
| ONNX | Cross-platform | onnx |
| TensorRT | NVIDIA optimized | tensorrt |
Merging LoRA for Export
from peft import PeftModel
from transformers import AutoModelForCausalLM
# Load base model
base = AutoModelForCausalLM.from_pretrained("base_model")
# Load adapter
model = PeftModel.from_pretrained(base, "adapter_path")
# Merge
model = model.merge_and_unload()
# Save merged model
model.save_pretrained("./merged_model")
tokenizer.save_pretrained("./merged_model")
GGUF Export
# Convert to GGUF for llama.cpp
python llama.cpp/convert.py ./merged_model --outtype f16
# Quantize for efficiency
./llama.cpp/quantize ./model.gguf ./model-q4_k_m.gguf Q4_K_M
Upload to Hub
model.push_to_hub("username/model-name")
tokenizer.push_to_hub("username/model-name")