llama.cpp Integration - Autohand Docs

Overview

llama.cpp is a high-performance C++ inference engine optimized for running large language models locally. When integrated with Autohand, you get:

Maximum inference speed with optimized C++ code
Support for quantized GGUF models (2-bit to 8-bit)
Efficient memory usage with memory-mapped files
Cross-platform support (macOS, Linux, Windows)
Hardware acceleration (Metal, CUDA, OpenCL, Vulkan)
Direct control over model loading and inference parameters

When to use llama.cpp: Choose llama.cpp over Ollama when you need maximum performance, fine-grained control over inference parameters, or want to run specific GGUF model variants.

Installation

Build llama.cpp from source or use pre-built binaries.

macOS

Build with Metal support for Apple Silicon:

# Clone the repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with Metal acceleration (Apple Silicon)
make LLAMA_METAL=1

# Or use CMake for more options
mkdir build && cd build
cmake .. -DLLAMA_METAL=ON
cmake --build . --config Release

# Verify installation
./llama-server --version

Using Homebrew:

# Install via Homebrew
brew install llama.cpp

# Verify installation
llama-server --version

Linux

Build with CUDA support for NVIDIA GPUs:

# Install dependencies
sudo apt install build-essential cmake

# Clone and build
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with CUDA (NVIDIA GPU)
make LLAMA_CUDA=1

# Or build with OpenBLAS (CPU optimized)
make LLAMA_OPENBLAS=1

# Or build with Vulkan (AMD/Intel GPU)
make LLAMA_VULKAN=1

# Verify
./llama-server --version

Windows

Build using Visual Studio or MinGW:

# Clone repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with CMake (Visual Studio)
mkdir build
cd build
cmake .. -DLLAMA_CUDA=ON
cmake --build . --config Release

# Or download pre-built binaries from:
# https://github.com/ggml-org/llama.cpp/releases

Start the server

Run the OpenAI-compatible API server:

# Start server with a model
./llama-server \
  --model /path/to/model.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 4096 \
  --n-gpu-layers 35

# Server is now running at http://127.0.0.1:8080

GGUF models

Download quantized GGUF models from Hugging Face or convert your own.

Download models

Popular models are available on Hugging Face:

# Using huggingface-cli
pip install huggingface_hub
huggingface-cli download TheBloke/CodeLlama-13B-GGUF \
  codellama-13b.Q4_K_M.gguf \
  --local-dir ./models

# Or use wget
wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf

Recommended models

Model	Quantization	Size	RAM
CodeLlama 7B	Q4_K_M	4.1GB	6GB
CodeLlama 13B	Q4_K_M	7.9GB	10GB
CodeLlama 34B	Q4_K_M	20GB	24GB
Llama 3.1 8B	Q4_K_M	4.9GB	7GB
Llama 3.1 70B	Q4_K_M	40GB	48GB
DeepSeek Coder 6.7B	Q4_K_M	4.0GB	6GB
Mistral 7B	Q4_K_M	4.4GB	6GB

Quantization levels

Quantization	Bits	Quality	Speed
Q2_K	2-bit	Lowest	Fastest
Q3_K_M	3-bit	Low	Fast
Q4_K_M	4-bit	Good	Balanced
Q5_K_M	5-bit	Better	Moderate
Q6_K	6-bit	High	Slower
Q8_0	8-bit	Highest	Slowest

Recommendation: Q4_K_M offers the best balance of quality and speed for most use cases. Use Q5_K_M or Q6_K for tasks requiring higher accuracy.

CLI configuration

Configure Autohand to use llama.cpp in ~/.autohand/config.json:

{
  "provider": "llamacpp",
  "llamacpp": {
    "baseUrl": "http://127.0.0.1:8080",
    "model": "codellama-13b",
    "contextSize": 4096,
    "temperature": 0.7,
    "topP": 0.9,
    "topK": 40,
    "repeatPenalty": 1.1,
    "seed": -1
  }
}

Configuration options

Option	Description	Default
`baseUrl`	llama.cpp server URL	`http://127.0.0.1:8080`
`model`	Model name (for display)	-
`contextSize`	Context window size	`4096`
`temperature`	Sampling temperature (0-2)	`0.7`
`topP`	Nucleus sampling threshold	`0.9`
`topK`	Top-K sampling	`40`
`repeatPenalty`	Repetition penalty	`1.1`
`seed`	Random seed (-1 for random)	`-1`

Server options

Fine-tune the llama.cpp server for optimal performance:

Basic options

./llama-server \
  --model ./models/codellama-13b.Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 8192 \
  --batch-size 512 \
  --threads 8

GPU acceleration

# Offload layers to GPU (more layers = more VRAM, faster)
./llama-server \
  --model ./models/codellama-13b.Q4_K_M.gguf \
  --n-gpu-layers 35 \
  --main-gpu 0

# Use multiple GPUs
./llama-server \
  --model ./models/llama-70b.Q4_K_M.gguf \
  --n-gpu-layers 80 \
  --tensor-split 0.5,0.5

Memory optimization

# Use memory-mapped files (recommended for large models)
./llama-server \
  --model ./models/large-model.gguf \
  --mmap \
  --mlock

# Reduce memory with smaller batch size
./llama-server \
  --model ./models/model.gguf \
  --batch-size 128 \
  --ctx-size 2048

Server options reference

Option	Description
`--model`	Path to GGUF model file
`--ctx-size`	Context size (tokens)
`--batch-size`	Batch size for prompt processing
`--threads`	Number of CPU threads
`--n-gpu-layers`	Layers to offload to GPU
`--main-gpu`	Primary GPU index
`--tensor-split`	Split ratio for multi-GPU
`--mmap`	Use memory-mapped files
`--mlock`	Lock model in memory

Performance tuning

Optimal GPU layers

Calculate layers to offload based on your VRAM:

# Check model layers
./llama-cli --model model.gguf --verbose 2>&1 | grep "n_layer"

# General formula: VRAM_GB * 2 = approximate layers
# 8GB VRAM ~ 16-20 layers
# 12GB VRAM ~ 24-30 layers
# 24GB VRAM ~ 48-60 layers

Batch size tuning

VRAM	Batch Size	Context
4GB	128	2048
8GB	256	4096
12GB	512	8192
24GB+	1024	16384+

CPU-only optimization

# Optimize for CPU inference
./llama-server \
  --model model.gguf \
  --threads $(nproc) \
  --batch-size 512 \
  --ctx-size 4096 \
  --mmap

Convert models

Convert Hugging Face models to GGUF format:

Install dependencies

# Install Python dependencies
pip install torch transformers sentencepiece

Convert to GGUF

# Convert from Hugging Face format
python convert_hf_to_gguf.py \
  /path/to/hf-model \
  --outfile model-f16.gguf \
  --outtype f16

# Quantize the model
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Quantization options

# Available quantization types
./llama-quantize --help

# Common quantizations:
# Q4_K_M - Best balance (recommended)
# Q5_K_M - Higher quality
# Q8_0   - Highest quality, largest size
# Q2_K   - Smallest, lowest quality

Best practices

Match quantization to hardware: Use Q4_K_M for most systems, Q5_K_M+ for high-end GPUs.
Maximize GPU offload: Offload as many layers as VRAM allows.
Use memory mapping: Enable --mmap for faster model loading.
Tune batch size: Larger batches improve throughput but use more memory.
Monitor temperature: GPU inference can cause thermal throttling.

Recommended configurations

8GB VRAM (RTX 3070/4060)

./llama-server \
  --model codellama-13b.Q4_K_M.gguf \
  --n-gpu-layers 20 \
  --ctx-size 4096 \
  --batch-size 256

16GB VRAM (RTX 4080)

./llama-server \
  --model codellama-34b.Q4_K_M.gguf \
  --n-gpu-layers 40 \
  --ctx-size 8192 \
  --batch-size 512

Apple Silicon (M1/M2/M3)

./llama-server \
  --model codellama-13b.Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 8192 \
  --batch-size 512

Troubleshooting

Common issues

Issue	Solution
Out of memory	Reduce `--n-gpu-layers` or `--ctx-size`
Slow inference	Increase GPU layers or use smaller quantization
Model not loading	Verify GGUF format and file integrity
CUDA error	Update GPU drivers, rebuild with correct CUDA version
Metal not working	Rebuild with `LLAMA_METAL=1`

Debug mode

# Run with verbose output
./llama-server \
  --model model.gguf \
  --verbose

# Check model info
./llama-cli --model model.gguf --verbose 2>&1 | head -50

Test the server

# Test API endpoint
curl http://127.0.0.1:8080/v1/models

# Test completion
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'