---
title: "llama.cpp Integration"
source: https://docs.autohand.ai/integrations/llama-cpp
---

# llama.cpp

Run GGUF models locally with maximum performance. A lightweight, high-performance inference engine for running quantized models on any hardware.

## Overview

llama.cpp is a high-performance C++ inference engine optimized for running large language models locally. When integrated with Autohand, you get:

-   Maximum inference speed with optimized C++ code
-   Support for quantized GGUF models (2-bit to 8-bit)
-   Efficient memory usage with memory-mapped files
-   Cross-platform support (macOS, Linux, Windows)
-   Hardware acceleration (Metal, CUDA, OpenCL, Vulkan)
-   Direct control over model loading and inference parameters

**When to use llama.cpp:** Choose llama.cpp over Ollama when you need maximum performance, fine-grained control over inference parameters, or want to run specific GGUF model variants.

## Installation

Build llama.cpp from source or use pre-built binaries.

### macOS

Build with Metal support for Apple Silicon:

``` bash
# Clone the repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with Metal acceleration (Apple Silicon)
make LLAMA_METAL=1

# Or use CMake for more options
mkdir build && cd build
cmake .. -DLLAMA_METAL=ON
cmake --build . --config Release

# Verify installation
./llama-server --version
```

Using Homebrew:

``` bash
# Install via Homebrew
brew install llama.cpp

# Verify installation
llama-server --version
```

### Linux

Build with CUDA support for NVIDIA GPUs:

``` bash
# Install dependencies
sudo apt install build-essential cmake

# Clone and build
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with CUDA (NVIDIA GPU)
make LLAMA_CUDA=1

# Or build with OpenBLAS (CPU optimized)
make LLAMA_OPENBLAS=1

# Or build with Vulkan (AMD/Intel GPU)
make LLAMA_VULKAN=1

# Verify
./llama-server --version
```

### Windows

Build using Visual Studio or MinGW:

``` bash
# Clone repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Build with CMake (Visual Studio)
mkdir build
cd build
cmake .. -DLLAMA_CUDA=ON
cmake --build . --config Release

# Or download pre-built binaries from:
# https://github.com/ggml-org/llama.cpp/releases
```

### Start the server

Run the OpenAI-compatible API server:

``` bash
# Start server with a model
./llama-server \
  --model /path/to/model.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 4096 \
  --n-gpu-layers 35

# Server is now running at http://127.0.0.1:8080
```

## GGUF models

Download quantized GGUF models from Hugging Face or convert your own.

### Download models

Popular models are available on Hugging Face:

``` bash
# Using huggingface-cli
pip install huggingface_hub
huggingface-cli download TheBloke/CodeLlama-13B-GGUF \
  codellama-13b.Q4_K_M.gguf \
  --local-dir ./models

# Or use wget
wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf
```

### Recommended models

| Model | Quantization | Size | RAM |
|---|---|---|---|
| CodeLlama 7B | Q4_K_M | 4.1GB | 6GB |
| CodeLlama 13B | Q4_K_M | 7.9GB | 10GB |
| CodeLlama 34B | Q4_K_M | 20GB | 24GB |
| Llama 3.1 8B | Q4_K_M | 4.9GB | 7GB |
| Llama 3.1 70B | Q4_K_M | 40GB | 48GB |
| DeepSeek Coder 6.7B | Q4_K_M | 4.0GB | 6GB |
| Mistral 7B | Q4_K_M | 4.4GB | 6GB |

### Quantization levels

| Quantization | Bits | Quality | Speed |
|---|---|---|---|
| Q2_K | 2-bit | Lowest | Fastest |
| Q3_K_M | 3-bit | Low | Fast |
| Q4_K_M | 4-bit | Good | Balanced |
| Q5_K_M | 5-bit | Better | Moderate |
| Q6_K | 6-bit | High | Slower |
| Q8_0 | 8-bit | Highest | Slowest |

**Recommendation:** Q4\_K\_M offers the best balance of quality and speed for most use cases. Use Q5\_K\_M or Q6\_K for tasks requiring higher accuracy.

## CLI configuration

Configure Autohand to use llama.cpp in `~/.autohand/config.json`:

``` json
{
  "provider": "llamacpp",
  "llamacpp": {
    "baseUrl": "http://127.0.0.1:8080",
    "model": "codellama-13b",
    "contextSize": 4096,
    "temperature": 0.7,
    "topP": 0.9,
    "topK": 40,
    "repeatPenalty": 1.1,
    "seed": -1
  }
}
```

### Configuration options

| Option | Description | Default |
|---|---|---|
| baseUrl | llama.cpp server URL | http://127.0.0.1:8080 |
| model | Model name (for display) | - |
| contextSize | Context window size | 4096 |
| temperature | Sampling temperature (0-2) | 0.7 |
| topP | Nucleus sampling threshold | 0.9 |
| topK | Top-K sampling | 40 |
| repeatPenalty | Repetition penalty | 1.1 |
| seed | Random seed (-1 for random) | -1 |

## Server options

Fine-tune the llama.cpp server for optimal performance:

### Basic options

``` bash
./llama-server \
  --model ./models/codellama-13b.Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 8192 \
  --batch-size 512 \
  --threads 8
```

### GPU acceleration

``` bash
# Offload layers to GPU (more layers = more VRAM, faster)
./llama-server \
  --model ./models/codellama-13b.Q4_K_M.gguf \
  --n-gpu-layers 35 \
  --main-gpu 0

# Use multiple GPUs
./llama-server \
  --model ./models/llama-70b.Q4_K_M.gguf \
  --n-gpu-layers 80 \
  --tensor-split 0.5,0.5
```

### Memory optimization

``` bash
# Use memory-mapped files (recommended for large models)
./llama-server \
  --model ./models/large-model.gguf \
  --mmap \
  --mlock

# Reduce memory with smaller batch size
./llama-server \
  --model ./models/model.gguf \
  --batch-size 128 \
  --ctx-size 2048
```

### Server options reference

| Option | Description |
|---|---|
| --model | Path to GGUF model file |
| --ctx-size | Context size (tokens) |
| --batch-size | Batch size for prompt processing |
| --threads | Number of CPU threads |
| --n-gpu-layers | Layers to offload to GPU |
| --main-gpu | Primary GPU index |
| --tensor-split | Split ratio for multi-GPU |
| --mmap | Use memory-mapped files |
| --mlock | Lock model in memory |

## Performance tuning

### Optimal GPU layers

Calculate layers to offload based on your VRAM:

``` bash
# Check model layers
./llama-cli --model model.gguf --verbose 2>&1 | grep "n_layer"

# General formula: VRAM_GB * 2 = approximate layers
# 8GB VRAM ~ 16-20 layers
# 12GB VRAM ~ 24-30 layers
# 24GB VRAM ~ 48-60 layers
```

### Batch size tuning

| VRAM | Batch Size | Context |
|---|---|---|
| 4GB | 128 | 2048 |
| 8GB | 256 | 4096 |
| 12GB | 512 | 8192 |
| 24GB+ | 1024 | 16384+ |

### CPU-only optimization

``` bash
# Optimize for CPU inference
./llama-server \
  --model model.gguf \
  --threads $(nproc) \
  --batch-size 512 \
  --ctx-size 4096 \
  --mmap
```

## Convert models

Convert Hugging Face models to GGUF format:

### Install dependencies

``` bash
# Install Python dependencies
pip install torch transformers sentencepiece
```

### Convert to GGUF

``` bash
# Convert from Hugging Face format
python convert_hf_to_gguf.py \
  /path/to/hf-model \
  --outfile model-f16.gguf \
  --outtype f16

# Quantize the model
./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
```

### Quantization options

``` bash
# Available quantization types
./llama-quantize --help

# Common quantizations:
# Q4_K_M - Best balance (recommended)
# Q5_K_M - Higher quality
# Q8_0   - Highest quality, largest size
# Q2_K   - Smallest, lowest quality
```

## Best practices

-   **Match quantization to hardware**: Use Q4\_K\_M for most systems, Q5\_K\_M+ for high-end GPUs.
-   **Maximize GPU offload**: Offload as many layers as VRAM allows.
-   **Use memory mapping**: Enable `--mmap` for faster model loading.
-   **Tune batch size**: Larger batches improve throughput but use more memory.
-   **Monitor temperature**: GPU inference can cause thermal throttling.

### Recommended configurations

#### 8GB VRAM (RTX 3070/4060)

``` bash
./llama-server \
  --model codellama-13b.Q4_K_M.gguf \
  --n-gpu-layers 20 \
  --ctx-size 4096 \
  --batch-size 256
```

#### 16GB VRAM (RTX 4080)

``` bash
./llama-server \
  --model codellama-34b.Q4_K_M.gguf \
  --n-gpu-layers 40 \
  --ctx-size 8192 \
  --batch-size 512
```

#### Apple Silicon (M1/M2/M3)

``` bash
./llama-server \
  --model codellama-13b.Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 8192 \
  --batch-size 512
```

## Troubleshooting

### Common issues

| Issue | Solution |
|---|---|
| Out of memory | Reduce --n-gpu-layers or --ctx-size |
| Slow inference | Increase GPU layers or use smaller quantization |
| Model not loading | Verify GGUF format and file integrity |
| CUDA error | Update GPU drivers, rebuild with correct CUDA version |
| Metal not working | Rebuild with LLAMA_METAL=1 |

### Debug mode

``` bash
# Run with verbose output
./llama-server \
  --model model.gguf \
  --verbose

# Check model info
./llama-cli --model model.gguf --verbose 2>&1 | head -50
```

### Test the server

``` bash
# Test API endpoint
curl http://127.0.0.1:8080/v1/models

# Test completion
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'
```