---
title: "Ollama Integration"
source: https://docs.autohand.ai/integrations/ollama
---

# Ollama

Run local AI models for private, offline code assistance. Keep your code on your machine while using powerful open-source models.

## Overview

Ollama provides a simple way to run large language models locally. When integrated with Autohand, you get:

-   Complete privacy - your code never leaves your machine
-   Offline capability - work without internet connectivity
-   No API costs - run unlimited queries at no charge
-   Low latency - responses come from your local hardware
-   Model flexibility - choose from dozens of open-source models

**Hardware requirements:** Ollama works best with at least 8GB RAM for 7B models, 16GB for 13B models, and 32GB+ for larger models. GPU acceleration dramatically improves performance.

## Installation

Install Ollama on your operating system to get started with local models.

### macOS

Download and install Ollama using Homebrew or the official installer:

``` bash
# Using Homebrew (recommended)
brew install ollama

# Or download from ollama.com
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version
```

Start the Ollama service:

``` bash
# Start Ollama server (runs in background)
ollama serve

# Or run as a macOS service
brew services start ollama
```

### Linux

Install Ollama using the official install script:

``` bash
# Download and install
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Start the service
sudo systemctl start ollama

# Enable on boot
sudo systemctl enable ollama
```

For manual installation or specific distributions:

``` bash
# Ubuntu/Debian
sudo apt update
sudo apt install -y curl
curl -fsSL https://ollama.com/install.sh | sh

# Fedora/RHEL
sudo dnf install -y curl
curl -fsSL https://ollama.com/install.sh | sh

# Arch Linux (AUR)
yay -S ollama
```

### Windows

Download the Windows installer from the Ollama website or use winget:

``` bash
# Using winget
winget install Ollama.Ollama

# Or download the installer from:
# https://ollama.com/download/windows
```

After installation, Ollama runs as a Windows service automatically. Verify by opening PowerShell:

``` powershell
# Check Ollama is running
ollama --version

# List available models
ollama list
```

### Pull your first model

Download a model to use with Autohand:

``` bash
# Pull a coding-optimized model
ollama pull codellama:13b

# Or use Llama 3.2 for general tasks
ollama pull llama3.2

# Verify the model is available
ollama list
```

## CLI configuration

Configure Autohand to use Ollama as your model provider in `~/.autohand/config.json`:

``` json
{
  "provider": "ollama",
  "ollama": {
    "enabled": true,
    "baseUrl": "http://localhost:11434",
    "model": "codellama:13b",
    "contextLength": 8192,
    "keepAlive": "5m",
    "numCtx": 4096,
    "temperature": 0.7
  }
}
```

Or set environment variables:

``` bash
# Set Ollama as the default provider
export AUTOHAND_PROVIDER="ollama"
export OLLAMA_HOST="http://localhost:11434"
export AUTOHAND_MODEL="codellama:13b"
```

### Configuration options

| Option | Description | Default |
|---|---|---|
| baseUrl | Ollama server URL | http://localhost:11434 |
| model | Model name to use | llama3.2 |
| contextLength | Maximum context window | Model default |
| keepAlive | How long to keep model in memory | 5m |
| numCtx | Context size for generation | 4096 |
| temperature | Response randomness (0-1) | 0.7 |
| numGpu | Number of GPU layers to use | Auto-detect |

## Available models

Ollama supports a wide range of open-source models. Here are the recommended ones for coding tasks:

### Coding models

| Model | Size | RAM | Best for |
|---|---|---|---|
| codellama:7b | 3.8GB | 8GB | Fast code completion |
| codellama:13b | 7.4GB | 16GB | Balanced performance |
| codellama:34b | 19GB | 32GB | Complex reasoning |
| deepseek-coder:6.7b | 3.8GB | 8GB | Code generation |
| deepseek-coder:33b | 19GB | 32GB | Advanced coding |
| qwen2.5-coder:7b | 4.7GB | 8GB | Multi-language coding |

### General-purpose models

| Model | Size | RAM | Best for |
|---|---|---|---|
| llama3.2:3b | 2.0GB | 4GB | Quick responses |
| llama3.2:latest | 4.7GB | 8GB | General tasks |
| mistral:7b | 4.1GB | 8GB | Efficient reasoning |
| mixtral:8x7b | 26GB | 48GB | Expert-level tasks |

### Pull and switch models

``` bash
# Pull multiple models
ollama pull codellama:13b
ollama pull deepseek-coder:6.7b
ollama pull llama3.2

# List installed models
ollama list

# Switch models in Autohand
autohand config set ollama.model "deepseek-coder:6.7b"

# Or use the /model command in a session
/model ollama/deepseek-coder:6.7b
```

## GPU acceleration

GPU acceleration significantly improves inference speed. Ollama automatically detects and uses available GPUs.

### macOS (Apple Silicon)

Apple Silicon Macs use Metal for GPU acceleration automatically. No additional configuration needed.

``` bash
# Verify Metal is being used
ollama run llama3.2 --verbose

# Check GPU memory usage
sudo powermetrics --samplers gpu_power -i 1000 -n 1
```

### Linux (NVIDIA)

Install NVIDIA drivers and CUDA toolkit:

``` bash
# Ubuntu/Debian
sudo apt install nvidia-driver-535 nvidia-cuda-toolkit

# Verify CUDA is available
nvidia-smi

# Ollama will automatically use CUDA
ollama run llama3.2 --verbose
```

### Linux (AMD)

Install ROCm for AMD GPU support:

``` bash
# Install ROCm
sudo apt install rocm-hip-libraries

# Set environment variable
export HSA_OVERRIDE_GFX_VERSION=10.3.0

# Run Ollama with ROCm
ollama run llama3.2 --verbose
```

### Windows (NVIDIA)

Ollama on Windows automatically uses NVIDIA GPUs with CUDA support. Ensure you have the latest drivers installed.

``` powershell
# Check GPU is detected
nvidia-smi

# Ollama uses GPU by default
ollama run llama3.2 --verbose
```

### Configure GPU layers

Control how many model layers run on GPU:

``` json
{
  "ollama": {
    "numGpu": 35,
    "numThread": 8
  }
}
```

## Commands

Use these commands to interact with Ollama through Autohand:

### Switch to Ollama

``` bash
# Start Autohand with Ollama
autohand --provider ollama --model codellama:13b

# Or use the /model command during a session
/model ollama/codellama:13b
```

### Model management

``` bash
# List available models
ollama list

# Show model details
ollama show codellama:13b

# Remove a model
ollama rm codellama:7b

# Copy/rename a model
ollama cp codellama:13b mycodellama
```

### Server management

``` bash
# Start Ollama server
ollama serve

# Check server status
curl http://localhost:11434/api/tags

# List running models
curl http://localhost:11434/api/ps
```

## Memory management

Optimize memory usage for your hardware configuration.

### Context window

Larger context windows use more memory. Adjust based on your RAM:

``` json
{
  "ollama": {
    "numCtx": 4096,
    "numBatch": 512
  }
}
```

### Model persistence

Control how long models stay loaded in memory:

``` json
{
  "ollama": {
    "keepAlive": "5m"
  }
}
```

| Value | Behavior |
|---|---|
| 0 | Unload immediately after request |
| 5m | Keep loaded for 5 minutes |
| 1h | Keep loaded for 1 hour |
| -1 | Keep loaded indefinitely |

### Unload models

``` bash
# Unload all models from memory
curl -X POST http://localhost:11434/api/generate \
  -d '{"model": "codellama:13b", "keep_alive": 0}'
```

## Remote Ollama server

Run Ollama on a remote machine with more powerful hardware.

### Server setup

On the remote machine, configure Ollama to accept external connections:

``` bash
# Set the host to listen on all interfaces
export OLLAMA_HOST=0.0.0.0:11434

# Or edit the systemd service
sudo systemctl edit ollama.service

# Add this content:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

# Restart the service
sudo systemctl restart ollama
```

### Client configuration

Configure Autohand to connect to the remote server:

``` json
{
  "ollama": {
    "baseUrl": "http://192.168.1.100:11434",
    "model": "codellama:13b"
  }
}
```

**Security note:** Only expose Ollama on trusted networks. Consider using SSH tunneling or a VPN for remote access over the internet.

## Best practices

-   **Match model to task**: Use coding-specific models like CodeLlama for code tasks and general models for documentation.
-   **Size appropriately**: Start with smaller models (7B) for quick responses, move to larger ones (13B+) for complex reasoning.
-   **Manage memory**: Set `keepAlive` to unload models when not in use on memory-constrained systems.
-   **Use GPU acceleration**: GPU significantly speeds up inference. Prioritize VRAM over CPU RAM.
-   **Pre-pull models**: Download models before starting work to avoid wait times.
-   **Monitor resources**: Watch memory and GPU usage to avoid system slowdowns.

### Recommended configurations

#### 8GB RAM (no GPU)

``` json
{
  "ollama": {
    "model": "codellama:7b",
    "numCtx": 2048,
    "keepAlive": "1m"
  }
}
```

#### 16GB RAM + GPU

``` json
{
  "ollama": {
    "model": "codellama:13b",
    "numCtx": 4096,
    "keepAlive": "10m"
  }
}
```

#### 32GB+ RAM + GPU

``` json
{
  "ollama": {
    "model": "deepseek-coder:33b",
    "numCtx": 8192,
    "keepAlive": "30m"
  }
}
```

## Troubleshooting

### Common issues

| Issue | Solution |
|---|---|
| Connection refused | Start Ollama server: ollama serve |
| Model not found | Pull the model: ollama pull <model> |
| Out of memory | Use a smaller model or reduce numCtx |
| Slow responses | Enable GPU acceleration or use a smaller model |
| GPU not detected | Install proper drivers (CUDA, ROCm, or Metal) |

### Check server status

``` bash
# Test the API endpoint
curl http://localhost:11434/api/tags

# Check running models
curl http://localhost:11434/api/ps

# View Ollama logs (Linux)
journalctl -u ollama -f
```

### Debug mode

Run with verbose output to diagnose issues:

``` bash
# Run Autohand with debug logging
AUTOHAND_DEBUG=true autohand --provider ollama

# Run Ollama with verbose output
ollama run codellama:13b --verbose
```

### Reset Ollama

``` bash
# Stop the service
sudo systemctl stop ollama

# Clear the cache (Linux/macOS)
rm -rf ~/.ollama/models/.tmp

# Restart
sudo systemctl start ollama
```