Ollama Integration - Autohand Docs

Overview

Ollama provides a simple way to run large language models locally. When integrated with Autohand, you get:

Complete privacy - your code never leaves your machine
Offline capability - work without internet connectivity
No API costs - run unlimited queries at no charge
Low latency - responses come from your local hardware
Model flexibility - choose from dozens of open-source models

Hardware requirements: Ollama works best with at least 8GB RAM for 7B models, 16GB for 13B models, and 32GB+ for larger models. GPU acceleration dramatically improves performance.

Installation

Install Ollama on your operating system to get started with local models.

macOS

Download and install Ollama using Homebrew or the official installer:

# Using Homebrew (recommended)
brew install ollama

# Or download from ollama.com
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

Start the Ollama service:

# Start Ollama server (runs in background)
ollama serve

# Or run as a macOS service
brew services start ollama

Linux

Install Ollama using the official install script:

# Download and install
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Start the service
sudo systemctl start ollama

# Enable on boot
sudo systemctl enable ollama

For manual installation or specific distributions:

# Ubuntu/Debian
sudo apt update
sudo apt install -y curl
curl -fsSL https://ollama.com/install.sh | sh

# Fedora/RHEL
sudo dnf install -y curl
curl -fsSL https://ollama.com/install.sh | sh

# Arch Linux (AUR)
yay -S ollama

Windows

Download the Windows installer from the Ollama website or use winget:

# Using winget
winget install Ollama.Ollama

# Or download the installer from:
# https://ollama.com/download/windows

After installation, Ollama runs as a Windows service automatically. Verify by opening PowerShell:

# Check Ollama is running
ollama --version

# List available models
ollama list

Pull your first model

Download a model to use with Autohand:

# Pull a coding-optimized model
ollama pull codellama:13b

# Or use Llama 3.2 for general tasks
ollama pull llama3.2

# Verify the model is available
ollama list

CLI configuration

Configure Autohand to use Ollama as your model provider in ~/.autohand/config.json:

{
  "provider": "ollama",
  "ollama": {
    "enabled": true,
    "baseUrl": "http://localhost:11434",
    "model": "codellama:13b",
    "contextLength": 8192,
    "keepAlive": "5m",
    "numCtx": 4096,
    "temperature": 0.7
  }
}

Or set environment variables:

# Set Ollama as the default provider
export AUTOHAND_PROVIDER="ollama"
export OLLAMA_HOST="http://localhost:11434"
export AUTOHAND_MODEL="codellama:13b"

Configuration options

Option	Description	Default
`baseUrl`	Ollama server URL	`http://localhost:11434`
`model`	Model name to use	`llama3.2`
`contextLength`	Maximum context window	Model default
`keepAlive`	How long to keep model in memory	`5m`
`numCtx`	Context size for generation	`4096`
`temperature`	Response randomness (0-1)	`0.7`
`numGpu`	Number of GPU layers to use	Auto-detect

Available models

Ollama supports a wide range of open-source models. Here are the recommended ones for coding tasks:

Coding models

Model	Size	RAM	Best for
`codellama:7b`	3.8GB	8GB	Fast code completion
`codellama:13b`	7.4GB	16GB	Balanced performance
`codellama:34b`	19GB	32GB	Complex reasoning
`deepseek-coder:6.7b`	3.8GB	8GB	Code generation
`deepseek-coder:33b`	19GB	32GB	Advanced coding
`qwen2.5-coder:7b`	4.7GB	8GB	Multi-language coding

General-purpose models

Model	Size	RAM	Best for
`llama3.2:3b`	2.0GB	4GB	Quick responses
`llama3.2:latest`	4.7GB	8GB	General tasks
`mistral:7b`	4.1GB	8GB	Efficient reasoning
`mixtral:8x7b`	26GB	48GB	Expert-level tasks

Pull and switch models

# Pull multiple models
ollama pull codellama:13b
ollama pull deepseek-coder:6.7b
ollama pull llama3.2

# List installed models
ollama list

# Switch models in Autohand
autohand config set ollama.model "deepseek-coder:6.7b"

# Or use the /model command in a session
/model ollama/deepseek-coder:6.7b

GPU acceleration

GPU acceleration significantly improves inference speed. Ollama automatically detects and uses available GPUs.

macOS (Apple Silicon)

Apple Silicon Macs use Metal for GPU acceleration automatically. No additional configuration needed.

# Verify Metal is being used
ollama run llama3.2 --verbose

# Check GPU memory usage
sudo powermetrics --samplers gpu_power -i 1000 -n 1

Linux (NVIDIA)

Install NVIDIA drivers and CUDA toolkit:

# Ubuntu/Debian
sudo apt install nvidia-driver-535 nvidia-cuda-toolkit

# Verify CUDA is available
nvidia-smi

# Ollama will automatically use CUDA
ollama run llama3.2 --verbose

Linux (AMD)

Install ROCm for AMD GPU support:

# Install ROCm
sudo apt install rocm-hip-libraries

# Set environment variable
export HSA_OVERRIDE_GFX_VERSION=10.3.0

# Run Ollama with ROCm
ollama run llama3.2 --verbose

Windows (NVIDIA)

Ollama on Windows automatically uses NVIDIA GPUs with CUDA support. Ensure you have the latest drivers installed.

# Check GPU is detected
nvidia-smi

# Ollama uses GPU by default
ollama run llama3.2 --verbose

Configure GPU layers

Control how many model layers run on GPU:

{
  "ollama": {
    "numGpu": 35,
    "numThread": 8
  }
}

Commands

Use these commands to interact with Ollama through Autohand:

Switch to Ollama

# Start Autohand with Ollama
autohand --provider ollama --model codellama:13b

# Or use the /model command during a session
/model ollama/codellama:13b

Model management

# List available models
ollama list

# Show model details
ollama show codellama:13b

# Remove a model
ollama rm codellama:7b

# Copy/rename a model
ollama cp codellama:13b mycodellama

Server management

# Start Ollama server
ollama serve

# Check server status
curl http://localhost:11434/api/tags

# List running models
curl http://localhost:11434/api/ps

Memory management

Optimize memory usage for your hardware configuration.

Context window

Larger context windows use more memory. Adjust based on your RAM:

{
  "ollama": {
    "numCtx": 4096,
    "numBatch": 512
  }
}

Model persistence

Control how long models stay loaded in memory:

{
  "ollama": {
    "keepAlive": "5m"
  }
}

Value	Behavior
`0`	Unload immediately after request
`5m`	Keep loaded for 5 minutes
`1h`	Keep loaded for 1 hour
`-1`	Keep loaded indefinitely

Unload models

# Unload all models from memory
curl -X POST http://localhost:11434/api/generate \
  -d '{"model": "codellama:13b", "keep_alive": 0}'

Remote Ollama server

Run Ollama on a remote machine with more powerful hardware.

Server setup

On the remote machine, configure Ollama to accept external connections:

# Set the host to listen on all interfaces
export OLLAMA_HOST=0.0.0.0:11434

# Or edit the systemd service
sudo systemctl edit ollama.service

# Add this content:
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

# Restart the service
sudo systemctl restart ollama

Client configuration

Configure Autohand to connect to the remote server:

{
  "ollama": {
    "baseUrl": "http://192.168.1.100:11434",
    "model": "codellama:13b"
  }
}

Security note: Only expose Ollama on trusted networks. Consider using SSH tunneling or a VPN for remote access over the internet.

Best practices

Match model to task: Use coding-specific models like CodeLlama for code tasks and general models for documentation.
Size appropriately: Start with smaller models (7B) for quick responses, move to larger ones (13B+) for complex reasoning.
Manage memory: Set keepAlive to unload models when not in use on memory-constrained systems.
Use GPU acceleration: GPU significantly speeds up inference. Prioritize VRAM over CPU RAM.
Pre-pull models: Download models before starting work to avoid wait times.
Monitor resources: Watch memory and GPU usage to avoid system slowdowns.

Recommended configurations

8GB RAM (no GPU)

{
  "ollama": {
    "model": "codellama:7b",
    "numCtx": 2048,
    "keepAlive": "1m"
  }
}

16GB RAM + GPU

{
  "ollama": {
    "model": "codellama:13b",
    "numCtx": 4096,
    "keepAlive": "10m"
  }
}

32GB+ RAM + GPU

{
  "ollama": {
    "model": "deepseek-coder:33b",
    "numCtx": 8192,
    "keepAlive": "30m"
  }
}

Troubleshooting

Common issues

Issue	Solution
Connection refused	Start Ollama server: `ollama serve`
Model not found	Pull the model: `ollama pull <model>`
Out of memory	Use a smaller model or reduce `numCtx`
Slow responses	Enable GPU acceleration or use a smaller model
GPU not detected	Install proper drivers (CUDA, ROCm, or Metal)

Check server status

# Test the API endpoint
curl http://localhost:11434/api/tags

# Check running models
curl http://localhost:11434/api/ps

# View Ollama logs (Linux)
journalctl -u ollama -f

Debug mode

Run with verbose output to diagnose issues:

# Run Autohand with debug logging
AUTOHAND_DEBUG=true autohand --provider ollama

# Run Ollama with verbose output
ollama run codellama:13b --verbose

Reset Ollama

# Stop the service
sudo systemctl stop ollama

# Clear the cache (Linux/macOS)
rm -rf ~/.ollama/models/.tmp

# Restart
sudo systemctl start ollama