---
title: "MLX Integration"
source: https://docs.autohand.ai/integrations/mlx
---

# MLX

Run local AI models on Apple Silicon with MLX for fast, private inference. Native Metal acceleration delivers exceptional performance on M1, M2, M3, and M4 chips.

**Version requirement:** MLX support requires Autohand CLI v0.7.0 or later. Run `autohand --version` to check your version. Update with `npm update -g autohand` or `bun update -g autohand`.

## Overview

[MLX](https://opensource.apple.com/projects/mlx/) is Apple's machine learning framework optimized for Apple Silicon. Built by Apple's machine learning research team and available on [GitHub](https://github.com/ml-explore/mlx), MLX provides a NumPy-like API with automatic differentiation, lazy computation, and unified memory. When integrated with Autohand, you get:

-   Native Metal acceleration on M1, M2, M3, and M4 chips
-   Unified memory architecture for efficient model loading
-   Complete privacy since your code never leaves your machine
-   No API costs with unlimited local inference
-   Low latency responses from your local hardware
-   Support for popular open-source models in MLX format

**Apple Silicon only:** MLX is designed for Macs with Apple Silicon (M1, M2, M3, M4). It will not work on Intel-based Macs or other platforms. For those systems, use [Ollama](/docs/integrations/ollama.html) or [llama.cpp](/docs/integrations/llama-cpp.html) instead.

## Requirements

Before setting up MLX, make sure you meet these requirements:

### Hardware

-   Mac with Apple Silicon (M1, M2, M3, or M4 series)
-   Minimum 8GB unified memory (16GB+ recommended)
-   macOS Sonoma 14.0 or later recommended

### Software

-   Python 3.9 or later
-   pip or uv package manager
-   Autohand CLI v0.7.0 or later

### Memory recommendations

| Unified Memory | Recommended Models |
|---|---|
| 8GB | 3B-7B parameter models (4-bit quantized) |
| 16GB | 7B-13B parameter models |
| 32GB | 13B-34B parameter models |
| 64GB+ | 34B-70B+ parameter models |

## Installation

Install the [MLX LM](https://github.com/ml-explore/mlx-lm) package to run a local inference server.

### Install mlx-lm

The [mlx-lm](https://github.com/ml-explore/mlx-lm) package provides both the inference engine and an OpenAI-compatible server:

``` bash
# Using pip
pip install mlx-lm

# Using uv (faster)
uv pip install mlx-lm

# Verify installation
python -c "import mlx_lm; print('MLX LM installed successfully')"
```

### Download a model

MLX models are available on Hugging Face. Download one optimized for coding:

``` bash
# Download a coding-optimized model
mlx_lm.convert --hf-path codellama/CodeLlama-7b-Instruct-hf -q

# Or download a pre-converted MLX model
pip install huggingface_hub
huggingface-cli download mlx-community/CodeLlama-7b-Instruct-mlx
```

### Start the server

Run the OpenAI-compatible API server:

``` bash
# Start the server with your model
mlx_lm.server --model mlx-community/CodeLlama-7b-Instruct-mlx

# Or specify a local model path
mlx_lm.server --model ./models/codellama-7b-mlx

# The server runs on http://localhost:8080 by default
```

### Verify the server

``` bash
# Check if the server is running
curl http://localhost:8080/v1/models

# Test a completion
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'
```

## CLI configuration

Configure Autohand to use MLX as your model provider in `~/.autohand/config.json`:

``` json
{
  "provider": "mlx",
  "mlx": {
    "baseUrl": "http://localhost:8080",
    "model": "codellama-7b-instruct",
    "temperature": 0.7,
    "maxTokens": 4096
  }
}
```

Or use YAML format in `~/.autohand/config.yaml`:

``` yaml
provider: mlx
mlx:
  baseUrl: http://localhost:8080
  model: codellama-7b-instruct
  temperature: 0.7
  maxTokens: 4096
```

### Configuration options

| Option | Description | Default |
|---|---|---|
| baseUrl | MLX server URL | http://localhost:8080 |
| port | Server port (alternative to baseUrl) | 8080 |
| model | Model name for display | mlx-model |
| temperature | Response randomness (0-1) | 0.7 |
| maxTokens | Maximum tokens to generate | 4096 |

### Environment variables

You can also configure MLX using environment variables:

``` bash
# Set MLX as the default provider
export AUTOHAND_PROVIDER="mlx"
export AUTOHAND_MODEL="codellama-7b-instruct"
```

## Available models

The MLX community maintains optimized versions of popular models on Hugging Face.

### Coding models

| Model | Parameters | Memory | Best for |
|---|---|---|---|
| mlx-community/CodeLlama-7b-Instruct-mlx | 7B | 8GB | Fast code completion |
| mlx-community/CodeLlama-13b-Instruct-mlx | 13B | 16GB | Balanced performance |
| mlx-community/CodeLlama-34b-Instruct-mlx | 34B | 32GB | Complex reasoning |
| mlx-community/deepseek-coder-6.7b-instruct-mlx | 6.7B | 8GB | Code generation |
| mlx-community/Qwen2.5-Coder-7B-Instruct-4bit | 7B | 8GB | Multi-language coding |

### General-purpose models

| Model | Parameters | Memory | Best for |
|---|---|---|---|
| mlx-community/Llama-3.2-3B-Instruct-4bit | 3B | 4GB | Quick responses |
| mlx-community/Meta-Llama-3.1-8B-Instruct-4bit | 8B | 8GB | General tasks |
| mlx-community/Mistral-7B-Instruct-v0.3-4bit | 7B | 8GB | Efficient reasoning |
| mlx-community/Mixtral-8x7B-Instruct-v0.1-4bit | 47B | 32GB | Expert-level tasks |

### Download and run models

``` bash
# Start server with a Hugging Face model (downloads automatically)
mlx_lm.server --model mlx-community/CodeLlama-7b-Instruct-mlx

# Use a quantized model for lower memory usage
mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

# Switch models in Autohand during a session
/model mlx/codellama-13b-instruct
```

**Finding models:** Browse the [mlx-community](https://huggingface.co/mlx-community) on Hugging Face for the latest MLX-optimized models. Look for models with "mlx" or "4bit" in the name.

## Server options

Fine-tune the mlx-lm server for optimal performance:

### Basic options

``` bash
mlx_lm.server \
  --model mlx-community/CodeLlama-7b-Instruct-mlx \
  --host 127.0.0.1 \
  --port 8080
```

### Memory and performance

``` bash
# Use 4-bit quantization for lower memory
mlx_lm.server \
  --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

# Specify cache directory for models
mlx_lm.server \
  --model mlx-community/CodeLlama-7b-Instruct-mlx \
  --cache-dir ~/.cache/mlx-models
```

### Server options reference

| Option | Description |
|---|---|
| --model | Hugging Face model ID or local path |
| --host | Server host address |
| --port | Server port (default: 8080) |
| --cache-dir | Model cache directory |

## Commands

Use these commands to interact with MLX through Autohand:

### Switch to MLX

``` bash
# Start Autohand with MLX
autohand --provider mlx --model codellama-7b-instruct

# Or use the /model command during a session
/model mlx/codellama-7b-instruct
```

### Check connection

``` bash
# Test the MLX server
curl http://localhost:8080/v1/models

# Check server health
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "test"}], "max_tokens": 5}'
```

## Converting models

Convert Hugging Face models to MLX format for optimal performance:

### Convert a model

``` bash
# Convert with default settings
mlx_lm.convert --hf-path codellama/CodeLlama-7b-Instruct-hf

# Convert with 4-bit quantization (recommended for most users)
mlx_lm.convert --hf-path codellama/CodeLlama-7b-Instruct-hf -q

# Specify output directory
mlx_lm.convert \
  --hf-path codellama/CodeLlama-7b-Instruct-hf \
  --mlx-path ./models/codellama-7b-mlx \
  -q
```

### Quantization options

| Option | Bits | Memory | Quality |
|---|---|---|---|
| None (default) | 16 | Highest | Best |
| -q | 4 | Lowest | Good |
| --q-bits 8 | 8 | Medium | Better |

## Best practices

-   **Use quantized models**: 4-bit quantized models offer the best balance of quality and memory usage on Apple Silicon.
-   **Match model to memory**: Choose models that fit comfortably in your unified memory with room for the OS and other apps.
-   **Keep the server running**: Start the MLX server once and keep it running to avoid model reload times.
-   **Use coding models for code**: CodeLlama and DeepSeek Coder models perform better on coding tasks than general-purpose models.
-   **Monitor Activity Monitor**: Check memory pressure in Activity Monitor to ensure you're not swapping.

### Recommended configurations

#### 8GB unified memory (M1/M2 base)

``` json
{
  "provider": "mlx",
  "mlx": {
    "model": "codellama-7b-instruct-4bit",
    "maxTokens": 2048
  }
}
```

#### 16GB unified memory (M1/M2 Pro)

``` json
{
  "provider": "mlx",
  "mlx": {
    "model": "codellama-13b-instruct",
    "maxTokens": 4096
  }
}
```

#### 32GB+ unified memory (M1/M2/M3 Max)

``` json
{
  "provider": "mlx",
  "mlx": {
    "model": "codellama-34b-instruct",
    "maxTokens": 8192
  }
}
```

## Troubleshooting

### Common issues

| Issue | Solution |
|---|---|
| MLX not supported error | MLX only works on Apple Silicon Macs. Use Ollama or llama.cpp instead. |
| Connection refused | Start the MLX server: mlx_lm.server --model your-model |
| Out of memory | Use a smaller or 4-bit quantized model |
| Slow responses | Close other memory-intensive apps or use a smaller model |
| Model not found | Check the model path or Hugging Face model ID |

### Check platform support

``` bash
# Verify Apple Silicon
uname -m
# Should output: arm64

# Check macOS version
sw_vers
# Should show macOS 14.0 or later for best performance
```

### Debug mode

``` bash
# Run Autohand with debug logging
AUTOHAND_DEBUG=true autohand --provider mlx

# Check MLX server logs
mlx_lm.server --model your-model 2>&1 | tee mlx-server.log
```

### Memory pressure

If you see yellow or red memory pressure in Activity Monitor:

-   Switch to a smaller model (7B instead of 13B)
-   Use 4-bit quantization
-   Close other memory-intensive applications
-   Restart the MLX server to clear cached data

## MLX vs other providers

Choose the right local provider for your setup:

| Feature | MLX | Ollama | llama.cpp |
|---|---|---|---|
| Platform | Apple Silicon only | All platforms | All platforms |
| Setup | Python package | Single binary | Build from source |
| Performance on Mac | Excellent | Good | Good |
| Memory efficiency | Excellent | Good | Good |
| Model format | MLX/Safetensors | Ollama format | GGUF |
| Best for | Apple Silicon users | Easy setup | Maximum control |

**Recommendation:** If you're on Apple Silicon and want the best performance, use MLX. If you need cross-platform support or easier setup, use [Ollama](/docs/integrations/ollama.html).