> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# LlamaCpp

> Run local models with LlamaCpp in Agno agents.

v2.0.7 (Introduced in v2.0.7)

[LlamaCpp](https://github.com/ggerganov/llama.cpp) runs large language models locally with efficient inference. LlamaCpp supports multiple open-source models and provides an OpenAI-compatible API server.

LlamaCpp supports a wide variety of models in GGUF format. You can find models on Hugging Face, including the default `ggml-org/gpt-oss-20b-GGUF` used in the examples below.

Model recommendations:

### Google Gemma Models

* `google/gemma-2b-it-GGUF` - Lightweight 2B parameter model for resource-constrained environments
* `google/gemma-7b-it-GGUF` - Balanced 7B model for general tasks
* `ggml-org/gemma-3-1b-it-GGUF` - Gemma 3 series, efficient for everyday use

### Default Options

* `ggml-org/gpt-oss-20b-GGUF` - Default model for general use cases
* Models with different quantizations (Q4\_K\_M, Q8\_0, etc.) for different speed/quality tradeoffs

## Set up LlamaCpp

### Install LlamaCpp

First, install LlamaCpp following the [official installation guide](https://github.com/ggerganov/llama.cpp):

```bash install theme={null}
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release
```

Or using package managers:

```bash brew install theme={null}
# macOS with Homebrew
brew install llama.cpp
```

### Download a Model

Download a model in GGUF format following the [llama.cpp model guide](https://github.com/ggerganov/llama.cpp#obtaining-and-quantizing-models). For the examples below, we use `ggml-org/gpt-oss-20b-GGUF`.

### Start the Server

Start the LlamaCpp server with your model:

```bash start server theme={null}
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 0 --jinja -ub 2048 -b 2048
```

This starts the server at `http://127.0.0.1:8080` with OpenAI-compatible endpoints.

## Example

After starting the LlamaCpp server, use the `LlamaCpp` model class to access it. It uses the OpenAI client, so install the `openai` package:

```bash theme={null}
uv pip install -U openai agno
```

```python agent.py theme={null}
from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp

agent = Agent(
    model=LlamaCpp(id="ggml-org/gpt-oss-20b-GGUF"),
    markdown=True
)

# Print the response in the terminal
agent.print_response("Share a 2 sentence horror story.")
```

## Configuration

The `LlamaCpp` model supports customizing the server URL and model ID:

```python custom_config.py theme={null}
from agno.agent import Agent
from agno.models.llama_cpp import LlamaCpp

# Custom server configuration
agent = Agent(
    model=LlamaCpp(
        id="your-custom-model",
        base_url="http://localhost:8080/v1",  # Custom server URL
    ),
    markdown=True
)
```

> [!NOTE]
>
> View more examples [here](https://agno-v2.mintlify.site/models/providers/local/llama-cpp/usage/basic.md).

## Params

| Parameter | Type | Default | Description |
| - | - | - | - |
| `id` | `str` | `"ggml-org/gpt-oss-20b-GGUF"` | The identifier for the Llama.cpp model |
| `name` | `str` | `"LlamaCpp"` | The name of the model |
| `provider` | `str` | `"LlamaCpp"` | The provider of the model |
| `base_url` | `str` | `"http://127.0.0.1:8080/v1"` | The base URL for the Llama.cpp server |
| `api_key` | `Optional[str]` | `"not-provided"` | The API key (usually not needed for local Llama.cpp) |
| `temperature` | `Optional[float]` | `None` | Sampling temperature (0.0 to 2.0) |
| `top_p` | `Optional[float]` | `None` | Top-p sampling parameter |

`LlamaCpp` is a subclass of the [OpenAILike](https://agno-v2.mintlify.site/models/providers/openai-like.md) class and has access to the same params.

## Server Configuration

The LlamaCpp server supports many configuration options:

### Common Server Options

* `--ctx-size`: Context size (0 loads the size from the model)
* `--batch-size`, `-b`: Batch size for prompt processing
* `--ubatch-size`, `-ub`: Physical batch size for prompt processing
* `--threads`, `-t`: Number of threads to use
* `--host`: IP address to listen on (default: 127.0.0.1)
* `--port`: Port to listen on (default: 8080)

### Model Options

* `--model`, `-m`: Model file path
* `--hf-repo`: Hugging Face model repository
* `--jinja`: Use Jinja templating for chat formatting

For a complete list of server options, run `llama-server --help`.

## Performance Optimization

### Hardware Acceleration

LlamaCpp supports various acceleration backends:

```bash gpu acceleration theme={null}
# NVIDIA GPU (CUDA)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Apple Metal (enabled by default on macOS)
cmake -B build
cmake --build build --config Release

# Vulkan
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release
```

### Model Quantization

Use quantized models for better performance:

* `Q4_K_M`: Balanced size and quality
* `Q8_0`: Higher quality, larger size
* `Q2_K`: Smallest size, lower quality

## Troubleshooting

### Server Connection Issues

Ensure the LlamaCpp server is running and accessible:

```bash check server theme={null}
curl http://127.0.0.1:8080/v1/models
```

### Model Loading Problems

* Verify the model file exists and is in GGUF format
* Check available memory for large models
* Ensure the model is compatible with your LlamaCpp version

### Performance Issues

* Adjust batch sizes (`-b`, `-ub`) based on your hardware
* Use GPU acceleration if available
* Consider using quantized models for faster inference


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.