> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM

> Run models served by vLLM's OpenAI-compatible API in Agno agents.

[vLLM](https://docs.vllm.ai/en/latest/) is a fast and easy-to-use library for LLM inference and serving, designed for high throughput and memory efficiency.

## Prerequisites

Install vLLM and start serving a model:

> [!WARNING]
>
> The command below requires a supported Linux GPU environment. Choose the installation path for your hardware and operating system in the [vLLM GPU installation guide](https://docs.vllm.ai/en/stable/getting_started/installation/gpu/) before running it.

```bash install vLLM theme={null}
uv pip install -U agno openai vllm --torch-backend=auto
```

```bash start vLLM server theme={null}
vllm serve Qwen/Qwen2.5-7B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --dtype float16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9
```

This spins up the vLLM server with an OpenAI-compatible API.

Set the `VLLM_API_KEY` environment variable. `VLLM` raises an error without it. If you started the server without `--api-key`, any value works:

```bash set API key theme={null}
export VLLM_API_KEY=xxx
```

> [!NOTE]
>
> `VLLM` connects to `http://localhost:8000/v1/` by default. Override with the `base_url` parameter or the `VLLM_BASE_URL` environment variable.

## Example

```python agent.py theme={null}
from agno.agent import Agent
from agno.models.vllm import VLLM

agent = Agent(
    model=VLLM(
        id="Qwen/Qwen2.5-7B-Instruct",
        base_url="http://localhost:8000/v1/",
    ),
    markdown=True
)

agent.print_response("Share a 2 sentence horror story.")
```

## Advanced Usage

### With Tools

vLLM models work with Agno tools:

```python with_tools.py theme={null}
from agno.agent import Agent
from agno.models.vllm import VLLM
from agno.tools.hackernews import HackerNewsTools

agent = Agent(
    model=VLLM(id="Qwen/Qwen2.5-7B-Instruct"),
    tools=[HackerNewsTools()],
    markdown=True
)

agent.print_response("What's the latest news about AI?")
```

> [!NOTE]
>
> View more examples [here](https://agno-v2.mintlify.site/models/providers/local/vllm/usage/tool-use.md).

For the full list of supported models, see the [vLLM documentation](https://docs.vllm.ai/en/latest/models/supported_models.html).

## Params

| Parameter | Type | Default | Description |
| - | - | - | - |
| `id` | `str` | `"not-set"` | The id of the model to use with vLLM |
| `name` | `str` | `"VLLM"` | The name of the model |
| `provider` | `str` | `"VLLM"` | The provider of the model |
| `api_key` | `Optional[str]` | `None` | The API key. Falls back to the `VLLM_API_KEY` environment variable and raises an error if neither is set |
| `base_url` | `Optional[str]` | `None` | The base URL for the vLLM server. Falls back to the `VLLM_BASE_URL` environment variable, then `"http://localhost:8000/v1/"` |
| `temperature` | `float` | `0.7` | Sampling temperature |
| `top_p` | `float` | `0.8` | Nucleus sampling probability |
| `presence_penalty` | `float` | `1.5` | Penalty for repeating tokens already present |
| `top_k` | `Optional[int]` | `None` | Top-k sampling, sent via `extra_body` |
| `enable_thinking` | `Optional[bool]` | `None` | Sets `chat_template_kwargs.enable_thinking` via `extra_body` |

`VLLM` is a subclass of the [OpenAILike](https://agno-v2.mintlify.site/models/providers/openai-like.md) class and has access to the same params.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.