> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl4AI

> Crawl4aiTools exposes a crawl function that fetches one or more URLs with Crawl4ai's AsyncWebCrawler and returns extracted markdown content, optionally filtered by a search query using BM25.

**Crawl4aiTools** enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.

## Prerequisites

The following example requires the `crawl4ai` library.

```shell theme={null}
uv pip install -U crawl4ai openai
```

## Example

The following agent will scrape the content from the [https://github.com/agno-agi/agno](https://github.com/agno-agi/agno) webpage:

```python theme={null}
from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools

agent = Agent(tools=[Crawl4aiTools(max_length=None)])
agent.print_response("Tell me about https://github.com/agno-agi/agno.")
```

## Toolkit Params

| Parameter | Type | Default | Description |
| - | - | - | - |
| `max_length` | `Optional[int]` | `5000` | Specifies the maximum length of the text from the webpage to be returned. |
| `timeout` | `int` | `60` | Timeout in seconds for web crawling operations. |
| `use_pruning` | `bool` | `False` | Enable content pruning to remove less relevant content. |
| `pruning_threshold` | `float` | `0.48` | Threshold for content pruning relevance scoring. |
| `bm25_threshold` | `float` | `1.0` | BM25 scoring threshold for content relevance. |
| `headless` | `bool` | `True` | Run browser in headless mode. |
| `wait_until` | `str` | `"domcontentloaded"` | Browser wait condition before crawling (e.g., "domcontentloaded", "load", "networkidle"). |
| `proxy_config` | `Optional[Dict[str, Any]]` | `None` | Proxy configuration passed to the browser. |
| `enable_crawl` | `bool` | `True` | Enable the web crawling functionality. |
| `all` | `bool` | `False` | Enable all available functions. When True, all enable flags are ignored. |

## Toolkit Functions

| Function | Description |
| - | - |
| `crawl` | Crawls one or more URLs and returns the extracted content, optionally filtered by a `search_query` for relevance-based extraction. |

## Developer Resources

* [Tools](https://github.com/agno-agi/agno/blob/main/libs/agno/agno/tools/crawl4ai.py)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.