> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Website Reader

> Crawl websites and convert pages into knowledge base documents with WebsiteReader.

`WebsiteReader` crawls a starting page and discovered links, then creates documents from the extracted page text.

```python website_reader.py theme={null}
from agno.knowledge.reader.website_reader import WebsiteReader

reader = WebsiteReader(
    max_depth=2,
    max_links=5,
    allowed_hosts=["docs.agno.com"],
)

documents = reader.read("https://docs.agno.com/")
for document in documents:
    print(document.meta_data["url"], len(document.content))
```

## Run the Reader

<Snippet file="create-venv-step.mdx" />

**1. Install dependencies**

```bash theme={null}
uv pip install -U beautifulsoup4 agno
```

**2. Run the script**

```bash theme={null}
python website_reader.py
```

## Reader Parameters

<Snippet file="website-reader-reference.mdx" />

> [!WARNING]
>
> Set `allowed_hosts` for every crawl in v2.7.2. The fallback primary-domain check uses a hostname suffix and can admit unrelated hosts whose names end with the same text.

`WebsiteReader.async_read()` performs the crawl with asynchronous HTTP requests.

## Next Steps

| Task | Guide |
| - | - |
| Restrict outbound requests | [Restricting URL Fetches](https://agno-v2.mintlify.site/knowledge/concepts/readers/overview.md#restricting-url-fetches) |
| Configure content chunking | [Chunking Overview](https://agno-v2.mintlify.site/knowledge/concepts/chunking/overview.md) |
| Inspect the complete API | [Website Reader Reference](https://agno-v2.mintlify.site/reference/knowledge/reader/website.md) |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.