> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Spider

> SpiderTools wraps the Spider API to expose search_web, scrape, and crawl functions that return web search, scrape, and crawl results.

**SpiderTools** is an open-source web scraper and crawler that returns LLM-ready data. To start using Spider, you need an API key from the [Spider dashboard](https://spider.cloud).

## Prerequisites

The following example requires the `spider-client` library.

```shell theme={null}
uv pip install -U spider-client openai
```

## Example

The following agent will run a search query to get the latest news in USA and scrape the first search result. The agent will return the scraped data in markdown format.

```python cookbook/91_tools/spider_tools.py theme={null}
from agno.agent import Agent
from agno.tools.spider import SpiderTools

agent = Agent(tools=[SpiderTools()])
agent.print_response('Can you scrape the first search result from a search on "news in USA"?', markdown=True)
```

## Toolkit Params

| Parameter | Type | Default | Description |
| - | - | - | - |
| `max_results` | `Optional[int]` | `None` | Default maximum number of results. |
| `url` | `Optional[str]` | `None` | Default URL for operations. |
| `optional_params` | `Optional[dict]` | `None` | Additional parameters for operations. |
| `enable_search` | `bool` | `True` | Enable web search functionality. |
| `enable_scrape` | `bool` | `True` | Enable web scraping functionality. |
| `enable_crawl` | `bool` | `True` | Enable web crawling functionality. |
| `all` | `bool` | `False` | Enable all tools. Overrides individual flags when True. |

## Toolkit Functions

| Function | Description |
| - | - |
| `search_web` | Searches the web for the given query. Parameters include `query` (str) for the search query and `max_results` (int, default=5) for maximum results. Returns search results in JSON format. |
| `scrape` | Scrapes the content of a webpage. Parameters include `url` (str) for the URL of the webpage to scrape. Returns markdown of the webpage. |
| `crawl` | Crawls the web starting from a URL. Parameters include `url` (str) for the URL to crawl and `limit` (Optional\[int], default=10) for maximum pages to crawl. Returns crawl results in JSON format. |

## Developer Resources

* [Tools](https://github.com/agno-agi/agno/blob/main/libs/agno/agno/tools/spider.py)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.