Crawl4AI - Agno

Crawl4aiTools enable an Agent to perform web crawling and scraping tasks using the Crawl4ai library.

Prerequisites

The following example requires the crawl4ai library.

uv pip install -U crawl4ai openai

Example

The following agent will scrape the content from the https://github.com/agno-agi/agno webpage:

from agno.agent import Agent
from agno.tools.crawl4ai import Crawl4aiTools

agent = Agent(tools=[Crawl4aiTools(max_length=None)])
agent.print_response("Tell me about https://github.com/agno-agi/agno.")

Toolkit Params

Parameter	Type	Default	Description
`max_length`	`Optional[int]`	`5000`	Specifies the maximum length of the text from the webpage to be returned.
`timeout`	`int`	`60`	Timeout in seconds for web crawling operations.
`use_pruning`	`bool`	`False`	Enable content pruning to remove less relevant content.
`pruning_threshold`	`float`	`0.48`	Threshold for content pruning relevance scoring.
`bm25_threshold`	`float`	`1.0`	BM25 scoring threshold for content relevance.
`headless`	`bool`	`True`	Run browser in headless mode.
`wait_until`	`str`	`"domcontentloaded"`	Browser wait condition before crawling (e.g., “domcontentloaded”, “load”, “networkidle”).
`proxy_config`	`Optional[Dict[str, Any]]`	`None`	Proxy configuration passed to the browser.
`enable_crawl`	`bool`	`True`	Enable the web crawling functionality.
`all`	`bool`	`False`	Enable all available functions. When True, all enable flags are ignored.

Toolkit Functions

Function	Description
`crawl`	Crawls one or more URLs and returns the extracted content, optionally filtered by a `search_query` for relevance-based extraction.

Developer Resources

Tools