Tavily Web Tools in ChatAgents: Search, Extract, and Crawl

ChatAgents gives the LangGraph ReAct agent three distinct web tools — TavilySearch, TavilyExtract, and TavilyCrawl. On every turn the agent reads the user’s question, reasons about which tool (or combination of tools) will best answer it, and invokes them in sequence. This ReAct loop — Thought → Action → Observation → repeat — means you never need to specify how to search; the agent picks the right tool for the job autonomously based on guidance embedded in the system prompt.

TavilySearch

TavilySearch is the agent’s starting point for almost every web query. Given a natural-language search string, it retrieves semantically ranked results from the public internet and returns titles, URLs, and content snippets. Purpose: Retrieve a ranked list of relevant web pages for a query — the fastest way to gather broad context before deciding whether deeper extraction is needed.

Parameters

Parameter	Type	Fast Mode	Deep Mode	Description
`max_results`	`int`	`3`	`5`	Number of results returned
`search_depth`	`str`	`"basic"`	`"advanced"`	`basic` is faster and cheaper; `advanced` is more thorough
`topic`	`str`	`"general"`	`"general"`	Index to search: `general`, `news`, or `finance`
`time_range`	`str \| None`	`None`	`None`	Recency filter: `day`, `week`, `month`, `year`, or `None`
`include_images`	`bool`	`False`	`True`	Whether to include image URLs in results
`include_answer`	`bool`	`False`	`False`	Always disabled — lets the LLM reason over raw results rather than use a pre-generated answer
`include_favicon`	`bool`	`True`	`True`	Include site favicon for UI display

ReAct Format

The system prompt teaches the agent to call TavilySearch using the ReAct action format. A typical turn looks like:

Question: What are the latest AI research papers?
Thought: I should search for recent AI research publications.
Action: TavilySearch
Action Input: "latest AI research papers 2024"
Observation: [ranked results with titles, URLs, snippets]
Thought: I now have enough to answer.
Final Answer: ...

Fast vs Deep configuration

# backend/agent.py — TavilySearch instantiation
search = TavilySearch(
    max_results=max_results,      # 3 (fast) or 5 (deep)
    tavily_api_key=api_key,
    include_favicon=True,
    search_depth=depth,           # "basic" (fast) or "advanced" (deep)
    include_answer=False,         # let the LLM reason, not Tavily
    topic=topic,
    include_images=include_images, # False (fast) or True (deep)
    # time_range injected only if explicitly set
)

TavilyExtract

TavilyExtract fetches the full page content from one or more specific URLs. Where TavilySearch returns snippets, TavilyExtract returns the complete article or documentation page — useful when a snippet is not enough to answer the question. Purpose: Extract full content from targeted URLs found during search, enabling the agent to read entire pages rather than brief excerpts.

Parameters

Parameter	Type	Fast Mode	Deep Mode	Description
`extract_depth`	`str`	`"basic"`	`"advanced"`	`basic` returns main content; `advanced` performs deeper DOM parsing
`include_favicon`	`bool`	`True`	`True`	Include site favicon for UI display
`include_images`	`bool`	`False`	`True`	Whether to include image URLs from the extracted page

Output Summarization

Raw extraction responses can be very large — entire articles or documentation pages sent back to the agent — which inflates token usage and latency. To address this, TavilyExtract is wrapped in a SummarizingTavilyExtract subclass defined in agent.py. During build_graph(), create_output_summarizer() is called once with the Claude Haiku summary_llm to produce an output_summarizer closure. The wrapper then calls that closure after every extraction to condense the raw content relative to the user’s original question:

# backend/agent.py — create_output_summarizer returns a closure
def create_output_summarizer(summary_llm: BaseChatModel) -> Callable[[str, str], dict]:
    def summarize_output(tool_output: str, user_message: str = "") -> dict:
        # Parse JSON or ast.literal_eval; extract urls, favicons, raw_content
        # Truncate raw_content to 3000 chars and summarize with summary_llm
        summary = summary_llm.invoke(summary_prompt).content
        return {"summary": summary, "urls": urls, "favicons": favicons}
    return summarize_output

# Called once per request in build_graph():
output_summarizer = create_output_summarizer(summary_llm)

# SummarizingTavilyExtract calls the returned closure on each invocation:
class SummarizingTavilyExtract(TavilyExtract):
    def _run(self, *args, **kwargs):
        kwargs.pop('run_manager', None)
        result = super()._run(*args, **kwargs)
        return output_summarizer(str(result), user_message)

The summarizer returns a structured dict containing the condensed summary, a list of source urls, and their favicons — which the Streamlit frontend uses to render source-link cards in the tool-call visualization.

Batching URLs

Never call TavilyExtract twice in a row. If you need content from multiple pages, pass all URLs in a single call. Consecutive extract calls waste API credits and violate the agent’s operational guidelines.

The system prompt explicitly enforces this rule:

Important guideline: You should never perform two extractions consecutively!
If you need to extract multiple pages, provide all URLs in the Action Input.

Action Input for a single URL:

Action: TavilyExtract
Action Input: ["https://example.com/page1"]

Action Input batching multiple URLs in one call:

Action: TavilyExtract
Action Input: ["https://example.com/page1", "https://example.com/page2"]

TavilyCrawl

TavilyCrawl starts from a root URL and automatically discovers and summarizes nested linked pages within the same domain. It is the right tool when the agent needs to explore a whole site rather than extract a single known page. Purpose: Deep-crawl a website from a starting URL to discover related pages and gather site-wide context without knowing individual page URLs in advance.

Parameters

Parameter	Type	Fast Mode	Deep Mode	Description
`limit`	`int`	`5`	`15`	Maximum number of pages to crawl from the starting URL
`include_favicon`	`bool`	`True`	`True`	Include site favicon for UI display

When to Use Crawl vs Extract

Scenario	Recommended tool
You have a specific URL and need its full text	`TavilyExtract`
You want to survey a whole site or doc hierarchy	`TavilyCrawl`
You need content from several unrelated pages	`TavilyExtract` (batch URLs)
You are discovering what pages exist on a domain	`TavilyCrawl`

The system prompt guidance reinforces this distinction:

TavilyCrawl — Given a starting URL, it finds all nested links and page summaries.
Useful for deep information discovery from a single source when we have a specific URL.
Action Input should be a URL (e.g., "https://blog.geekie.site")

Output Summarization

Like TavilyExtract, TavilyCrawl is wrapped in a SummarizingTavilyCrawl subclass that applies the same output_summarizer closure (created by create_output_summarizer()) to its output:

# backend/agent.py
class SummarizingTavilyCrawl(TavilyCrawl):
    def _run(self, *args, **kwargs):
        kwargs.pop('run_manager', None)
        result = super()._run(*args, **kwargs)
        return output_summarizer(str(result), user_message)

    async def _arun(self, *args, **kwargs):
        kwargs.pop('run_manager', None)
        result = await super()._arun(*args, **kwargs)
        return output_summarizer(str(result), user_message)

The summarizer condenses the crawled pages into a focused summary and extracts source URLs for the Streamlit tool-card display — so a 15-page crawl returns a tidy dict rather than a wall of raw HTML.

If you crawl a page, the agent already has a summary of all discovered sub-pages. There is no need to then extract or search those same pages individually — doing so just burns extra credits on content the agent already has.

Tool Selection Logic

The agent decides which tool to call by reasoning over the user’s message and the tool descriptions embedded in the system prompt. Both the Fast and Deep Thinking prompts describe each tool’s purpose and input format, then instruct the agent to follow the ReAct pattern:

Question: The input question you must answer
Thought: You should always think about what to do
Action: The action to take — one of TavilySearch, TavilyCrawl, or TavilyExtract
Action Input: The input to the action
Observation: The result of the action
... (Thought/Action/Action Input/Observation may repeat N times)
Thought: I now know the final answer
Final Answer: The final answer to the original input question

In practice the agent follows a natural pattern:

Unknown topic → start with TavilySearch to discover relevant URLs
Specific URL from search results → use TavilyExtract (batch if multiple)
Need to explore a whole site → use TavilyCrawl on the root URL
Answer already found → skip tools and go directly to Final Answer

The Deep Thinking prompt additionally caps tool use at 5 calls per query to prevent runaway credit consumption on complex research tasks.

Tool Call Visualization

Every tool invocation is reflected in the Streamlit UI in real time. The backend emits tool_start and tool_end NDJSON events for each tool call:

# app.py — tool_start event
yield json.dumps({
    "type": "tool_start",
    "tool_name": tool_name,          # e.g. "TavilySearch"
    "tool_type": tool_type,          # "search", "extract", or "crawl"
    "operation_index": operation_counter,
    "content": serializable_input,   # the parameters passed to the tool
}, ensure_ascii=False) + "\n"

# app.py — tool_end event
yield json.dumps({
    "type": "tool_end",
    "tool_name": tool_name,
    "tool_type": tool_type,
    "operation_index": operation_counter,
    "content": serializable_output,  # summarized output + source URLs
}, ensure_ascii=False) + "\n"

The Streamlit frontend renders these as collapsible cards showing:

Tool name and type (🔍 Search, 📄 Extract, 🕷️ Crawl)
Input parameters — the query string or URL list sent to the tool
Output source links — clickable URLs with favicons extracted from the tool response

Getting Started

Core Concepts

Configuration

Guides

Tavily Web Tools in ChatAgents: Search, Extract, and Crawl

TavilySearch

Parameters

ReAct Format

Fast vs Deep configuration

TavilyExtract

Parameters

Output Summarization

Batching URLs

TavilyCrawl

Parameters

When to Use Crawl vs Extract

Output Summarization

Tool Selection Logic

Tool Call Visualization

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Configuration

Guides

Documentation Index

​TavilySearch

​Parameters

​ReAct Format

​Fast vs Deep configuration

​TavilyExtract

​Parameters

​Output Summarization

​Batching URLs

​TavilyCrawl

​Parameters

​When to Use Crawl vs Extract

​Output Summarization

​Tool Selection Logic

​Tool Call Visualization

Build docs developers (and LLMs) love

TavilySearch

Parameters

ReAct Format

Fast vs Deep configuration

TavilyExtract

Parameters

Output Summarization

Batching URLs

TavilyCrawl

Parameters

When to Use Crawl vs Extract

Output Summarization

Tool Selection Logic

Tool Call Visualization