Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Trafilatura’s discovery module provides three complementary strategies for finding content URLs on a website: RSS/Atom feed extraction, XML sitemap traversal, and an autonomous focused web crawler. Together they let you build comprehensive link lists before downloading and extracting page content.

​
find_feed_urls()

Attempts to discover and parse RSS, Atom, or JSON feeds for a given website, then extracts all article URLs from those feeds. If the input URL is already a feed, it is parsed directly. Otherwise, Trafilatura probes the page for <link rel="alternate"> elements and known feed URL patterns, fetches any discovered feeds, and returns the collected links. 
from trafilatura.feeds import find_feed_urls

urls = find_feed_urls("https://www.example.com")
for url in urls:
    print(url)
Signature 
find_feed_urls(
    url,
    target_lang=None,
    external=False,
    sleep_time=2.0,
) -> list[str]
Parameters
​
url
str
required
A homepage URL or a direct feed URL (RSS, Atom, JSON Feed). When the URL points to a subpage rather than the homepage (i.e., it contains more path segments), Trafilatura applies a URL prefix filter so that only links matching that path are returned. On failure, it automatically retries against the site homepage.
​
target_lang
str
default:"None"
An ISO 639-1 two-letter language code (e.g., "en", "fr"). When set, URL heuristics are used to filter out links that appear to belong to a different language version of the site.
​
external
bool
default:"False"
When False (default), only URLs belonging to the same domain or closely related subdomains are returned. Set to True to also include links pointing to external domains (e.g., Feedburner, Google News).
​
sleep_time
float
default:"2.0"
Number of seconds to wait between HTTP requests to the same website, respecting politeness conventions.
Returns: list[str] — a sorted, deduplicated list of article URLs found in the feeds. Returns an empty list if no feeds are discovered or no valid links are found. Examples 
from trafilatura.feeds import find_feed_urls
from trafilatura import fetch_url, extract

# Discover all article URLs from a site's feeds
urls = find_feed_urls("https://www.theguardian.com")
print(f"Found {len(urls)} URLs")

# Filter by language (e.g., only English articles)
en_urls = find_feed_urls("https://www.spiegel.de", target_lang="de")

# Point directly at a known feed URL
urls = find_feed_urls("https://feeds.feedburner.com/TechCrunch")

# Download and extract all discovered articles
for url in urls[:10]:  # process first 10
    html = fetch_url(url)
    if html:
        text = extract(html, url=url)
        if text:
            print(f"Extracted: {url}")
Discovers and traverses XML sitemaps for a website to collect page URLs. Trafilatura first checks robots.txt for declared Sitemap: entries, then tries a list of common sitemap paths (sitemap.xml, sitemap_index.xml, etc.). Nested sitemap index files are followed recursively up to the max_sitemaps limit. 
from trafilatura.sitemaps import sitemap_search

urls = sitemap_search("https://www.example.com")
print(f"Found {len(urls)} URLs in sitemaps")
Signature 
sitemap_search(
    url,
    target_lang=None,
    external=False,
    sleep_time=2.0,
    max_sitemaps=10000,
) -> list[str]
Parameters
​
url
str
required
A homepage URL, a direct sitemap URL (ending in .xml, .xml.gz, or sitemap), or any page URL. When a subpage URL is given, Trafilatura applies a URL prefix filter to the results. The function verifies that the base URL is reachable before starting.
​
target_lang
str
default:"None"
An ISO 639-1 two-letter language code. When set, hreflang attributes in the sitemap are used to select only URLs for the target language. Falls back to URL-pattern heuristics when hreflang is absent.
​
external
bool
default:"False"
Allow links pointing to external domains. By default, only URLs on the same domain (or closely related subdomains) are kept.
​
sleep_time
float
default:"2.0"
Seconds to pause between sitemap fetch requests to the same host.
​
max_sitemaps
int
default:"10000"
Maximum number of individual sitemap files to fetch and process. Acts as a safeguard against very large or circular sitemap trees.
Returns: list[str] — a list of page URLs collected from all discovered sitemaps. Returns an empty list if no sitemaps are reachable or the site is unreachable. Examples 
from trafilatura.sitemaps import sitemap_search
from trafilatura import fetch_url, extract

# Collect all URLs from a site's sitemaps
urls = sitemap_search("https://www.example.com")
print(f"Found {len(urls)} pages")

# Target German-language pages only
de_urls = sitemap_search("https://www.bbc.com", target_lang="de")

# Point directly at a sitemap file
urls = sitemap_search("https://www.example.com/sitemap_index.xml")

# Limit sitemap traversal depth
urls = sitemap_search("https://www.largsite.com", max_sitemaps=50)

# Extract content from all discovered URLs
for url in urls[:20]:
    html = fetch_url(url)
    if html:
        text = extract(html, url=url, favor_precision=True)
        if text:
            print(f"OK: {url} ({len(text)} chars)")

​
focused_crawler()

A lightweight, focused web crawler that starts from a homepage, follows internal links, and builds up a list of discovered URLs. It respects robots.txt, prioritizes navigation pages (category listings, tag pages, etc.) to maximize URL discovery, and supports optional language filtering. The crawler operates statelessly via two lists: a todo list (URLs to visit next) and a known links list (all URLs seen so far). Both are returned after each call so you can resume crawling incrementally. 
from trafilatura.spider import focused_crawler

todo, known_links = focused_crawler("https://www.example.com", max_seen_urls=10)
print(f"Next to visit: {len(todo)}")
print(f"Known URLs:    {len(known_links)}")
Signature 
focused_crawler(
    homepage,
    max_seen_urls=10,
    max_known_urls=100000,
    todo=None,
    known_links=None,
    lang=None,
    config=DEFAULT_CONFIG,
    rules=None,
    prune_xpath=None,
) -> tuple[list[str], list[str]]
Parameters
​
homepage
str
required
The starting URL for the crawl, preferably the homepage of a website (e.g., "https://www.example.com"). The crawler automatically fetches and processes this page to seed the link queue.
​
max_seen_urls
int
default:"10"
Maximum number of pages to actually visit (fetch and process for links) during this call. The crawl stops when this limit is reached or the site is exhausted, whichever comes first.
​
max_known_urls
int
default:"100000"
Stop the crawl if the total number of discovered (known) URLs exceeds this value. Useful as a memory safeguard on very large sites.
​
todo
list[str]
default:"None"
A previously generated crawl frontier (list of URLs to visit). Pass the todo return value from a prior call to resume an interrupted crawl.
​
known_links
list[str]
default:"None"
A list of previously discovered URLs (visited or not). Pass the known_links return value from a prior call to avoid revisiting the same pages.
​
lang
str
default:"None"
An ISO 639-1 two-letter language code. When set, pages whose detected language does not match are skipped and their links are not followed. Requires py3langid to be installed.
​
config
ConfigParser
default:"DEFAULT_CONFIG"
A configparser.ConfigParser instance with Trafilatura settings. Used to read the default sleep time between requests.
​
rules
RobotFileParser
default:"None"
A pre-parsed urllib.robotparser.RobotFileParser object. When None, the crawler automatically fetches and parses robots.txt from the homepage domain.
​
prune_xpath
str
default:"None"
An XPath expression. Matching elements are removed from each fetched page’s HTML before link extraction. Useful for suppressing navigation menus or sidebars that would pollute the link queue.
Returns: A 2-tuple:
  • list[str] — the remaining crawl frontier (URLs yet to be visited)
  • list[str] — all known URLs discovered during the crawl (visited and unvisited)
Step-by-step example 
from trafilatura.spider import focused_crawler, is_still_navigation
from trafilatura import fetch_url, extract

homepage = "https://www.example.com"

# --- Step 1: Initial crawl to discover navigation and article URLs ---
todo, known_links = focused_crawler(homepage, max_seen_urls=10)
print(f"After initial crawl: {len(todo)} to visit, {len(known_links)} known")

# --- Step 2: Continue crawling in batches until only article links remain ---
while is_still_navigation(todo):
    todo, known_links = focused_crawler(
        homepage,
        max_seen_urls=10,
        todo=todo,
        known_links=known_links,
    )
    print(f"Still navigating: {len(todo)} to visit, {len(known_links)} known")

print(f"Navigation exhausted. {len(todo)} article URLs ready.")

# --- Step 3: Extract content from discovered article URLs ---
extracted = []
for url in todo:
    html = fetch_url(url)
    if html:
        text = extract(html, url=url, with_metadata=True, output_format="json")
        if text:
            extracted.append(text)

print(f"Extracted content from {len(extracted)} articles")

​
is_still_navigation()

A simple helper that inspects the current crawl frontier and returns True if it still contains navigation or listing pages (e.g., category archives, tag pages, paginated indexes). Used to decide whether to keep crawling for more link discovery or switch to downloading article content. 
from trafilatura.spider import is_still_navigation

todo = [
    "https://www.example.com/category/tech",
    "https://www.example.com/article-about-python",
]

if is_still_navigation(todo):
    print("Still finding new pages — keep crawling.")
else:
    print("Only article URLs remain — start extracting.")
Signature 
is_still_navigation(todo) -> bool
Parameters
​
todo
list[str]
required
The current crawl frontier — a list of URLs that have been discovered but not yet visited. Typically the first element of the tuple returned by focused_crawler().
Returns: bool — True if at least one URL in todo is classified as a navigation page (using courlan.is_navigation_page()); False if all remaining URLs appear to be leaf/article pages. Example 
from trafilatura.spider import focused_crawler, is_still_navigation
from trafilatura import fetch_url, extract

homepage = "https://www.example.com"
todo, known_links = focused_crawler(homepage, max_seen_urls=5)

# Keep discovering links while navigation pages are present
max_iterations = 20
iteration = 0

while is_still_navigation(todo) and iteration < max_iterations:
    todo, known_links = focused_crawler(
        homepage,
        max_seen_urls=5,
        todo=todo,
        known_links=known_links,
    )
    iteration += 1
    print(f"Iteration {iteration}: {len(todo)} URLs in queue")

print(f"Ready to extract from {len(todo)} article URLs")

# Collect extracted text from all discovered URLs
texts = {}
for url in todo:
    html = fetch_url(url)
    if html:
        result = extract(html, url=url)
        if result:
            texts[url] = result

print(f"Extracted content from {len(texts)} pages")

Build docs developers (and LLMs) love

Get started for freeTalk to us