Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Before downloading and extracting text at scale, you need a list of URLs to process. Trafilatura can automatically discover those URLs by parsing feeds (Atom, RSS, JSON Feed) and sitemaps (XML, sitemap index, gzip-compressed). Both mechanisms let you filter results by language and restrict discovery to specific sections of a site.

Feeds

Parse Atom, RSS, and JSON Feed documents to extract article URLs, or auto-discover feed links from any homepage.

Sitemaps

Parse XML sitemaps and sitemap indexes, or auto-discover them via robots.txt. Supports nested sitemaps and hreflang filtering.

​
Feeds

​
Finding Feed URLs with Python

The find_feed_urls() function in trafilatura.feeds accepts either a homepage URL or a direct feed URL. When given a homepage it tries to locate feed links in the page’s <link rel="alternate"> elements and linked <a href> anchors; when given a feed URL it parses the feed directly. 
from trafilatura.feeds import find_feed_urls

# auto-discover feeds from a homepage and return article URLs
urls = find_feed_urls("https://www.example.org")

# parse a known Atom or RSS feed URL directly
urls = find_feed_urls("https://www.example.org/feed.xml")

print(urls)  # sorted list of unique article URLs
The function returns a sorted list of unique URLs extracted from the feed entries.

​
Language Filtering

Pass target_lang (a two-letter ISO 639-1 code) to filter URLs using language heuristics applied to the URL string itself: 
# only return URLs that appear to be in English
urls = find_feed_urls("https://www.example.org", target_lang="en")

# German content only
urls = find_feed_urls("https://www.example.org", target_lang="de")

​
Cross-Domain URLs

By default, find_feed_urls() discards links that point to a different domain. Set external=True to include them: 
# allow URLs from external or partner domains
urls = find_feed_urls("https://www.example.org", external=True)

​
Full Parameter Reference

from trafilatura.feeds import find_feed_urls

urls = find_feed_urls(
    url="https://www.example.org",   # homepage or feed URL
    target_lang="en",                # ISO 639-1 language code, or None
    external=False,                  # allow cross-domain URLs
    sleep_time=2.0,                  # seconds between same-domain requests
)

​
Feeds on the CLI

Use --feed together with --list to print discovered URLs without downloading them: 
# discover feed URLs and print the list
trafilatura --feed "https://www.example.org" --list

# pass a direct feed URL
trafilatura --feed "https://www.example.org/atom.xml" --list

# download and extract all discovered articles
trafilatura --feed "https://www.example.org" -o output/
Combine --feed with --target-language to pre-filter URLs by language before bulk downloading.

​
Sitemaps

​
Searching Sitemaps with Python

The sitemap_search() function in trafilatura.sitemaps works similarly to find_feed_urls(). It accepts a homepage URL or a direct sitemap URL. For a homepage it first reads robots.txt for Sitemap: directives and then tries a set of standard paths (sitemap.xml, sitemap_index.xml, etc.). Nested sitemap indexes are followed recursively. 
from trafilatura.sitemaps import sitemap_search

# auto-discover and parse sitemaps for a website
urls = sitemap_search("https://www.example.org")

# parse a known sitemap URL directly
urls = sitemap_search("https://www.example.org/sitemap.xml")

# gzip-compressed sitemaps are supported
urls = sitemap_search("https://www.example.org/sitemap.xml.gz")

print(urls)  # sorted list of unique page URLs

​
Language Filtering

Pass target_lang to filter URLs using language heuristics and hreflang attributes found in the sitemap: 
# English pages only
urls = sitemap_search("https://www.example.org", target_lang="en")

# French pages only
urls = sitemap_search("https://www.example.org", target_lang="fr")
When hreflang attributes are present in the sitemap, Trafilatura uses them for precise language matching. Otherwise it falls back to URL-based heuristics.

​
Cross-Domain URLs

Set external=True to include URLs from different domains or subdomains that appear in the sitemap: 
urls = sitemap_search("https://www.example.org", external=True)

​
Filtering by URL Pattern (Subdirectory)

Pass a subdirectory URL instead of the homepage to limit results to pages under that path. This works because sitemap_search() applies the subdirectory URL as a filter on the collected links: 
# only return URLs under /blog/
urls = sitemap_search("https://www.example.org/blog/")

# only return URLs under /en/news/
urls = sitemap_search("https://www.example.org/en/news/")

​
Full Parameter Reference

from trafilatura.sitemaps import sitemap_search

urls = sitemap_search(
    url="https://www.example.org",  # homepage, sitemap, or subdirectory URL
    target_lang="en",               # ISO 639-1 language code, or None
    external=False,                 # allow cross-domain URLs
    sleep_time=2.0,                 # seconds between requests to the same domain
    max_sitemaps=10000,             # maximum number of sitemap files to process
)

​
Sitemaps on the CLI

Use --sitemap together with --list to discover and print URLs without downloading them: 
# discover sitemap URLs and print the list
trafilatura --sitemap "https://www.example.org" --list

# parse a known sitemap URL
trafilatura --sitemap "https://www.example.org/sitemap.xml" --list

# download and extract all discovered pages
trafilatura --sitemap "https://www.example.org" -o output/

​
URL Filtering

Both the feed and sitemap discovery tools can be narrowed down further using the --url-filter option on the CLI. Only URLs that contain the given string (or any of a comma-separated list of strings) are kept: 
# only keep URLs containing "technology"
trafilatura --sitemap "https://www.example.org" --list --url-filter "technology"

# multiple patterns
trafilatura --feed "https://www.example.org" --list --url-filter "sports,culture"

​
Allowing External URLs via Settings

By default, both find_feed_urls() and sitemap_search() discard links that point outside the source domain. You can change this default globally by setting EXTERNAL_URLS = on in your settings.cfg file: 
[DEFAULT]

# allow URLs from external or different domains in feeds and sitemaps
EXTERNAL_URLS = on
See the Settings guide for how to load a custom configuration file.

​
Supported Feed Formats

Trafilatura handles the most common syndication formats automatically:

Atom

Parses <link href="..."> elements for article URLs.

RSS

Parses <link> text content for article URLs.

JSON Feed

Parses the items[].url and items[].id fields per the JSON Feed 1.1 spec.
Feedburner and Google Feeds proxy URLs are preserved as-is when standard URL validation would otherwise discard them.

Build docs developers (and LLMs) love

Get started for freeTalk to us