Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Trafilatura provides robust HTTP downloading functionality through the trafilatura.downloads module. Whether you need a single decoded HTML string or a full response object with headers, the library handles connection pooling, retries, SSL, and domain-aware throttling automatically.

​
Simple Downloads

The fetch_url() function is the easiest way to download a web page. It sends an HTTP GET request and returns the page content as a decoded Unicode string, or None if the download fails or the response doesn’t meet size requirements. 
from trafilatura.downloads import fetch_url

# single download – returns a decoded HTML string or None
downloaded = fetch_url("https://www.example.org")

# sequential downloads using a list
mylist = ["https://www.example.org", "https://httpbin.org"]
for url in mylist:
    downloaded = fetch_url(url)
    # process the result
    if downloaded:
        print(downloaded[:200])
fetch_url() uses a connection pool internally, keeping connections open across calls for efficiency. You may see warnings in logs about this — they can safely be ignored.

​
The Response Object

For richer access to the HTTP response — including status codes, headers, and raw binary data — use fetch_response(). It returns a Response object with the following attributes:
AttributeTypeDescription
databytesRaw response body as a bytestring
headersdict | NoneResponse headers (populated when with_headers=True)
htmlstr | NoneDecoded HTML string (populated when decode=True)
statusintHTTP status code
urlstrFinal URL after any redirects
from trafilatura.downloads import fetch_response

# basic response object
response = fetch_response("https://www.example.org")
print(response.status)   # e.g. 200
print(response.url)      # final URL after redirects
print(response.data)     # raw bytes

# decode HTML and capture response headers
response = fetch_response(
    "https://www.example.org",
    decode=True,
    with_headers=True,
)
print(response.html)     # decoded HTML string
print(response.headers)  # dict of response headers
fetch_response() was introduced in version 1.7.0. Use fetch_url() for a simpler interface when you only need the HTML string.

​
Parallel Downloads

For bulk collection, Trafilatura provides a domain-aware parallel download system backed by threads. This is the recommended approach when retrieving pages from multiple different websites at once, as it throttles requests per domain to avoid overloading any single server.
1

Convert your URL list to the internal format

add_to_compressed_dict() deduplicates URLs and organises them by domain into a UrlStore object.
from trafilatura.downloads import add_to_compressed_dict

mylist = ["https://www.example.org", "https://www.httpbin.org/html"]
url_store = add_to_compressed_dict(mylist)
2

Draw a download buffer respecting back-off rules

load_download_buffer() draws a batch of URLs that are safe to fetch now, waiting if all remaining URLs are in their back-off window.
from trafilatura.downloads import load_download_buffer

bufferlist, url_store = load_download_buffer(url_store, sleep_time=5)
3

Dispatch downloads with a thread pool

buffered_downloads() fans out the buffer list across the specified number of threads and yields (url, result) pairs as they complete.
from trafilatura.downloads import buffered_downloads

threads = 4
for url, result in buffered_downloads(bufferlist, threads):
    print(url, result[:100] if result else None)
4

Loop until all URLs are processed

Combine the steps above in a while loop that runs until the UrlStore reports it is done.
from trafilatura.downloads import (
    add_to_compressed_dict,
    buffered_downloads,
    load_download_buffer,
)

mylist = ["https://www.example.org", "https://www.httpbin.org/html"]
threads = 4

url_store = add_to_compressed_dict(mylist)

while url_store.done is False:
    bufferlist, url_store = load_download_buffer(url_store, sleep_time=5)
    for url, result in buffered_downloads(bufferlist, threads):
        # process each downloaded page here
        print(url)
        print(result)
This approach is highly recommended for large-scale collection. The domain-aware throttling means you can queue thousands of URLs from different sites without manually managing delays per host.

​
CLI Parallel Downloads

On the command line, Trafilatura automatically uses threads and domain-aware throttling. Pass a file of URLs with -i/--input-file and specify an output directory with -o: 
# download pages from a URL list, save extracted text and back up raw HTML
trafilatura -i list.txt -o txtfiles/ --backup-dir htmlbackup/
You can check the exit code to detect download errors: 0 means all pages downloaded successfully, 1 means at least one failed. Detailed errors appear in the logs.

​
Internet Archive Fallback

When a page cannot be fetched directly, Trafilatura can fall back to a cached version in the Internet Archive (Wayback Machine). Enable this with the --archived flag on the CLI: 
# retry failed downloads from the Internet Archive
trafilatura --archived -i list.txt -o output/
Internally, failed URLs are prefixed with https://web.archive.org/web/20/ and re-queued for download.

​
Politeness Rules

Politely behaved crawlers respect the servers they visit. Trafilatura implements several mechanisms to keep resource usage acceptable.

​
Robots Exclusion Standard

The robots.txt file at the root of a website describes which parts may be crawled and at what rate. Use Python’s built-in urllib.robotparser to read and enforce these rules: 
import urllib.robotparser

base_url = "https://www.example.org"

rules = urllib.robotparser.RobotFileParser()
rules.set_url(base_url + "/robots.txt")
rules.read()

# check if a specific page may be fetched
allowed = rules.can_fetch("*", "https://www.example.org/page1234.html")
print(allowed)  # True or False

# retrieve the crawl delay specified in robots.txt (None if not set)
delay = rules.crawl_delay("*")
# provide a safe fallback if no rule is set
if delay is None:
    delay = 30

​
Spacing Requests

Pass sleep_time (in seconds) to load_download_buffer() to enforce a minimum interval between successive requests to the same domain: 
from trafilatura.downloads import load_download_buffer

# 30 seconds is a safe choice for considerate crawling
bufferlist, url_store = load_download_buffer(url_store, sleep_time=30)

​
SOCKS Proxy

Set the http_proxy environment variable before importing Trafilatura to route all downloads through a SOCKS proxy: 
# plain proxy
export http_proxy=socks5://PROXYHOST:PROXYPORT

# with authentication
export http_proxy=socks5://USER:PASSWORD@PROXYHOST:PROXYPORT
Alternatively, set trafilatura.downloads.PROXY_URL at runtime (affects the current Python session only).
SOCKS proxy support was introduced in version 1.12.2.

​
Configuration

Download behaviour is controlled via a settings.cfg file. The relevant keys live in the [DEFAULT] section: 
[DEFAULT]

# seconds to wait for a connection/response
DOWNLOAD_TIMEOUT = 30

# maximum accepted response size in bytes (default ~20 MB)
MAX_FILE_SIZE = 20000000

# minimum accepted response size in bytes
MIN_FILE_SIZE = 10

# seconds to sleep between requests to the same domain
SLEEP_TIME = 5.0

# maximum number of redirects to follow
MAX_REDIRECTS = 2

# custom user-agents (one per line, chosen at random)
USER_AGENTS =
#     "MyBot/1.0"
#     "AnotherAgent/2.0"

# cookie to include in HTTP requests
COOKIE =
See the Settings guide for how to create and load a custom configuration file.

​
Custom User-Agents and Cookies

When a settings.cfg file with non-default values is loaded, fetch_url() and fetch_response() automatically pick up the configured user-agent and cookie: 
from configparser import ConfigParser
from trafilatura.downloads import fetch_url

config = ConfigParser()
config.read("my_settings.cfg")

# the custom user-agent and cookie are applied automatically
result = fetch_url("https://www.example.org", config=config)
If multiple user-agent strings are listed under USER_AGENTS, one is selected at random for each request, which can reduce the likelihood of rate-limiting.

​
Connection Pooling

Trafilatura reuses a single urllib3.PoolManager (up to 50 pools) for the lifetime of the Python process. This means TCP connections to the same host are kept alive across multiple fetch_url() calls, reducing latency and overhead for sequential downloads to the same domain. When pycurl is installed, a shared CurlShare object provides additional DNS and SSL session caching.

Build docs developers (and LLMs) love

Get started for freeTalk to us