Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Trafilatura’s download module handles all HTTP communication, including SSL negotiation, retries, proxy support, and decoding. It automatically selects between urllib3 (default) and pycurl (when installed) for optimal performance. The three public functions — fetch_url(), fetch_response(), and load_html() — cover progressively more detailed levels of access, from a plain decoded string all the way to a parsed lxml tree.

​
fetch_url()

Downloads a web page and returns the decoded HTML content as a Unicode string. This is the simplest and most commonly used download function, suitable for use with extract() and most other Trafilatura functions. 
from trafilatura import fetch_url, extract

html = fetch_url("https://www.example.com/article")
if html:
    text = extract(html)
    print(text)
Signature 
fetch_url(
    url,
    no_ssl=False,
    config=DEFAULT_CONFIG,
    options=None,
) -> str | None
Parameters
​
url
str
required
The fully-qualified URL of the page to download (must begin with http:// or https://).
​
no_ssl
bool
default:"False"
Disable SSL certificate verification. Useful when a host has a misconfigured or self-signed certificate. Trafilatura will automatically retry with no_ssl=True after an SSLError, so manually setting this is rarely necessary.
​
config
ConfigParser
default:"DEFAULT_CONFIG"
A configparser.ConfigParser instance controlling download behaviour (timeouts, max file size, max redirects, user agents, cookies). Superseded by options when both are provided.
​
options
Extractor
default:"None"
A pre-built trafilatura.settings.Extractor object. When provided, options.config is used as the configuration and overrides the config parameter.
Returns: str | None — the decoded HTML content as a Unicode string, or None if the download fails, returns a non-200 status, or the response is outside the acceptable size limits. Examples 
from trafilatura import fetch_url, extract

# Single page download
html = fetch_url("https://www.example.com/article")
if html:
    text = extract(html)
    print(text)

# Sequential download of a URL list
urls = [
    "https://www.example.com/post-1",
    "https://www.example.com/post-2",
    "https://www.example.com/post-3",
]

results = []
for url in urls:
    html = fetch_url(url)
    if html:
        text = extract(html, url=url)
        if text:
            results.append({"url": url, "text": text})

print(f"Successfully extracted {len(results)} of {len(urls)} pages")

# Download with custom settings (e.g., longer timeout)
from configparser import ConfigParser
config = ConfigParser()
config.read_dict({"DEFAULT": {"DOWNLOAD_TIMEOUT": "60", "MAX_FILE_SIZE": "20000000"}})
html = fetch_url("https://www.example.com/large-page", config=config)

​
fetch_response()

Downloads a web page and returns a full Response object containing the raw bytes, HTTP status, final URL (after any redirects), and optionally the decoded HTML and response headers. Use this when you need lower-level control — for example, to inspect redirect chains, access response headers, or handle the binary data yourself. 
from trafilatura import fetch_response

response = fetch_response("https://www.example.com/article", decode=True, with_headers=True)
if response:
    print(f"Final URL:    {response.url}")
    print(f"Status:       {response.status}")
    print(f"Content-Type: {response.headers.get('content-type')}")
    print(f"HTML preview: {response.html[:200]}")
Signature 
fetch_response(
    url,
    *,
    decode=False,
    no_ssl=False,
    with_headers=False,
    config=DEFAULT_CONFIG,
) -> Response | None
Parameters
​
url
str
required
The URL to fetch.
​
decode
bool
default:"False"
Decode the raw bytes and populate response.html with a Unicode string. When False, response.html is None and only response.data (bytes) is populated.
​
no_ssl
bool
default:"False"
Disable SSL certificate verification. Trafilatura retries automatically on SSLError, so this is typically not needed.
​
with_headers
bool
default:"False"
Store the HTTP response headers in response.headers as a lowercase-keyed dictionary.
​
config
ConfigParser
default:"DEFAULT_CONFIG"
Configuration object controlling download parameters.
Returns: Response | None — a Response object, or None if the download fails. The Response object The Response class stores all information gathered during a single HTTP request:
AttributeTypeDescription
databytesRaw response body bytes
htmlstr | NoneDecoded HTML string (populated when decode=True)
statusintHTTP status code (e.g. 200, 404)
urlstrFinal URL after following all redirects
headersdict[str, str] | NoneLowercase-keyed response headers (populated when with_headers=True)
Response also supports bool(response) (truthy when data is not None), str(response) (renders as HTML if decoded, otherwise decodes bytes), and response.as_dict() which returns all attributes as a plain dictionary. Examples 
from trafilatura import fetch_response, extract
from trafilatura.utils import load_html

# Inspect redirect chain and headers
response = fetch_response(
    "https://www.example.com/redirect-me",
    decode=True,
    with_headers=True,
)

if response:
    print(f"Landed at: {response.url}")       # final URL after redirects
    print(f"Status:    {response.status}")
    print(f"Server:    {response.headers.get('server')}")

    # Pass decoded HTML to extract()
    text = extract(response.html, url=response.url)
    print(text)

# Work with raw bytes (e.g., for binary content or custom decoding)
response = fetch_response("https://www.example.com/page")
if response and response.data:
    print(f"Downloaded {len(response.data)} bytes from {response.url}")

    # Manual decoding and parsing
    tree = load_html(response.data)

# Convert response to a dict (e.g., for logging or caching)
if response:
    payload = response.as_dict()
    # payload keys: "data", "headers", "html", "status", "url"

​
load_html()

Parses an HTML document into an lxml element tree. Accepts a Unicode string, raw bytes, a pre-parsed HtmlElement (returned as-is), or a Response-like object with a .data attribute. Handles encoding detection, faulty HTML repair, and falling back to byte-level parsing when needed. 
from trafilatura.utils import load_html

html = "<html><body><p>Hello world</p></body></html>"
tree = load_html(html)
if tree is not None:
    print(tree.find(".//p").text)  # → "Hello world"
Signature 
load_html(htmlobject) -> HtmlElement | None
Parameters
​
htmlobject
str | bytes | HtmlElement
required
The HTML input to parse. Accepted types:
  • str — a Unicode HTML string
  • bytes — raw byte content; encoding is auto-detected
  • lxml.html.HtmlElement — returned unchanged without re-parsing
  • Any object with a .data attribute (e.g., a Response or urllib3 response) — .data is extracted and parsed as bytes
Returns: lxml.html.HtmlElement | None — the root element of the parsed HTML tree, or None if parsing fails or the input is not valid HTML. Examples 
from trafilatura import fetch_url
from trafilatura.utils import load_html

# Parse from a downloaded string
html_string = fetch_url("https://www.example.com")
tree = load_html(html_string)
if tree is not None:
    # Use XPath directly on the parsed tree
    titles = tree.xpath("//h1/text()")
    print(titles)

# Parse raw bytes (e.g., from a file)
with open("page.html", "rb") as f:
    content = f.read()
tree = load_html(content)

# Re-use an already-parsed tree — load_html returns it unchanged
from lxml.html import HtmlElement
if isinstance(tree, HtmlElement):
    same_tree = load_html(tree)  # no-op, returns tree directly
    assert same_tree is tree

# Feed the tree directly to extract()
from trafilatura import extract
if tree is not None:
    text = extract(tree, url="https://www.example.com")
    print(text)

Build docs developers (and LLMs) love

Get started for freeTalk to us