Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

This page covers the most common problems encountered when using Trafilatura for content extraction and web downloads, along with practical fixes for each. Trafilatura is thoroughly tested, but some issues are environment-specific. If your problem is not listed here, check the open issues on GitHub.
Make sure you are on the latest version before debugging: pip install -U trafilatura

​
Empty or incomplete output

​
Not enough content extracted

Content extraction is a trade-off between precision (avoiding boilerplate) and recall (capturing all relevant text). Trafilatura is optimized for article and blog-post pages — results can be sparse on link lists, galleries, or catalog pages. If you are getting too little content, try the following in order: 1. Switch to recall mode 
from trafilatura import extract

result = extract(downloaded, favor_recall=True)
2. Lower the minimum extraction size The MIN_EXTRACTED_SIZE setting (default: 250 characters) determines when fallback extraction is triggered. Lowering it makes Trafilatura more willing to return short content. Edit settings.cfg: 
[DEFAULT]
MIN_EXTRACTED_SIZE = 100
Or in Python: 
from copy import deepcopy
from trafilatura import extract
from trafilatura.settings import DEFAULT_CONFIG

my_config = deepcopy(DEFAULT_CONFIG)
my_config['DEFAULT']['MIN_EXTRACTED_SIZE'] = '100'

result = extract(my_html, config=my_config)
3. Use the baseline or html2txt functions These are simpler, faster functions that do not attempt to filter main content — they return all extractable text: 
from trafilatura import baseline, html2txt

# baseline: extracts text from paragraph, code, and quote elements
result = baseline(downloaded)

# html2txt: converts all HTML to plain text (fastest, least filtered)
result = html2txt(downloaded)
Trafilatura is primarily designed for article pages, blog posts, and main text sections. Output quality varies on link-heavy pages, image galleries, product catalogs, and similar non-article content types.

​
Too much noise or irrelevant content

​
Output contains too much boilerplate

If the extracted text includes navigation menus, footers, sidebars, or other unwanted content, try the following: 1. Switch to precision mode 
result = extract(downloaded, favor_precision=True)
2. Prune specific HTML elements with XPath The prune_xpath parameter lets you remove elements by XPath expression before extraction runs: 
result = extract(
    downloaded,
    prune_xpath=["//div[@class='sidebar']", "//footer", "//nav"]
)

​
Download failures

​
Network timeout

The default DOWNLOAD_TIMEOUT is 30 seconds. Increase it in settings.cfg for slow servers: 
[DEFAULT]
DOWNLOAD_TIMEOUT = 60

​
User-agent blocked

Some websites block Trafilatura’s default user-agent string. You can supply a custom one in settings.cfg: 
[DEFAULT]
USER_AGENTS = Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36
For pages that require authentication cookies, add them to settings.cfg: 
[DEFAULT]
COOKIE = yummy_cookie=choco; tasty_cookie=strawberry
Alternatively, use the standard library cookiejar with urllib3.

​
Using pycurl for better download reliability

The default download library (urllib3) may fail on some servers. Installing pycurl gives Trafilatura an alternative: 
pip install pycurl
# or install all optional dependencies at once:
pip install trafilatura[all]
You can also use wget or curl on the command-line as a workaround: 
wget -O - "https://example.org/" | trafilatura

​
Falling back to the Internet Archive

Use --archived on the CLI to automatically retrieve a cached version from the Internet Archive when a direct download fails: 
trafilatura --archived -u "https://example.org/some-page"
In Python, construct the archive URL manually: 
from trafilatura import fetch_url

url = "https://example.org/some-page"
downloaded = fetch_url(url)

if downloaded is None:
    archive_url = "https://web.archive.org/web/20/" + url
    downloaded = fetch_url(archive_url)
Downloads may fail because your IP address or user-agent is blocked. Trafilatura’s built-in download capabilities do not bypass such restrictions. For pages behind heavy anti-bot measures, consider browser automation (e.g. Playwright) or the nodriver / browserforge packages.

​
JavaScript-rendered and paywall-protected pages

​
Content injected by JavaScript

Trafilatura processes raw HTML and does not execute JavaScript. If a page relies on JavaScript to render its main content, the raw HTML will not contain the text you expect. The solution is to use a headless browser to render the page first, then pass the rendered HTML to Trafilatura. Browser automation libraries like Playwright can handle this. See the list of headless browsers for available alternatives. For more refined masking and automation methods, see the nodriver and browserforge packages.

​
Bypassing paywalls

Browser automation can also help with cookie-based paywalls by combining a headless browser with a paywall-bypass browser extension.

​
Encoding issues

​
Garbled or incorrectly decoded text

Trafilatura uses several heuristics to detect character encoding. Install faust-cchardet for faster and more accurate detection — particularly important for non-Latin scripts (Asian languages, Arabic, etc.): 
pip install faust-cchardet
# or:
pip install trafilatura[all]
Trafilatura automatically detects and uses faust-cchardet if it is installed.

​
Manually decoding a file

The decode_file() utility handles byte-to-string conversion with encoding detection: 
from trafilatura.utils import decode_file

with open("page.html", "rb") as f:
    raw_bytes = f.read()

html_string = decode_file(raw_bytes)

​
Date extraction issues

​
Date not found

If date extraction is failing, pass the URL as context — Trafilatura’s htmldate component can often extract a date directly from the URL path: 
from trafilatura import bare_extraction

result = bare_extraction(downloaded, url=url)
For higher recall at the cost of some precision, enable extensive_search in date_extraction_params: 
result = extract(
    downloaded,
    output_format="xml",
    date_extraction_params={"extensive_search": True}
)
This activates additional heuristics in htmldate, the underlying date extraction library.

​
CLI not found after pip install

If you installed Trafilatura with pip but the trafilatura command is not found in your terminal, the user-level bin directory is likely not on your PATH. On Linux / macOS: 
export PATH="$HOME/.local/bin:$PATH"
Add this line to your ~/.bashrc, ~/.zshrc, or equivalent shell configuration file to make it permanent. Further reference:

​
Signal / timeout errors

If you see errors related to the signal module when running Trafilatura (particularly in multi-threaded environments or on Windows), the extraction timeout is the cause. EXTRACTION_TIMEOUT (default: 30 seconds) is active in CLI mode to prevent runaway CPU usage from malformed or adversarial input files. To disable it, set the value to 0 in settings.cfg: 
[DEFAULT]
EXTRACTION_TIMEOUT = 0
Alternatively, install defusedxml to harden XML parsing against malicious input: 
pip install defusedxml

​
Memory issues in large-scale processing

Trafilatura uses internal caches (in the extraction engine, date extractor, and URL normalizer) to speed up repeated operations. In long-running processes or large batch jobs, these caches can grow and cause memory pressure. Reset all caches periodically to release RAM: 
from trafilatura.meta import reset_caches

# Call this periodically in your processing loop
reset_caches()
Example pattern for large batches: 
from trafilatura import fetch_url, extract
from trafilatura.meta import reset_caches

urls = [...]  # your URL list

for i, url in enumerate(urls):
    downloaded = fetch_url(url)
    if downloaded:
        result = extract(downloaded)
        # ... process result ...

    # Reset caches every 1000 documents
    if i % 1000 == 0:
        reset_caches()

Build docs developers (and LLMs) love

Get started for freeTalk to us