Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Trafilatura exposes two configuration layers. The first is settings.cfg, a standard INI-style file that controls runtime tunables such as download timeouts, size thresholds, and deduplication limits — these are the values most users will want to adjust. The second is settings.py, a Python module that governs package-wide constants (HTML element lists, parallel-processing limits, LRU cache sizes) and requires a local reinstall when changed.

​
settings.cfg

The default configuration file ships inside the package at trafilatura/settings.cfg. You can inspect it directly on GitHub. Providing a custom file — either on the command line or in Python — overrides only the keys you specify; all other values fall back to the defaults.

​
Download

​
DOWNLOAD_TIMEOUT
int
default:"30"
Time in seconds before an HTTP request is dropped. Increase this on slow networks or when scraping heavy pages.
​
MAX_FILE_SIZE
int
default:"20000000"
Maximum acceptable input size in bytes (≈ 20 MB). Documents larger than this value are skipped.
​
MIN_FILE_SIZE
int
default:"10"
Minimum acceptable input size in bytes. Responses smaller than this value are treated as empty.
​
SLEEP_TIME
float
default:"5.0"
Pause in seconds between consecutive requests when crawling. Higher values reduce the risk of being rate-limited or flagged as a bot.
​
USER_AGENTS
string
default:""
One user-agent string per line. Leave empty to use the library default. Example:
USER_AGENTS =
    "Mozilla/5.0 (compatible; MyBot/1.0)"
​
COOKIE
string
default:""
Cookie string to send with every HTTP request. Useful for sites that require session authentication.
​
MAX_REDIRECTS
int
default:"2"
Maximum number of URL redirections to follow. Set to 0 to disable redirect following entirely.

​
Extraction

​
MIN_EXTRACTED_SIZE
int
default:"250"
Minimum character count for the main-text extraction to be considered successful before falling back to alternative extractors. Lowering this value enables more opportunistic extraction on short pages.
​
MIN_EXTRACTED_COMM_SIZE
int
default:"1"
Same threshold applied to comment extraction.
​
MIN_OUTPUT_SIZE
int
default:"1"
Absolute minimum character count for the final main-text output. Documents below this threshold are discarded entirely.
​
MIN_OUTPUT_COMM_SIZE
int
default:"1"
Same threshold applied to the final comment output.
​
MAX_TREE_SIZE
int
default:""
Discard documents whose parsed HTML tree contains more elements than this number. Leave empty (the default) to impose no limit. Useful for skipping abnormally large or malformed pages.
​
EXTRACTION_TIMEOUT
int
default:"30"
Seconds before a single extraction is aborted. Only active in CLI/file-processing mode. Set to 0 to disable. If you see errors related to the signal module, set this to 0 or consider using defusedxml.

​
Deduplication

Deduplication is not active by default. Enable it per-call with deduplicate=True or options.dedup = True. See the Deduplication page for full details.
​
MIN_DUPLCHECK_SIZE
int
default:"100"
Minimum character length of a text segment before it is evaluated for duplicates. Shorter segments are always kept.
​
MAX_REPETITIONS
int
default:"2"
Maximum number of times a segment may appear before it is flagged as a duplicate and removed.

​
Metadata

​
EXTENSIVE_DATE_SEARCH
bool
default:"on"
Controls htmldate’s opportunistic date search across the full page content. Set to off for higher precision at the cost of lower recall. See Metadata for full details.
​
EXTERNAL_URLS
bool
default:"off"
Whether to follow URLs pointing to other domains when processing feeds and sitemaps in CLI mode. Keeping this off limits crawling to the target domain.

​
Using a custom configuration file

​
On the command line

Pass the path to your custom file with the --config-file option. The custom file is merged over the defaults, so you only need to include the keys you want to override: 
trafilatura --config-file myfile.cfg -u https://example.org
All required variables must be present in the custom file, or defaults will be used for missing keys.

​
In Python

Load a complete settings file and pass it to any extraction function via the config= parameter:
from trafilatura import extract
from trafilatura.settings import use_config

# Load settings from a custom file
newconfig = use_config("myfile.cfg")

# Pass to extraction
result = extract(downloaded, config=newconfig)

# Or provide the file name directly (slightly slower)
result = extract(downloaded, settingsfile="myfile.cfg")

​
The Extractor class

Since version 1.9, all extraction parameters — including those that used to require a config file — can be bundled into an Extractor object and passed directly to extract() or bare_extraction(). This is the preferred approach for programmatic use because it is explicit, type-safe, and avoids repeating keyword arguments across many calls.

​
Import and instantiation

from trafilatura import extract
from trafilatura.settings import Extractor

options = Extractor(
    output_format="json",
    with_metadata=True,
    comments=False,
    tables=True,
    images=False,
    links=False,
    dedup=True,
    precision=True,
)

result = extract(html_content, options=options)

​
Key attributes

​
output_format
str
default:"txt"
Output format. Supported values: "csv", "html", "json", "markdown", "txt", "xml", "xmltei". Use "python" with bare_extraction() to receive a Document object.
​
fast
bool
default:"False"
Use faster heuristics and skip the backup extractor. Trades recall for speed.
​
focus
str
default:"balanced"
Extraction strategy. Set via precision=True (fewer but correct segments) or recall=True (more text, less strict). Passing both raises a warning and recall takes precedence.
​
comments
bool
default:"True"
Whether to extract comment sections alongside the main text.
​
formatting
bool | None
default:"None"
Preserve formatting elements such as bold, italic, and headings. Defaults to True when output_format is "markdown". Ignored for JSON output.
​
links
bool
default:"False"
Keep hyperlinks and their targets in the output (experimental).
​
images
bool
default:"False"
Include image references in the output (experimental).
​
tables
bool
default:"True"
Extract content from HTML <table> elements.
​
dedup
bool
default:"False"
Activate segment-level deduplication using the LRU cache.
​
lang
str | None
default:"None"
Target language as an ISO 639-1 code (e.g. "en", "de"). Documents detected as a different language are discarded.
​
with_metadata
bool
default:"False"
Extract and include metadata fields (title, author, date, etc.) in the output.
​
only_with_metadata
bool
default:"False"
Discard documents that do not have all three essential metadata fields: date, title, and url.
​
tei_validation
bool
default:"False"
Validate the XML-TEI output against the TEI standard. Only meaningful when output_format="xmltei".
​
date_params
dict | None
default:"None"
Pass custom parameters to htmldate for date extraction. Overrides the EXTENSIVE_DATE_SEARCH config value.
​
author_blacklist
set[str] | None
default:"None"
A set of author names to exclude from the output.
​
url_blacklist
set[str] | None
default:"None"
A set of URLs to filter out during extraction.

​
Reusing options across many pages

from trafilatura import extract
from trafilatura.settings import Extractor

# Define once, reuse everywhere
options = Extractor(
    output_format="xml",
    with_metadata=True,
    tables=False,
    dedup=True,
    lang="en",
)

pages = [html1, html2, html3]
results = [extract(page, options=options) for page in pages]
Passing an Extractor object via options= takes precedence over any individual keyword arguments passed to extract(). Build the Extractor once and reuse it for maximum efficiency.

​
Advanced: editing settings.py

For deeper customization — such as changing the list of HTML elements that are cleaned or kept, adjusting the LRU cache size (LRU_SIZE), configuring parallel processing (PARALLEL_CORES), or tuning file-output behavior — you can edit settings.py directly.
These are package-wide constants. Editing them changes the behavior of every function in the package. A local reinstall is required after each change.
1

Locate or clone the package

Find the locally installed copy (e.g. inside your virtualenv’s site-packages) or clone the repository:
git clone https://github.com/adbar/trafilatura.git
cd trafilatura
2

Edit settings.py

Open trafilatura/settings.py and modify the constants you need — for example, increase LRU_SIZE for larger deduplication caches or adjust MANUALLY_CLEANED to preserve specific HTML elements.
3

Reinstall the package

Apply the changes without pulling in fresh dependencies:
pip install --no-deps -U .

Build docs developers (and LLMs) love

Get started for freeTalk to us