Extractor & use_config(): Trafilatura Settings API

The Extractor class is the single source of truth for all extraction options in Trafilatura. Rather than passing a long list of keyword arguments to extract() on every call, you construct one Extractor object with your desired settings, and then reuse it across as many calls as you like. This makes batch-processing pipelines cleaner, allows settings to be serialised and logged, and gives you a named, inspectable object to pass through your own code.

Extractor class

trafilatura.settings.Extractor is a slot-based Python class (equivalent to a dataclass(slots=True)) that stores every option that governs a single extraction run.

Constructor

from trafilatura.settings import Extractor

options = Extractor(
    output_format="markdown",
    comments=False,
    formatting=True,
    tables=True,
    images=True,
    links=True,
    dedup=True,
)

All constructor parameters are keyword-only. Positional arguments are not accepted.

Setting attributes after construction

Every slot is a plain writable attribute — you can update it at any time before passing the object to extract():

options = Extractor()
options.lang = "en"
options.dedup = True
options.focus = "precision"

Passing to `extract()`

import trafilatura
from trafilatura.settings import Extractor

options = Extractor(output_format="json", with_metadata=True)

html = trafilatura.fetch_url("https://example.org/article")
result = trafilatura.extract(html, options=options)

Parameters

config

ConfigParser

default:"DEFAULT_CONFIG"

A configparser.ConfigParser instance that supplies numeric thresholds and download settings. Defaults to DEFAULT_CONFIG (the bundled settings.cfg). Pass the result of use_config() here when you need custom thresholds.

output_format

str

default:"\"txt\""

Output serialisation format. Must be one of the values in SUPPORTED_FORMATS. Accepted values: "csv", "json", "html", "markdown", "txt", "xml", "xmltei", "python" (the last one is for bare_extraction() only). Raises AttributeError if an unsupported string is supplied.Stored internally as options.format.

fast

bool

default:"False"

When True, Trafilatura skips the fallback extraction pass (readability and justext). Faster, but may miss content on difficult pages.Stored internally as options.fast.

precision

bool

default:"False"

Favour precision over recall during content selection. Sets options.focus to "precision". Mutually exclusive with recall; if both are supplied, recall takes precedence (with a warning).

recall

bool

default:"False"

Favour recall over precision during content selection. Sets options.focus to "recall". Mutually exclusive with precision.

comments

bool

default:"True"

Include comment sections in the extraction output.

formatting

bool | None

default:"None"

Preserve inline formatting (bold, italic, headings, code spans) in the output. Defaults to True when output_format="markdown", and False for all other formats. An explicit False overrides the markdown default. Has no effect on JSON output (a warning is emitted in that case).

links

bool

default:"False"

Preserve hyperlinks in the output.

images

bool

default:"False"

Include image references in the output.

tables

bool

default:"True"

Extract HTML table content.

dedup

bool

default:"False"

Activate in-process deduplication. Repeated text blocks are detected via an LRU cache and discarded when they appear more than max_repetitions times. See also deduplication.duplicate_test().

lang

str | None

default:"None"

BCP 47 language code (e.g. "en", "de", "fr"). When set, documents that are detected as a different language are discarded. Requires py3langid to be installed.

url

str | None

default:"None"

Canonical URL of the document being processed. Used to populate metadata and to derive options.source. Encoded to UTF-8 before storage.

source

str | None

default:"None"

Arbitrary label attached to the extraction run — useful for debugging and log messages. When both url and source are given, url takes precedence. Encoded to UTF-8 before storage.

with_metadata

bool

default:"False"

Include document metadata (title, author, date, URL, …) in the output. Automatically set to True when only_with_metadata, url_blacklist, author_blacklist, or output_format="xmltei" is active.

only_with_metadata

bool

default:"False"

Discard documents for which no metadata could be extracted. Implies with_metadata=True.

tei_validation

bool

default:"False"

Validate the TEI-XML output against the bundled DTD after generation. Only meaningful when output_format="xmltei". A warning is emitted if this flag is set for any other format.

author_blacklist

set[str] | None

default:"None"

A set of author strings to filter out. Documents attributed to any of these authors are discarded. Implies with_metadata=True.

url_blacklist

set[str] | None

default:"None"

A set of URL strings to skip. Implies with_metadata=True.

date_params

dict[str, str] | None

default:"None"

Parameters forwarded to htmldate for date extraction. When None, defaults are derived from the EXTENSIVE_DATE_SEARCH setting in the config file.

Config-derived attributes

The following integer attributes are not constructor parameters. They are read automatically from the config ConfigParser via CONFIG_MAPPING and stored as plain int slots:

Attribute	Config key	Default
`min_extracted_size`	`MIN_EXTRACTED_SIZE`	`250`
`min_output_size`	`MIN_OUTPUT_SIZE`	`1`
`min_output_comm_size`	`MIN_OUTPUT_COMM_SIZE`	`1`
`min_extracted_comm_size`	`MIN_EXTRACTED_COMM_SIZE`	`1`
`min_duplcheck_size`	`MIN_DUPLCHECK_SIZE`	`100`
`max_repetitions`	`MAX_REPETITIONS`	`2`
`max_file_size`	`MAX_FILE_SIZE`	`20000000`
`min_file_size`	`MIN_FILE_SIZE`	`10`

max_tree_size is also read from config (MAX_TREE_SIZE) but is stored as int | None because the config key accepts an empty value to disable the limit.

use_config()

trafilatura.settings.use_config reads and parses a settings.cfg file and returns a ready-to-use ConfigParser. Pass the result to Extractor(config=…) to apply custom thresholds.

from trafilatura.settings import use_config

config = use_config()                        # load built-in defaults
config = use_config(filename="my.cfg")       # load a custom file on top of defaults
config = use_config(config=existing_parser)  # pass through an existing ConfigParser

Parameters

filename

str | None

default:"None"

Path to a custom .cfg file. The file must exist; a FileNotFoundError is raised otherwise. Built-in defaults are always seeded first so that your partial file only needs to override the keys it changes.

config

ConfigParser | None

default:"None"

Pass an already-constructed ConfigParser to be used as-is. When this argument is not None, filename is ignored and the object is returned unchanged.

Returns ConfigParser — the populated configuration object.

Examples

from trafilatura.settings import use_config

config = use_config()
print(config.get("DEFAULT", "MIN_EXTRACTED_SIZE"))  # "250"

Always deepcopy DEFAULT_CONFIG before mutating it. Because DEFAULT_CONFIG is a module-level singleton, mutating it in place will affect every subsequent Extractor() call that does not pass an explicit config argument.

DEFAULT_CONFIG

DEFAULT_CONFIG is the pre-parsed ConfigParser built from the bundled settings.cfg at import time. It is available for direct use when you only need to read a default threshold without constructing a full Extractor:

from trafilatura.settings import DEFAULT_CONFIG

min_size = DEFAULT_CONFIG.getint("DEFAULT", "MIN_EXTRACTED_SIZE")  # 250
extensive = DEFAULT_CONFIG.getboolean("DEFAULT", "EXTENSIVE_DATE_SEARCH")  # True

Do not mutate DEFAULT_CONFIG directly. Use deepcopy (as shown above) or call use_config(filename=…) to obtain a separate instance.

The full set of keys and their built-in values lives in trafilatura/settings.cfg:

Key	Default	Description
`DOWNLOAD_TIMEOUT`	`30`	HTTP request timeout in seconds
`MAX_FILE_SIZE`	`20000000`	Maximum document size in bytes
`MIN_FILE_SIZE`	`10`	Minimum document size in bytes
`SLEEP_TIME`	`5.0`	Pause between download requests
`MAX_REDIRECTS`	`2`	Maximum HTTP redirects to follow
`MIN_EXTRACTED_SIZE`	`250`	Minimum main-text character count
`MIN_EXTRACTED_COMM_SIZE`	`1`	Minimum comment character count
`MIN_OUTPUT_SIZE`	`1`	Minimum output character count
`MIN_OUTPUT_COMM_SIZE`	`1`	Minimum comment output character count
`MAX_TREE_SIZE`	(empty)	Max elements in parsed HTML tree; empty = no limit
`EXTRACTION_TIMEOUT`	`30`	Per-file timeout for CLI batch mode
`MIN_DUPLCHECK_SIZE`	`100`	Minimum string length to check for duplicates
`MAX_REPETITIONS`	`2`	How many times a block may repeat before being dropped
`EXTENSIVE_DATE_SEARCH`	`on`	Enable exhaustive date-finding heuristics
`EXTERNAL_URLS`	`off`	Follow external URLs in feeds and sitemaps

SUPPORTED_FORMATS

SUPPORTED_FORMATS is the set of output format strings accepted by Extractor (and therefore by extract()). The CLI accepts a subset of these (SUPPORTED_FMT_CLI) that excludes "python", which is only meaningful for bare_extraction().

from trafilatura.settings import SUPPORTED_FORMATS

print(SUPPORTED_FORMATS)
# {'csv', 'html', 'json', 'markdown', 'python', 'txt', 'xml', 'xmltei'}

Format	Description
`"csv"`	Tab-separated values, one document per row
`"html"`	Cleaned HTML
`"json"`	JSON object with all fields
`"markdown"`	Markdown with optional formatting markers
`"python"`	Python dictionary (via `bare_extraction()` only)
`"txt"`	Plain text (default)
`"xml"`	Trafilatura XML dialect
`"xmltei"`	TEI-XML compliant output

Core Functions

Configuration

Extractor & use_config(): Trafilatura Settings API

Extractor class

Constructor

Setting attributes after construction

Passing to `extract()`

Parameters

Config-derived attributes

use_config()

Parameters

Examples

DEFAULT_CONFIG

SUPPORTED_FORMATS

Build docs developers (and LLMs) love

Core Functions

Configuration

Documentation Index

​Extractor class

​Constructor

​Setting attributes after construction

​Passing to extract()

​Parameters

​Config-derived attributes

​use_config()

​Parameters

​Examples

​DEFAULT_CONFIG

​SUPPORTED_FORMATS

Build docs developers (and LLMs) love

Extractor class

Constructor

Setting attributes after construction

Passing to `extract()`

Parameters

Config-derived attributes

use_config()

Parameters

Examples

DEFAULT_CONFIG

SUPPORTED_FORMATS