Documentation Index
Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt
Use this file to discover all available pages before exploring further.
Trafilatura ships with a robust command-line interface that requires no Python code. You can download web pages, extract their main text, process entire directories, batch-download URL lists, discover links from feeds and sitemaps, and crawl websites — all from the terminal.
All commands are followed by pressing Enter. Use trafilatura -h to display the full help message at any time.
Basic Usage
Download and Extract a Single URL
# extract main text as plain text (default)
trafilatura -u "https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/"
# extract with XML structure preserved
trafilatura --xml --URL "https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/"
# display the help message
trafilatura -h
# process an existing local file
cat myfile.html | trafilatura
# alternative stdin syntax
< myfile.html trafilatura
# use a custom download tool and pipe to trafilatura
wget -qO- "https://de.creativecommons.org/index.php/was-ist-cc/" | trafilatura
Single URL
URL List File
Directory
Parallel Processing
Blacklist
# -u / --URL: download and extract a single URL
trafilatura -u "https://www.example.org/article"
trafilatura --URL "https://www.example.org/article"
# -i / --input-file: read one URL per line from a file
# lines not starting with a valid URL are silently skipped
trafilatura -i links.txt -o output/
trafilatura --input-file links.txt --output-dir output/
# with XML output
trafilatura --xml -i list.txt -o xmlfiles/
# --input-dir: process all HTML files in a local directory
trafilatura --input-dir html-pages/ -o extracted/
# preserve the original directory structure in the output
trafilatura --input-dir html-pages/ -o extracted/ --keep-dirs
# --parallel N: use N threads/processes for downloads and processing
trafilatura -i links.txt -o output/ --parallel 4
# -b / --blacklist: skip URLs listed in a file (one URL per line)
trafilatura -i links.txt -o output/ -b unwanted-urls.txt
trafilatura -i links.txt -o output/ --blacklist unwanted-urls.txt
Trafilatura waits a few seconds between requests to the same domain by default. Some websites may block the requests user-agent after a high number of requests in a short period.
Output Options
# -o / --output-dir: write results to a directory (created if missing)
trafilatura -i list.txt -o results/
# --backup-dir: keep a copy of the original downloaded HTML
trafilatura --input-file links.txt --output-dir converted/ --backup-dir html-sources/ --xml
# --keep-dirs: mirror input directory structure in output (requires -o)
trafilatura --input-dir pages/ --output-dir results/ --keep-dirs
# --list: print discovered URLs without downloading or extracting them
trafilatura --feed "https://www.dwds.de/" --list
trafilatura --sitemap "https://www.sitemaps.org/" --list > mylinks.txt
When no --output-dir is specified, results are printed to standard output (STDOUT).
Navigation and Link Discovery
Trafilatura supports three link-discovery strategies: feeds (fresh content), sitemaps (exhaustive page lists), and crawling (following internal links).
Feeds
Sitemaps
Crawl
Probe
Internet Archive
# auto-detect feeds starting from a homepage
trafilatura --feed "https://www.dwds.de/" --list
# use a known feed URL directly
trafilatura --feed "https://www.dwds.de/api/feed/themenglossar/Corona" --list
# process a list of sources in parallel
trafilatura -i mylist.txt --feed --list
# auto-detect and list all pages via sitemap
trafilatura --sitemap "https://www.sitemaps.org/" --list > mylinks.txt
# use a known sitemap URL directly
trafilatura --sitemap "https://www.sitemaps.org/sitemap.xml" --list
# filter by target language
trafilatura --sitemap "https://www.sitemaps.org/" --list --target-language "de"
# --crawl: follow internal links from the given URL (fixed depth)
trafilatura --crawl "https://www.example.org/" --list
# --explore: combination of sitemap and crawl strategies
trafilatura --explore "https://www.example.org/" --list
# --probe: check pages for extractable content
# works best when combined with --target-language
trafilatura --probe "https://www.example.org/" --target-language "en"
# --archived: fall back to the Wayback Machine for failed downloads
trafilatura -i links.txt -o output/ --archived
URL Filtering
Apply string-pattern filters to restrict which discovered URLs are processed or listed.
# only include URLs containing a specific path segment
trafilatura --sitemap "https://www.sitemaps.org/" --list --url-filter "https://www.sitemaps.org/de"
# filter by keyword pattern
trafilatura --sitemap "https://www.sitemaps.org/" --list --url-filter "protocol"
# using a subpath as the starting URL also acts as a filter
trafilatura --sitemap "https://www.sitemaps.org/de/" --list
Speed
HTML Elements
Metadata
Language
Quality
Config File
# -f / --fast: skip fallback extraction (approx. 2× faster)
trafilatura -u "https://www.example.org/article" -f
trafilatura -u "https://www.example.org/article" --fast
# include text formatting (bold, italic, etc.)
trafilatura -u "https://www.example.org/article" --formatting
# include link targets from href attributes (experimental)
trafilatura -u "https://www.example.org/article" --links
# include image sources and attributes (experimental)
trafilatura -u "https://www.example.org/article" --images
# disable comment extraction (enabled by default)
trafilatura -u "https://www.example.org/article" --no-comments
# disable table extraction (enabled by default)
trafilatura -u "https://www.example.org/article" --no-tables
# --with-metadata: include extracted metadata in output
trafilatura -u "https://www.example.org/article" --with-metadata --xml
# --only-with-metadata: skip documents missing title, URL, or date
trafilatura -i list.txt -o output/ --only-with-metadata
# --target-language: filter output by ISO 639-1 language code
# requires: pip install trafilatura[all]
trafilatura -u "https://www.example.org" --target-language "en"
trafilatura --sitemap "https://www.sitemaps.org/" --list --target-language "de"
# --precision: favor accuracy, reduce noise
trafilatura -u "https://www.example.org/article" --precision
# --recall: favor coverage, capture more text
trafilatura -u "https://www.example.org/article" --recall
# --deduplicate: filter out duplicate documents and sections
trafilatura -i list.txt -o output/ --deduplicate
# --config-file: override default extraction parameters with a custom settings file
trafilatura -i list.txt -o output/ --config-file my-settings.cfg
Certain elements are only visible in the output if the chosen format supports them (e.g. images require XML or XML-TEI). Including extra structural elements works best with --xml or --xmltei.
Choose an output format using a shorthand flag or the explicit --output-format option.
Shorthand Flags
--output-format
# plain text (default, no flag needed)
trafilatura -u "https://www.example.org/article"
# CSV
trafilatura -u "https://www.example.org/article" --csv
# HTML (v1.11+)
trafilatura -u "https://www.example.org/article" --html
# JSON
trafilatura -u "https://www.example.org/article" --json
# Markdown (v1.9+)
trafilatura -u "https://www.example.org/article" --markdown
# XML
trafilatura -u "https://www.example.org/article" --xml
# XML-TEI
trafilatura -u "https://www.example.org/article" --xmltei
# --output-format accepts: csv, json, html, markdown, txt, xml, xmltei
trafilatura -u "https://www.example.org/article" --output-format json
trafilatura -u "https://www.example.org/article" --output-format xmltei
# batch processing with explicit format
trafilatura -i list.txt -o output/ --output-format xml
Combining txt or csv with --formatting or --links automatically triggers Markdown output. Selecting --markdown always enables text formatting. The --formatting flag has no effect on JSON output.
TEI Validation
# --validate-tei: validate XML-TEI output against the TEI standard
trafilatura -u "https://www.example.org/article" --xmltei --validate-tei
trafilatura -i list.txt -o output/ --xmltei --validate-tei
Verbosity and Version
# -v: increase logging verbosity (INFO level)
trafilatura -u "https://www.example.org" -v
# -vv: maximum verbosity (DEBUG level)
trafilatura -u "https://www.example.org" -vv
# --version: print version information and exit
trafilatura --version
Full Argument Reference
trafilatura [-h] [-i INPUTFILE | --input-dir INPUTDIR | -u URL]
[--parallel PARALLEL] [-b BLACKLIST] [--list]
[-o OUTPUTDIR] [--backup-dir BACKUP_DIR] [--keep-dirs]
[--feed [FEED] | --sitemap [SITEMAP] | --crawl [CRAWL] |
--explore [EXPLORE] | --probe [PROBE]] [--archived]
[--url-filter URL_FILTER [URL_FILTER ...]] [-f]
[--formatting] [--links] [--images] [--no-comments]
[--no-tables] [--only-with-metadata] [--with-metadata]
[--target-language TARGET_LANGUAGE] [--deduplicate]
[--config-file CONFIG_FILE] [--precision] [--recall]
[--output-format {csv,json,html,markdown,txt,xml,xmltei} |
--csv | --html | --json | --markdown | --xml | --xmltei]
[--validate-tei] [-v] [--version]
Deprecated Arguments
The following CLI arguments have been removed. Use their replacements instead:
| Deprecated | Replacement |
|---|
--nocomments | --no-comments |
--notables | --no-tables |
--inputfile | --input-file |
--inputdir | --input-dir |
--outputdir | --output-dir |
-out | --output-format |
--hash-as-name | Hashes are used by default |
--with-metadata (old meaning) | --only-with-metadata (only docs with required metadata) |