Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Trafilatura turns raw HTML into clean, structured text. It discovers, downloads, and extracts the main content from web pages — stripping boilerplate, navigation, and ads — while preserving metadata like title, author, date, and tags. Use it as a Python library or as a command-line tool, with no database required.

Installation

Install via pip and set up optional speed and language-detection extras.

Quickstart

Fetch a URL and extract clean text in three lines of Python.

Python Usage

Full API walkthrough: extraction options, output formats, and metadata.

CLI Usage

Process URLs, files, and directories from the command line.

Web Discovery

Download pages, parse feeds and sitemaps, and run focused crawls.

API Reference

Complete reference for every public function and configuration option.

​
What Trafilatura Does

Trafilatura handles the full pipeline from URL to clean text:
1

Discover

Find URLs through RSS/Atom feeds, XML sitemaps, or focused web crawling. Filter by language, URL pattern, or domain.
2

Download

Fetch pages with polite rate-limiting, parallel threads, and automatic fallback to the Internet Archive for unavailable pages.
3

Extract

Remove boilerplate using a combination of custom heuristics, jusText, and readability algorithms. Tune precision vs. recall to match your use case.
4

Output

Export as plain text, Markdown, JSON, CSV, HTML, XML, or TEI-XML — with or without metadata (title, author, date, tags).

​
Key Features

Multiple Output Formats

TXT, CSV, JSON, HTML, Markdown, XML, and TEI-XML. Combine with metadata extraction for structured datasets.

Precision & Recall Tuning

Pass favor_precision=True or favor_recall=True to balance noise reduction against content completeness.

Feed & Sitemap Support

Discover URLs from Atom, RSS, and JSON feeds, or crawl XML sitemaps with language filtering.

Focused Web Crawler

Follow internal links with politeness rules, deduplication, and configurable depth limits.

Parallel Downloads

Process lists of URLs concurrently with --parallel or threaded Python code.

Deduplication

Remove duplicate segments and near-duplicate documents using LRU caches and locality-sensitive hashing.

​
Quick Example

from trafilatura import fetch_url, extract

# Download a page and extract its main text
downloaded = fetch_url("https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/")
result = extract(downloaded)
print(result)
Or from the command line: 
trafilatura -u "https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/"
Trafilatura is used by HuggingFace, IBM, Microsoft Research, Stanford, and thousands of open-source projects. It ranks as the most accurate open-source web text extractor in multiple independent benchmarks.

Build docs developers (and LLMs) love

Get started for freeTalk to us