Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Trafilatura is a Python package and command-line tool designed to gather text on the Web and simplify the process of turning raw HTML into structured, meaningful data. It includes all the necessary discovery and text-processing components to perform web crawling, downloads, scraping, and extraction of main texts, metadata, and comments. No database is required — output converts directly to commonly used formats, making it handy and modular from day one. Going from raw HTML to essential content alleviates many problems related to text quality: Trafilatura focuses on the actual content, avoids the noise caused by recurring elements like headers and footers, and makes sense of the data and metadata using selected information. The extractor strikes a balance between limiting noise (precision) and including all valid parts (recall). It is robust, actively maintained, and reasonably fast.

​
Key Features

Web Crawling & Discovery

Supports sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS) with smart URL management including filtering and deduplication.

Flexible Input Sources

Processes live URLs, download queues, previously saved HTML files, and already-parsed HTML trees in parallel.

Robust Text Extraction

Extracts main text using common patterns and generic algorithms (jusText and readability), plus formatting such as paragraphs, titles, lists, quotes, and code blocks.

Rich Metadata Extraction

Pulls title, author, publication date, site name, categories, and tags from web pages automatically.

Multiple Output Formats

Outputs to TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI — pick whatever fits your pipeline.

Optional Add-ons

Language detection on extracted text, faster encoding detection, SOCKS proxy support, and additional compression algorithms are available as optional extras.

​
How It Works

Trafilatura operates in three stages:
  1. Download — fetch_url() retrieves the raw HTML from a URL, managing HTTP headers, redirects, and polite crawl delays.
  2. Extract — extract() identifies the main content block, strips boilerplate (navigation, ads, footers), and returns clean text or structured data.
  3. Output — The result is serialised to your chosen format: plain text, Markdown, JSON with metadata, XML, XML-TEI, or CSV.
The same pipeline works on already-downloaded files and in-memory HTML strings, so you can integrate Trafilatura into existing scraping workflows without changing your download layer. 
import trafilatura

# Download and extract in two lines
downloaded = trafilatura.fetch_url("https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/")
text = trafilatura.extract(downloaded)
print(text)

​
Software Ecosystem

Trafilatura is the centrepiece of a broader set of jointly developed packages for web data extraction and analysis:
  • htmldate — fast and accurate extraction of publication and modification dates from web pages.
  • courlan — URL sampling, filtering, and management for crawling workflows.
  • adblock-rs (via Python bindings) — ad and tracker URL filtering.
Together these packages form an integrated toolkit for building text corpora from web data. Blog posts covering updates, tutorials, and text-mining topics are published at Bits of Language.

​
Trusted by the Community

Trafilatura is integrated into thousands of projects and is actively used by companies such as HuggingFace, IBM, and Microsoft Research, as well as academic institutions including the Allen Institute, Stanford, the Tokyo Institute of Technology, and the University of Munich.
Trafilatura is also widely used in academic research, chiefly for data acquisition. If you use it in a publication, please cite:
Barbaresi, A. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction, Proceedings of ACL/IJCNLP 2021: System Demonstrations, pp. 122–131. DOI: 10.18653/v1/2021.acl-demo.15 
@inproceedings{barbaresi-2021-trafilatura,
  title     = {{Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction}},
  author    = "Barbaresi, Adrien",
  booktitle = "Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations",
  pages     = "122--131",
  publisher = "Association for Computational Linguistics",
  url       = "https://aclanthology.org/2021.acl-demo.15",
  year      = 2021,
}

​
Get Started

Installation

Install Trafilatura with pip, add optional extras, and configure the CLI entry point.

Quickstart

Fetch a URL and extract clean text or structured JSON in under five minutes.

Build docs developers (and LLMs) love

Get started for freeTalk to us