Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Extracting the main content from web pages is harder than it looks. A page contains not just the article or blog post you want, but also headers, footers, sidebars, navigation menus, ads, and social media widgets — collectively known as boilerplate. The goal of main content extraction (also called boilerplate removal, DOM-based content extraction, or web page cleaning) is to return the central text while discarding everything else. This page documents how Trafilatura performs on this task compared to other Python tools, and explains how to reproduce the benchmark yourself.
To run the evaluation with the latest data and packages, see the corresponding README in the repository.

​
External evaluations

Trafilatura has been independently assessed across multiple benchmarks and academic studies:

ScrapingHub Article Extraction Benchmark

Trafilatura is the most efficient open-source library in ScrapingHub’s article extraction benchmark, which measures accuracy across a large and diverse set of web pages.

Lejeune & Barbaresi 2020

Ranked as the best overall tool in Bien choisir son outil d’extraction de contenu à partir du Web, a comparative study of content extraction tools for French web pages.

Bevendorff et al. 2023

Achieved the best single-tool score by ROUGE-LSum Mean F1 Page Score in An Empirical Comparison of Web Content Extraction Algorithms (Webis group, 2023).

Polish news comparison

Evaluated on a sample of Polish news texts and forums. This test set has since been incorporated into the internal benchmark; Trafilatura has improved further since then.

​
Benchmark results (2022-05-18)

The table below shows results on 750 documents containing 2,236 text segments and 2,250 boilerplate segments, measured with Python 3.8. Bold values indicate the best result in each column.
Python packagePrecisionRecallAccuracyF-ScoreSpeed (vs. baseline)
raw HTML0.5270.8740.5460.658—
html2text 2020.1.160.4860.7090.4810.5777.6×
html_text 0.5.20.5290.9580.5540.6822.2×
inscriptis 2.2.00.5340.9590.5630.6863.5×
newspaper3k 0.2.80.8950.5930.7620.71312×
jusText 3.0.0 (custom)0.8650.6500.7750.7425.2×
boilerpy3 1.0.6 (article mode)0.8140.7440.7870.7774.1×
baseline (text markup)0.7570.8270.7810.7901×
goose3 3.1.90.9340.6900.8210.79322×
readability-lxml 0.8.10.8910.7290.8200.8015.8×
news-please 1.5.220.8980.7340.8260.80861×
readabilipy 0.2.00.8770.8700.8740.874248×
trafilatura 1.2.2 (fast)0.9140.8860.9020.9004.8×
trafilatura 1.2.2 (precision)0.9320.8740.9050.9029.4×
trafilatura 1.2.2 (standard)0.9140.9040.9100.9097.1×
The Diff column shows execution time relative to the simple baseline. Lower is faster. The baseline extracts raw text from paragraph, code, and quote elements — a meaningful but unsophisticated approach.
Python packagePrecisionRecallAccuracyF-ScoreSpeed
raw HTML0.5270.8780.5470.659—
html2text 2020.1.160.4880.7140.4840.5808.9×
html_text 0.5.20.5260.9580.5480.6791.9×
inscriptis 1.10.5310.9580.5560.6832.4×
jusText 2.2.0 (custom)0.8700.5840.7490.6996.1×
newspaper3k 0.2.80.9210.5740.7630.70812.9×
boilerpy3 1.0.2 (article mode)0.8510.6960.7880.7664.8×
goose3 3.1.90.9500.6440.8060.76718.8×
baseline (text markup)0.7460.8040.7660.7741×
dragnet 2.0.40.9060.6890.8100.7833.1×
readability-lxml 0.8.10.9170.7160.8260.8045.9×
news-please 1.5.210.9240.7180.8300.80860×
trafilatura 0.8.2 (fast)0.9250.8680.8990.8963.9×
trafilatura 0.8.20.9340.8900.9140.9128.4×
Python packagePrecisionRecallAccuracyF-ScoreSpeed
raw HTML0.5240.8790.5430.657—
html2text 2020.1.160.4850.7180.4800.5798.4×
html_text 0.5.10.5210.9620.5420.6761.8×
inscriptis 1.00.5270.9650.5510.6811.9×
newspaper3k 0.2.80.9160.5770.7630.70811.8×
jusText 2.2.0 (tweaked)0.8670.6510.7770.7444.9×
goose3 3.1.60.9530.6350.8030.76217.3×
baseline (text markup)0.7380.8040.7600.7701×
boilerpy3 1.0.2 (article mode)0.8470.7110.7920.7734.4×
dragnet 2.0.40.9060.7040.8160.7922.8×
readability-lxml 0.8.10.9130.7390.8350.8175.4×
news-please 1.4.250.9180.7390.8370.81956.4×
trafilatura 0.5.10.9270.8540.8940.8893.1×
trafilatura 0.5.1 (+ fallbacks)0.9330.8850.9110.9086.8×

​
Understanding precision and recall

Content extraction always involves a trade-off:
  • High precision means less noise in the output, but you risk missing some relevant content. Tools like goose3 lean heavily in this direction.
  • High recall means you capture more of the main text, but at the cost of including some boilerplate. Tools like html_text and inscriptis — which simply strip all HTML tags — achieve near-perfect recall but cannot distinguish main content from sidebars or footers.
  • Balanced F-Score is what matters most in practice. Trafilatura’s standard mode achieves the best F-Score by combining a rule-based approach with algorithmic fallbacks.
Trafilatura itself exposes this trade-off through the favor_precision and favor_recall parameters, letting you shift the balance for your specific use case.

​
Alternatives

The following Python packages are relevant alternatives for HTML-to-text conversion and main content extraction.

​
Structure-preserving converters (no main-content focus)

These tools convert the entire HTML document to text or markup, without attempting to identify and isolate the main content area:
PackageDescription
html2textConverts HTML pages to Markdown-formatted text
html_textConverts HTML to plain text; achieves high recall but no content filtering
inscriptisHTML to text with particular emphasis on nested tables

​
Main content extraction tools

These packages, like Trafilatura, attempt to identify and return only the primary content of a page:
PackageNotes
boilerpy3Python port of the boilerpipe boilerplate removal algorithm
goose3Highest precision of the tested tools, but low recall; does not preserve markup
jusTextDesigned for linguistic resources; highly configurable
newspaper3kOriented towards news articles; additional functions but no structured output
news-pleaseNews crawler with structured information extraction; very slow
readability-lxmlCleans the page and preserves some markup
readabilipyPython wrapper for Mozilla’s Readability (Node.js); extremely slow
Some of these packages are no longer actively maintained. dragnet (referenced in older benchmarks) and some versions of newspaper3k are provided for historical reference only. Check the respective repositories for current maintenance status before using them in production.
The benchmark also includes boilerpy3, newspaper3k, and readabilipy with error handling enabled — these packages produce errors on some HTML files in the test set (malformed HTML, encoding bugs). Errors are silently ignored to complete the run, which may slightly inflate their apparent performance.

​
Benchmark methodology

Test set: Documents typical of Internet articles and blog posts, drawn primarily from large German web page collections (DWDS). For completeness, documents in English, French, and other European languages, as well as Chinese and Arabic (~20–30% of the total), are included. Some documents are deliberately challenging due to mixed content (lists, tables) or invalid HTML. Evaluation: Key document segments are annotated as either text (wanted content) or boilerplate (navigation, headers, footers, sidebars, social links, addresses). The evaluation measures how accurately each tool makes this distinction. Reproducibility: Clone the repository, install all packages, and run the evaluation script: 
git clone https://github.com/adbar/trafilatura.git
cd trafilatura
pip install -e ".[all]"
# See tests/README.rst for detailed instructions
Full instructions are in tests/README.rst.

Build docs developers (and LLMs) love

Get started for freeTalk to us