Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Trafilatura exposes a set of utility functions used internally during extraction that are also useful when you need fine-grained control over text processing, validation, caching, or deduplication. This page documents the public-facing helpers spread across trafilatura.utils, trafilatura.xml, trafilatura.readability_lxml, trafilatura.meta, and trafilatura.deduplication.

​
utils.sanitize()

trafilatura.utils.sanitize cleans and normalises a text string by removing HTML space entities ( , 
, 
), stripping control characters, collapsing redundant whitespace, and filtering blank lines. Optionally, it can process the string as a single logical line rather than splitting on newlines. 
from trafilatura.utils import sanitize

raw = "Hello\x00 World\u00a0!\n\n  Extra   spaces  "
clean = sanitize(raw)
# "Hello World !\nExtra spaces"

​
Parameters

​
text
str
The input string to clean. If None is passed, None is returned.
​
preserve_space
bool
default:"False"
When True, internal newlines and spacing are not collapsed. Useful for preserving the layout of <pre> / <code> blocks.
​
trailing_space
bool
default:"False"
When True, the entire string is treated as a single line and leading/trailing spaces adjacent to the content are preserved for inline formatting contexts.
Returns str | None — the cleaned string, or None when text is None or consists entirely of whitespace after cleaning.

​
Example

from trafilatura.utils import sanitize

# Basic clean-up
print(sanitize("Line 1\n\nLine 2\n"))
# "Line 1\nLine 2"

# Preserve spacing (e.g. inside a code block)
print(sanitize("  x = 1\n  y = 2\n", preserve_space=True))
# "  x = 1\n  y = 2\n"

# Inline context: trailing_space keeps the flanking spaces for later joining
print(sanitize(" bold text ", trailing_space=True))
# " bold text "

​
utils.trim()

trafilatura.utils.trim strips all leading and trailing whitespace from a string and collapses any internal runs of whitespace (including Unicode spaces and newlines) into single ASCII spaces. It is LRU-cached up to 1 024 unique inputs. 
from trafilatura.utils import trim

print(trim("  hello   world  "))  # "hello world"
print(trim(""))                   # ""
print(trim(None))                 # ""

​
Parameters

​
string
str
The string to normalise. Returns "" on None or non-string input rather than raising an exception.
Returns str — the trimmed string. Never returns None.

​
Example

from trafilatura.utils import trim

title = "\t My  Article  Title \n"
print(trim(title))  # "My Article Title"
Because trim is LRU-cached, avoid calling it with very large strings in a tight loop — prefer sanitize() for multi-line content and reserve trim() for short, frequently-repeated strings like titles or author names.

​
utils.decode_file()

trafilatura.utils.decode_file converts raw bytes to a Unicode string. It handles compressed payloads transparently (gzip, zstandard, brotli, zlib), auto-detects the character encoding using cchardet (if installed) and charset-normalizer, and falls back to a UTF-8 decode with replacement characters if all else fails. 
from trafilatura.utils import decode_file

with open("page.html.gz", "rb") as f:
    raw = f.read()

html_string = decode_file(raw)

​
Parameters

​
filecontent
bytes | str
The raw file content to decode. If a str is passed it is returned unchanged, making the function safe to call unconditionally.
Returns str — a decoded Unicode string.

​
Example

from trafilatura.utils import decode_file

with open("article.html", "rb") as fh:
    html = decode_file(fh.read())

print(type(html))  # <class 'str'>
Compression detection is based on magic bytes (file signatures), not HTTP headers. This makes it safe to use on files saved to disk where headers are unavailable.

​
xml.xmltotxt()

trafilatura.xml.xmltotxt serialises an lxml element tree produced by Trafilatura’s internal extraction pipeline to a plain-text or Markdown string. When include_formatting=True the function deep-copies the tree before modifying it, so the caller’s element is never mutated. 
from trafilatura.xml import xmltotxt
from lxml.etree import fromstring

tree = fromstring("<body><p>Hello <hi rend=\"#b\">world</hi></p></body>")
print(xmltotxt(tree, include_formatting=False))  # "Hello world"
print(xmltotxt(tree, include_formatting=True))   # "Hello **world**"

​
Parameters

​
xmloutput
lxml.etree._Element | None
The root element to serialise. Returns an empty string when None.
​
include_formatting
bool
When True, inline formatting tags (<hi>, <del>, <ref>, <code>) are converted to their Markdown equivalents and LaTeX math delimiters are rewritten to $…$ / $$…$$. When False, only plain text is emitted.
Returns str — the serialised text content.

​
Example

import trafilatura
from trafilatura.xml import xmltotxt

result = trafilatura.bare_extraction(
    "<html><body><article><p>Some <b>bold</b> text.</p></article></body></html>",
    output_format="python",
    include_formatting=True,
)

if result and result.get("body") is not None:
    text = xmltotxt(result["body"], include_formatting=True)
    print(text)  # "Some **bold** text."

​
xml.validate_tei()

trafilatura.xml.validate_tei checks whether an lxml element tree conforms to the TEI Corpus DTD bundled with Trafilatura (trafilatura/data/tei_corpus.dtd). The DTD object is loaded lazily and cached as a module-level global after the first call. 
from trafilatura.xml import validate_tei
from lxml.etree import fromstring

tree = fromstring("<TEI><text><body><div><p>Hello</p></div></body></text></TEI>")
is_valid = validate_tei(tree)
print(is_valid)  # True or False

​
Parameters

​
xmldoc
lxml.etree._Element
The root element of the document to validate. Typically the <TEI> element produced by trafilatura.xml.build_tei_output().
Returns bool — True if the document validates, False otherwise. A logging.WARNING is emitted that includes the first DTD error when validation fails.

​
Example

import trafilatura
from trafilatura.settings import Extractor
from trafilatura.xml import validate_tei, build_tei_output
from lxml.etree import fromstring

html = trafilatura.fetch_url("https://example.org/article")
options = Extractor(output_format="xmltei", tei_validation=True)
result = trafilatura.extract(html, options=options)
# Validation is run automatically when tei_validation=True.
# Call validate_tei() directly if you need the boolean result:
In normal workflows, set tei_validation=True on your Extractor and let Trafilatura run validation automatically. Call validate_tei() directly only when you have built or modified a TEI tree yourself.

​
readability_lxml.is_probably_readerable()

trafilatura.readability_lxml.is_probably_readerable is a Python port of Mozilla’s isProbablyReaderable function from Readability.js. It gives a fast, cheap estimate of whether a page is worth attempting full extraction on, without actually running the extraction pipeline. The heuristic scores visible <p>, <pre>, and <article> nodes (and parents of <div><br> constructs) by their text length, skipping nodes that are hidden or whose class/id attributes match a list of unlikely-candidate patterns. When the cumulative score exceeds a configurable threshold the page is considered readerable. 
from trafilatura.readability_lxml import is_probably_readerable

with open("page.html") as f:
    html = f.read()

if is_probably_readerable(html):
    # proceed with full extraction
    ...

​
Parameters

​
html
lxml.html.HtmlElement
The HTML document to evaluate. The formal type annotation is HtmlElement; in practice the argument is passed through trafilatura.utils.load_html() internally, so raw HTML strings and bytes objects are also accepted.
​
options
dict[str, Any] | None
default:"None"
Optional dictionary to override internal thresholds:
  • min_content_length (int, default 140): minimum text length for a node to contribute to the score.
  • min_score (int, default 20): cumulative score threshold above which the page is considered readerable.
  • visibility_checker (callable, default is_node_visible): a function (HtmlElement) -> bool used to skip hidden nodes.
Returns bool — True if the page is likely to contain extractable article content.

​
Example

from trafilatura.readability_lxml import is_probably_readerable
import trafilatura

url = "https://example.org/news/story"
html = trafilatura.fetch_url(url)

# Quick pre-filter before spending time on full extraction
if is_probably_readerable(html):
    text = trafilatura.extract(html)
    print(text)
else:
    print("Page does not appear to contain article content.")

​
meta.reset_caches()

trafilatura.meta.reset_caches clears every LRU cache used internally by Trafilatura and its dependencies (htmldate, courlan, justext) and triggers a garbage-collection cycle. Calling it periodically in long-running processes prevents unbounded memory growth from cached function results. 
from trafilatura.meta import reset_caches

reset_caches()

​
Parameters

None. The function takes no arguments. Returns None.

​
When to use

Long-running crawlers

Call reset_caches() every N documents or every few hours to keep RSS steady when processing millions of pages.

Memory-constrained environments

Call before processing a new large batch to reclaim memory held by cached text fragments from the previous batch.

​
Example

import trafilatura
from trafilatura.meta import reset_caches

urls = load_urls()  # your list of millions of URLs

for i, url in enumerate(urls):
    html = trafilatura.fetch_url(url)
    result = trafilatura.extract(html)
    process(result)

    # Flush caches every 10 000 documents
    if i % 10_000 == 0:
        reset_caches()
The following caches are cleared:
  • justext.core.define_stoplist
  • All htmldate and charset_normalizer caches (via htmldate.meta.reset_caches)
  • All courlan caches (via courlan.meta.clear_caches)
  • deduplication.is_similar_domain
  • utils.line_processing
  • utils.return_printables_and_spaces
  • utils.trim
  • deduplication.LRU_TEST (the deduplication LRU store)
  • deduplication._vector_to_add

​
deduplication.duplicate_test()

trafilatura.deduplication.duplicate_test checks whether the concatenated text content of an lxml element has been seen too many times during the current session. It uses an LRU cache (LRU_TEST) to track occurrence counts. Text shorter than options.min_duplcheck_size is always allowed through without a cache lookup. 
from trafilatura.deduplication import duplicate_test
from trafilatura.settings import Extractor
from lxml.etree import fromstring

options = Extractor(dedup=True)
element = fromstring("<p>This is a repeated paragraph.</p>")

if not duplicate_test(element, options):
    print("content is unique — keep it")
else:
    print("content is a duplicate — discard it")

​
Parameters

​
element
lxml.etree._Element
The element whose text content (all descendant text nodes joined) is tested. The test string is produced by trim(" ".join(element.itertext())).
​
options
trafilatura.settings.Extractor
The current extraction options. Two fields are used:
  • options.min_duplcheck_size — strings shorter than this are not checked.
  • options.max_repetitions — the number of allowed occurrences before the element is considered a duplicate.
Returns bool — True when the element is a duplicate and should be discarded, False when it should be kept.

​
Example

from trafilatura.deduplication import duplicate_test
from trafilatura.settings import Extractor
from lxml.etree import fromstring

options = Extractor(dedup=True)  # uses min_duplcheck_size=100, max_repetitions=2

boilerplate = fromstring("<p>" + "Subscribe to our newsletter. " * 10 + "</p>")

for _ in range(4):
    is_dup = duplicate_test(boilerplate, options)
    print(is_dup)
# False
# False
# True   ← exceeds max_repetitions=2
# True
The deduplication cache is process-wide and persistent for the lifetime of the Python process. Call trafilatura.meta.reset_caches() to clear it between unrelated crawl jobs.

​
deduplication.content_fingerprint()

trafilatura.deduplication.content_fingerprint computes a 64-bit Simhash fingerprint of the meaningful content tokens in a string and returns it as a hexadecimal string. Near-duplicate texts will produce very similar (low Hamming distance) hashes, making this useful for cross-document similarity checks. 
from trafilatura.deduplication import content_fingerprint

fp = content_fingerprint("The quick brown fox jumps over the lazy dog.")
print(fp)  # e.g. "d9e6b3a5f1c04728"
The implementation:
  1. Tokenises the input, strips punctuation, and filters to alphanumeric tokens via sample_tokens().
  2. Computes a BLAKE2b hash for each token.
  3. Accumulates a sign vector (Charikar’s Simhash algorithm) over the token hashes.
  4. Returns the resulting integer as a hex string.

​
Parameters

​
content
str
The text to fingerprint. Punctuation is stripped and only alphanumeric tokens contribute to the hash, so formatting differences do not affect the result.
Returns str — a hexadecimal Simhash fingerprint.

​
Example

from trafilatura.deduplication import content_fingerprint
from trafilatura.deduplication import Simhash

text_a = "Trafilatura is a Python library for web scraping and text extraction."
text_b = "Trafilatura is a Python library for web scraping & text extraction!"

fp_a = content_fingerprint(text_a)
fp_b = content_fingerprint(text_b)

# Use Simhash objects to measure similarity
sim_a = Simhash(existing_hash=fp_a)
sim_b = Simhash(existing_hash=fp_b)

print(f"Hamming distance: {sim_a.hamming_distance(sim_b)}")
print(f"Similarity: {sim_a.similarity(sim_b):.2f}")
# Similarity: 1.00  (near-duplicate)
Store fingerprints as strings and reconstruct Simhash objects with Simhash(existing_hash=fp) when you need to compare them later. This avoids reprocessing the original text.

Build docs developers (and LLMs) love

Get started for freeTalk to us