Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Web pages frequently repeat content: navigation menus, footers, cookie notices, and syndicated articles all appear across many URLs. Without deduplication, these repeated segments pollute a corpus and waste storage. Trafilatura addresses this at two granularities — individual text segments within a page and whole documents across a crawl — using efficient in-memory data structures that add negligible overhead to the extraction pipeline.
The deduplication functions were consolidated into trafilatura.deduplication in version 1.10.0. They are no longer accessible from the filters or hashing submodules.

​
Two types of deduplication

Segment-level (exact)

Uses a Least Recently Used (LRU) cache to detect and remove text segments that have appeared too many times. Best for boilerplate elements such as navigation, footers, and repeated callouts.

Document-level (near-duplicate)

Uses locality-sensitive hashing (SimHash / Charikar’s hash) to compute a fingerprint for each document. Comparing fingerprints identifies near-duplicate pages even when their text differs slightly.
Duplicate tracking is performed per thread. Each thread or process maintains its own independent cache and does not share state with other workers. This makes deduplication safe for parallel crawls without locking overhead.

​
Segment-level deduplication

Segment deduplication inspects each extracted text paragraph or block element. If the same text string has been seen more than max_repetitions times and its length exceeds min_duplcheck_size, it is flagged as a duplicate and excluded from the output.

​
Enabling in extraction functions

from trafilatura import extract

result = extract(html, deduplicate=True)

​
The duplicate_test() function

For lower-level control, call duplicate_test() directly on any lxml element. The function checks whether the element’s text content has been seen before, updates the LRU cache, and returns True if the segment should be considered a duplicate. 
from lxml.etree import fromstring
from trafilatura.deduplication import duplicate_test
from trafilatura.settings import Extractor

# Configure options
options = Extractor()
options.min_duplcheck_size = 0   # consider even short segments
options.max_repetitions = 0      # zero repetitions allowed

elem = fromstring("<p>Here is text.</p>")

# First time seen — not a duplicate
duplicate_test(elem, options)
# False

# Second time — now flagged as duplicate
duplicate_test(elem, options)
# True
​
element
_Element
required
An lxml _Element whose text content (including all descendant text nodes via itertext()) is checked against the cache.
​
options
Extractor
required
An Extractor instance. The function reads two attributes:
  • min_duplcheck_size — minimum character length for a segment to be evaluated (shorter segments are always kept).
  • max_repetitions — maximum number of times a segment may appear before being flagged.

​
Tuning via settings.cfg

The two deduplication thresholds can be set globally in settings.cfg: 
[DEFAULT]
# Minimum segment length to check (characters)
MIN_DUPLCHECK_SIZE = 100

# Maximum allowed repetitions before flagging
MAX_REPETITIONS = 2
Or override them at runtime using Extractor: 
options = Extractor()
options.min_duplcheck_size = 50   # catch shorter repeated segments
options.max_repetitions = 1       # allow only one repetition

​
Document-level deduplication

Document-level deduplication works on whole texts rather than individual segments. The SimHash algorithm produces a compact fingerprint that reflects the overall content of a document. Two documents with very similar fingerprints are near-duplicates even if a few words or sentences differ.

​
How SimHash works

Trafilatura implements Charikar’s SimHash (also called locality-sensitive hashing). The process:
  1. Tokenizes the input text and removes punctuation.
  2. Computes a 64-bit BLAKE2b hash for each token.
  3. Accumulates a weighted bit-vector across all tokens.
  4. Converts the final vector to an integer hash.
Similarity between two hashes is measured using Hamming distance: the ratio of matching bits to total bits, yielding a value between 0.0 (completely different) and 1.0 (identical).

​
The Simhash class

from trafilatura.deduplication import Simhash

first = Simhash("This is a text.")
second = Simhash("This is a test.")

# Similarity score between 0.0 and 1.0
second.similarity(first)
# 0.84375
Reuse an existing hash value to avoid recomputing it: 
# Reconstruct from a previously stored hash value
first_copy = Simhash(existing_hash=first.hash)
first_copy.similarity(first)
# 1.0
​
inputstring
str
default:""
Text to hash. Pass an empty string when supplying existing_hash.
​
length
int
default:"64"
Number of bits in the hash. Higher values increase accuracy at the cost of storage.
​
existing_hash
str | int | None
default:"None"
A previously computed hash value (integer or hex string) to reuse. When provided, inputstring is ignored.

​
content_fingerprint()

content_fingerprint() is a convenience wrapper that creates a Simhash, calls .to_hex(), and returns the hash as a hexadecimal string — a compact, serializable representation suitable for storing in a database or file: 
from trafilatura.deduplication import content_fingerprint

content_fingerprint("Here is text.")
# 'd2ff47ba297cc254'

​
Deduplicating a corpus with fingerprints

from trafilatura import bare_extraction
from trafilatura.deduplication import Simhash, content_fingerprint

SIMILARITY_THRESHOLD = 0.9

seen_hashes: list[Simhash] = []
unique_docs = []

for html in html_pages:
    doc = bare_extraction(html, with_metadata=True)
    if doc is None:
        continue

    # Generate a fingerprint for this document
    fp = content_fingerprint(doc.text or "")
    current = Simhash(existing_hash=fp)

    # Compare against all previously accepted documents
    is_duplicate = any(
        current.similarity(seen) >= SIMILARITY_THRESHOLD
        for seen in seen_hashes
    )

    if not is_duplicate:
        seen_hashes.append(current)
        unique_docs.append(doc)

print(f"Kept {len(unique_docs)} unique documents")
Store the hex fingerprint string alongside your extracted text (e.g. as a database column or JSON field). When restarting a crawl, reconstruct Simhash(existing_hash=stored_hex) to resume deduplication without re-extracting every page.

​
generate_hash_filename()

When writing extracted content to disk, generate_hash_filename() produces a filename-safe string from the content hash. Files with identical or near-identical content receive the same or very similar names, making it easy to spot duplicates in the file system: 
from trafilatura.cli_utils import generate_hash_filename

generate_hash_filename("This is a text.")
# 'qAgzZnskrcRgeftk'

​
CLI: —deduplicate flag

Activate segment-level deduplication for all documents processed in a CLI session: 
trafilatura --deduplicate -u https://example.org
Or combine with crawling: 
trafilatura --crawl https://example.org --deduplicate --output-dir ./output

​
Configuration reference

[DEFAULT]
# Minimum character length for a segment to be checked
MIN_DUPLCHECK_SIZE = 100

# Maximum times a segment may appear before being removed
MAX_REPETITIONS = 2

Build docs developers (and LLMs) love

Get started for freeTalk to us