Trafilatura Tutorials: Corpus Building, NLP, and Web Corpora

Trafilatura’s tutorials cover practical scenarios from discovering and filtering web content to building corpora, generating word frequency lists, and validating structured XML data. Whether you work on the command-line or in Python, these guides walk you through complete end-to-end workflows.

A Jupyter notebook covering core concepts interactively is available in the documentation repository: Trafilatura_Overview.ipynb.

Tutorial overview

Content Discovery & URL Management

Learn how to discover content using sitemaps, RSS/Atom feeds, and web crawling. Filter and manage URL lists with courlan before bulk downloading.

Word Frequency Lists & Text Analysis

Go from a list of URLs to a ranked frequency list. Tokenize with SoMaJo, filter stopwords, and compute n-gram statistics using command-line tools.

TEI XML Validation

Produce and validate TEI-compliant XML files. Use Trafilatura to extract content in the TEI standard and check documents against the TEI schema.

Vector Search & Text Embedding

Use Trafilatura with a vector database (Epsilla) to perform text embedding and semantic search over crawled web content.

DWDS Corpus Replication

Replicate DWDS web corpus data by exporting URLs from the DWDS portal and downloading them with Trafilatura (German-language tutorial).

Tutorial 1: Gathering a custom web corpus

This tutorial covers how to discover, filter, and download web content systematically — from finding URLs to storing extracted text files.

Get your system up and running

Install Trafilatura

See the installation page for full instructions.

pip install -U trafilatura

Understand the command-line interface

All following examples use the command-line interface. If you are new to the CLI, review the introduction on that page first.

Content discovery

Web sources used by Trafilatura can consist of previously known or listed pages. Trafilatura supports three methods to discover content within a website:

Sitemaps — XML files that list all available URLs for a site, intended for search engines
Web feeds — Atom and RSS feeds providing recently updated content
Web crawling — Hopping from page to page following internal links

Sitemaps and feeds are faster and more efficient than crawling. They are machine-readable and give you structured access to a site’s content without exhaustive link traversal. Trafilatura also supports multilingual and multinational sitemaps (e.g. /en/… and /de/…).

Gathering links from the command-line

The --list flag outputs discovered URLs without downloading them — useful for inspecting what you will collect before committing to a full download.

# Discover links from a sitemap (homepage or direct sitemap URL)
trafilatura --sitemap "https://www.sitemaps.org/" --list

# Discover links from an RSS or Atom feed
trafilatura --feed "https://www.dwds.de/" --list

# Discover internal links by crawling
trafilatura --crawl "https://example.org/" --list

For Python-based feed and sitemap usage, see the blog post Using RSS and Atom feeds to collect web pages with Python.

Link filtering

Before downloading at scale, it pays to inspect and filter your URL list to remove unwanted or redundant content.

Filtering with courlan

The courlan package is installed automatically with Trafilatura. It separates text-rich HTML pages from spam, ads, and non-content URLs:

courlan --inputfile raw-linklist.txt --outputfile filtered-linklist.txt

Custom filtering with grep

Use grep for pattern-based inclusion or exclusion:

# Keep only article URLs
grep "/article/" mylist.txt > filtered-list.txt

# Exclude video pages
grep -v "/video/" mylist.txt > filtered-list.txt

Sorting, deduplication, and sampling

# Sort and deduplicate
sort -u myfile.txt > myfile-sorted.txt

# Randomly shuffle
sort -R myfile.txt > myfile-random.txt
shuf myfile.txt > myfile-random.txt

# Draw a random sample of 100 URLs
shuf myfile.txt | head -100 > myfile-random-sample.txt

Trafilatura automatically deduplicates and sorts the input list to optimize the download order, so manual sorting is not required.

Process a list of links

Seamless download and extraction

Two main options control batch processing:

-i / --input-file — path to a file containing one URL per line
-o / --output-dir — directory to write the extracted files into
--backup-dir — (optional) keep a copy of the raw downloaded HTML

# Extract as plain text
trafilatura -i list.txt -o txtfiles/

# Extract as XML
trafilatura --xml -i list.txt -o xmlfiles/

# Extract as XML and keep HTML backups
trafilatura --xml -i list.txt -o xmlfiles/ --backup-dir htmlfiles/

Trafilatura automatically throttles requests to avoid overloading servers, making it the safest default for batch downloads.

Using existing archives

If you already have downloaded HTML files, process them directly:

# Download pages separately with wget
wget --directory-prefix=download/ --wait 5 --input-file=mylist.txt

# Process a directory of archived HTML files
trafilatura --input-dir download/ --output-dir corpus/ --xmltei --no-comments

Tutorial 2: Word frequency lists and text analysis

This tutorial shows how to go from a list of URLs all the way to a ranked word frequency list and n-gram statistics.

Additional requirement: SoMaJo tokenizer

pip install -U SoMaJo

SoMaJo is a high-quality tokenizer for German and English text.

Build frequency lists

Extract text from URLs

Download and extract text from your link list:

trafilatura -i list.txt -o txtfiles

Concatenate and tokenize

Combine all text files and tokenize them:

cat txtfiles/*.txt > txtfiles/all.txt
somajo-tokenizer txtfiles/all.txt > tokens.txt

Sort by frequency

Find the top 10 most frequent tokens:

sort tokens.txt | uniq -c | sort -nrk1 | head -10

Filter and clean tokens

Remove punctuation, empty lines, and normalize to lowercase:

< tokens.txt sed -e "s/[[:punct:]]//g" -e "/^$/d" -e "s/.*/\L\0/" > tokens-filtered.txt

# Display the top 20 most frequent clean tokens
< tokens-filtered.txt sort | uniq -c | sort -nrk1 | head -20

Export to CSV

Save frequency data for further analysis:

< tokens.txt sort | uniq -c | sort -nrk1 | sed -e "s|^ *||g" -e "s| |\t|" > txtfiles/frequencies.csv

N-gram lists

Bigrams and trigrams reveal how words co-occur in a corpus.

# Word bigrams (most frequent pairs)
< tokens-filtered.txt tr "\n" " " | awk '{for (i=1; i<NF; i++) print $i, $(i+1)}' | sort | uniq -c | sort -nrk1 | head -20

# Word trigrams (most frequent triples)
< tokens-filtered.txt tr "\n" " " | awk '{for (i=1; i<NF; i++) print $i, $(i+1), $(i+2)}' | sort | uniq -c | sort -nrk1 | head -20

Working with XML output

If you extracted XML files instead of plain text, SoMaJo can process them directly:

# Tokenize an XML file
somajo-tokenizer --xml xmlfiles/filename.xml

# Remove XML tags after tokenization
somajo-tokenizer --xml xmlfiles/filename.xml | sed -e "s|</*.*>||g" -e "/^$/d"

Tutorial 3: TEI XML validation

Trafilatura can produce and validate TEI-compliant XML files — an important standard for encoding cultural heritage and linguistic data.

Producing TEI files

from trafilatura import fetch_url, extract

downloaded = fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')

# Extract and validate against the TEI schema
result = extract(downloaded, output_format='xmltei', tei_validation=True)

Validating existing TEI files

from lxml import etree
from trafilatura.xml import validate_tei

mytree = etree.parse('document-name.xml')

# Returns True if valid, or an error message describing the first failure
validate_tei(mytree)

For a detailed walkthrough, see the blog post: Validating TEI-XML documents with Python.

Tutorial 4: Vector search and text embedding

This tutorial shows how to use Trafilatura together with Epsilla, an open-source vector database, to perform text embedding and semantic search over crawled web content. The same approach applies to alternative vector databases such as Qdrant, Redis, and ChromaDB.

A hands-on version of this tutorial is available as a Colab Notebook.

Why perform text embedding with crawled data?

Text embedding converts text into numerical vectors commonly used for:

Search — rank results by a query string
Clustering — group text strings by similarity
Anomaly detection — identify outliers

Setup

Start an Epsilla server locally with Docker:

docker pull epsilla/vectordb
docker run --pull=always -d -p 8888:8888 epsilla/vectordb

Install the required packages:

pip install -U pyepsilla langchain sentence_transformers

Connect and create a table

from pyepsilla import vectordb

client = vectordb.Client(host='localhost', port='8888')
client.load_db(db_name="TrafilaturaDB", db_path="/tmp/trafilatura_store")
client.use_db(db_name="TrafilaturaDB")

client.drop_table('Trafilatura')
client.create_table(
    table_name="Trafilatura",
    table_fields=[
        {"name": "ID", "dataType": "INT"},
        {"name": "Doc", "dataType": "STRING"},
        {"name": "Embedding", "dataType": "VECTOR_FLOAT", "dimensions": 384}
    ]
)

Crawl pages and store embeddings

from trafilatura import fetch_url, extract
from langchain.embeddings import HuggingFaceBgeEmbeddings

model_name = "BAAI/bge-small-en"
hf = HuggingFaceBgeEmbeddings(
    model_name=model_name,
    model_kwargs={'device': 'cpu'},
    encode_kwargs={'normalize_embeddings': False}
)

urls = [
    'https://www.tensorflow.org/',
    'https://pytorch.org/',
    'https://react.dev/',
]
results = [extract(fetch_url(url)) for url in urls]

embeddings = [hf.embed_query(result) for result in results]
records = [
    {"ID": idx, "Doc": results[idx], "Embedding": embeddings[idx]}
    for idx in range(len(results))
]
client.insert(table_name="Trafilatura", records=records)

Perform vector search

query = "A modern frontend library"
query_embedding = hf.embed_query(query)
status_code, response = client.query(
    table_name="Trafilatura",
    query_field="Embedding",
    query_vector=query_embedding,
    limit=1
)
print(response)

Tutorial 5: DWDS corpus replication

This tutorial (in German) explains how to replicate data from the DWDS (Digitales Wörterbuch der deutschen Sprache) web corpus using Trafilatura. It covers exporting URL lists from the DWDS portal and downloading the corresponding pages for independent storage and analysis.

The full tutorial is available in German in the Trafilatura documentation: tutorial-dwds.

The general workflow is:

Search the DWDS corpus portal and export URLs as a TSV file
Extract the URL column from the export
Use trafilatura -i urls.txt -o corpus/ to download and extract the pages

Blog posts

View all related blog posts

Video tutorials

A YouTube playlist covering web scraping how-tos and tutorials in several languages is available: Web scraping how-tos and tutorials →

External resources

GLAM-Workbench — cultural heritage and web archives
- Harvesting collections of text from archived web pages
- Compare two versions of an archived web page
User Ethics & Legal Concerns — Melanie Walsh’s introduction to ethical web data collection
Download von Web-Daten and Daten aufbereiten und verwalten — Tutorials in German by Noah Bubenhofer

Get Started

Usage

Web Discovery

Advanced

Resources

Documentation Index

​Tutorial overview

Content Discovery & URL Management

Word Frequency Lists & Text Analysis

TEI XML Validation

Vector Search & Text Embedding

DWDS Corpus Replication

​Tutorial 1: Gathering a custom web corpus

​Get your system up and running

​Content discovery

​Gathering links from the command-line

​Link filtering

​Filtering with courlan

​Custom filtering with grep

​Sorting, deduplication, and sampling

​Process a list of links

​Seamless download and extraction

​Using existing archives

​Tutorial 2: Word frequency lists and text analysis

​Additional requirement: SoMaJo tokenizer

​Build frequency lists

​N-gram lists

​Working with XML output

​Further reading on text analysis

​Tutorial 3: TEI XML validation

​Producing TEI files

​Validating existing TEI files

​Tutorial 4: Vector search and text embedding

​Why perform text embedding with crawled data?

​Setup

​Connect and create a table

​Crawl pages and store embeddings

​Perform vector search

​Tutorial 5: DWDS corpus replication

​Blog posts

​Video tutorials

​External resources

Build docs developers (and LLMs) love

Tutorial overview

Tutorial 1: Gathering a custom web corpus

Get your system up and running

Content discovery

Gathering links from the command-line

Link filtering

Filtering with courlan

Custom filtering with grep

Sorting, deduplication, and sampling

Process a list of links

Seamless download and extraction

Using existing archives

Tutorial 2: Word frequency lists and text analysis

Additional requirement: SoMaJo tokenizer

Build frequency lists

N-gram lists

Working with XML output

Further reading on text analysis

Tutorial 3: TEI XML validation

Producing TEI files

Validating existing TEI files

Tutorial 4: Vector search and text embedding

Why perform text embedding with crawled data?

Setup

Connect and create a table

Crawl pages and store embeddings

Perform vector search

Tutorial 5: DWDS corpus replication

Blog posts

Video tutorials

External resources