Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

A focused crawler — also known as a web spider — starts from one or more seed URLs, downloads each page, extracts the internal hyperlinks it finds, and adds those links to a queue for further exploration. Trafilatura’s crawler is designed for intra-domain use: it stays within the target website, prioritising navigation pages (archives, category listings, pagination) to maximise link discovery in as few requests as possible.

​
Intra-Domain vs. Inter-Domain Crawling

Intra-domain (focused)

Crawls within a single website. Manageable in scope, easy to tune, and the recommended starting point for building a text corpus from a known source.

Inter-domain (broad)

Hops across multiple websites. Requires much more careful deduplication, politeness logic, and RAM for URL tracking. Better served by dedicated crawl infrastructure.
In practice, intra-domain crawling paired with carefully curated seed lists is the most effective approach for linguistic or content-mining use cases.

​
How the Crawler Works

Crawling starts with a seed URL. Trafilatura fetches the page, extracts internal links, and classifies them as either navigation pages (e.g. /category/news/, /page/2/) or content pages (individual articles). Navigation pages are prioritised in the queue so that the crawler reaches as many content pages as possible with the fewest iterations. Visited URLs are marked in an internal UrlStore to ensure each page is fetched at most once. Crawl-delay directives from robots.txt are honoured automatically.

​
Focused Crawler in Python

​
Basic Usage

Import focused_crawler from trafilatura.spider and call it with the homepage URL: 
from trafilatura.spider import focused_crawler

# first pass: visit the homepage and collect up to 10 pages
to_visit, known_links = focused_crawler("https://www.example.org", max_seen_urls=10)
The function returns two values:
  • to_visit — list of URLs queued for future visits (the crawl frontier)
  • known_links — list of all URLs discovered so far (visited and queued)

​
Parameters

ParameterTypeDefaultDescription
homepagestr—Starting URL for the crawl
max_seen_urlsint10Stop after visiting this many pages
max_known_urlsint100000Stop if the total known URL count exceeds this
todolist[str]NonePreviously generated crawl frontier to resume from
known_linkslist[str]NonePreviously known URLs (prevents revisiting)
langstrNoneTwo-letter ISO 639-1 code — bias link selection toward this language
configConfigParserdefault configCustom Trafilatura configuration
rulesRobotFileParserauto-fetchedPre-loaded robots.txt rules
prune_xpathstrNoneXPath expression to remove unwanted elements from pages before link extraction

​
Step-by-Step Iteration

Because the crawler returns its state after each run, you can save progress between sessions and resume later. Pass the returned to_visit and known_links back into the next call: 
from trafilatura.spider import focused_crawler

# first run
to_visit, known_links = focused_crawler(
    "https://www.example.org",
    max_seen_urls=10,
    max_known_urls=100000,
)

# inspect or save the results here ...

# second run – picks up where the first left off
to_visit, known_links = focused_crawler(
    "https://www.example.org",
    max_seen_urls=10,
    max_known_urls=100000,
    todo=to_visit,
    known_links=known_links,
)
Keeping max_seen_urls small (e.g. 10–50) and iterating in a loop gives you natural checkpoints for saving intermediate results or inspecting link quality before committing to a full crawl.

​
Checking Navigation Progress

Use is_still_navigation() to test whether there are still navigation pages (archive pages, category indexes, etc.) in the queue. When it returns False, the crawler has likely exhausted the site’s link structure and remaining URLs are mostly content pages: 
from trafilatura.spider import is_still_navigation

if is_still_navigation(to_visit):
    print("Navigation pages remain — keep crawling")
else:
    print("No more navigation pages — switch to content extraction")

​
Full Iteration Pattern

The following pattern runs the crawler in a loop until the queue is empty or a size limit is reached: 
from trafilatura.spider import focused_crawler, is_still_navigation

homepage = "https://www.example.org"
to_visit, known_links = focused_crawler(homepage, max_seen_urls=10)

while to_visit:
    # keep crawling while navigation pages exist
    if not is_still_navigation(to_visit):
        break

    to_visit, known_links = focused_crawler(
        homepage,
        max_seen_urls=10,
        max_known_urls=100000,
        todo=to_visit,
        known_links=known_links,
    )

print(f"Discovered {len(known_links)} URLs")

​
CLI Crawling

Trafilatura exposes three discovery modes on the command line:
Crawl a website starting from the given URL, stopping at a maximum of 30 page visits or when all pages have been seen:
# crawl and print discovered URLs
trafilatura --crawl "https://www.example.org" > links.txt

# crawl multiple sites from a file
trafilatura --crawl -i sites.txt > links.txt
The --list flag does not apply to --crawl or --explore. Instead, URLs are returned as a plain list so you can inspect and filter them before committing to a bulk download run.

​
Parallel Crawling

Pass a file of seed URLs with -i/--input-file to crawl multiple websites concurrently: 
# crawl all sites listed in sites.txt in parallel
trafilatura --crawl -i sites.txt > all-links.txt

# explore (sitemap + crawl) a list of sites
trafilatura --explore -i sites.txt -o extracted/

​
Politeness Rules

Trafilatura’s crawler enforces politeness automatically:
  • robots.txt compliance — the crawler fetches and parses robots.txt for each domain it visits, respecting Disallow directives and honoring Crawl-Delay values
  • Automatic sleep — if the website’s robots.txt specifies a Crawl-Delay, that value is used; otherwise the SLEEP_TIME setting in your configuration file applies (default: 5 seconds)
You can pass pre-loaded robots.txt rules directly to focused_crawler() to avoid re-fetching them on every call: 
import urllib.robotparser
from trafilatura.spider import focused_crawler

rules = urllib.robotparser.RobotFileParser()
rules.set_url("https://www.example.org/robots.txt")
rules.read()

to_visit, known_links = focused_crawler(
    "https://www.example.org",
    rules=rules,
)

​
Deduplication

The crawler performs deduplication at two levels:
  1. URL-level — the internal UrlStore tracks every URL that has been seen, preventing the same page from being fetched twice across multiple iterations
  2. Content-level — when text extraction is combined with crawling, near-duplicate pages can be filtered out using Trafilatura’s deduplication module

​
Performance Considerations

The main resource constraints for large crawls are:

Time

Crawl delay between requests to the same host is the primary speed bottleneck. Crawling many different domains in parallel amortises this wait.

Bandwidth

Concurrent downloads across many domains can saturate a connection. Use threads and sleep_time to tune throughput.

RAM

Each known URL occupies memory. Above millions of URLs, RAM becomes the limiting factor. Keep max_known_urls within a reasonable bound or use external storage for very large crawls.
Setting both max_seen_urls and max_known_urls to very high values can result in a significant increase in processing time and memory use. Start small, inspect results, and scale up incrementally.

​
Adjusting Crawl Settings

The SLEEP_TIME default and other download parameters can be changed in a settings.cfg file: 
[DEFAULT]

# seconds to wait between requests to the same domain
SLEEP_TIME = 5.0

# download timeout in seconds
DOWNLOAD_TIMEOUT = 30
Pass a custom configuration to focused_crawler(): 
from configparser import ConfigParser
from trafilatura.spider import focused_crawler

config = ConfigParser()
config.read("my_settings.cfg")

to_visit, known_links = focused_crawler(
    "https://www.example.org",
    config=config,
)
See the Settings guide for full documentation of available options.

Build docs developers (and LLMs) love

Get started for freeTalk to us