Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/adbar/trafilatura/llms.txt

Use this file to discover all available pages before exploring further.

Beyond the main article text, Trafilatura can extract a rich set of structured metadata from HTML pages. It draws on multiple sources in order of reliability: JSON-LD structured data, OpenGraph tags, standard <meta> elements, HTML title and heading elements, and the canonical URL itself. The result is a Document object whose fields can be serialized to any of Trafilatura’s supported output formats.

​
Extracted metadata fields

title

The page or article title, resolved from <h1>, OpenGraph tags, <title>, or JSON-LD headline.

author

Author name(s), separated by semicolons when multiple authors are detected. Extracted from meta tags, JSON-LD, and author-specific XPaths.

date

Publication date as a string (default format YYYY-MM-DD). Powered by the htmldate library.

url

Canonical URL of the page, resolved from <link rel="canonical">, OpenGraph og:url, or the provided default URL.

hostname

Domain name extracted from the resolved URL.

description

Page description from OpenGraph, Twitter card, or standard description meta tags.

sitename

Name of the publishing site, from og:site_name, JSON-LD publisher, or the page title.

categories

List of article categories, from articleSection in JSON-LD or category links in the HTML.

tags

List of keywords or tags from meta keywords, article:tag, or tag links.

fingerprint

SimHash hex string for near-duplicate detection. See Deduplication.

id

Optional record identifier supplied via the record_id parameter.

license

Detected content license (e.g. CC BY 4.0) parsed from <a rel="license"> or footer links.

language

Detected or target language code (ISO 639-1).

image

URL of the primary image from og:image or Twitter card meta tags.

pagetype

Schema.org page type (e.g. article, blogposting, webpage) from JSON-LD or OpenGraph.

​
Enabling metadata in extract()

By default, extract() returns only the main text. Pass with_metadata=True to include all available metadata fields in the output: 
from trafilatura import extract

html = "<html>...</html>"

# Include metadata in the output
result = extract(html, output_format="json", with_metadata=True)
To skip documents that are missing essential metadata, use only_with_metadata=True. A document is discarded unless it has all three of date, title, and url: 
result = extract(html, with_metadata=True, only_with_metadata=True)
# Returns None if date, title, or url is missing

​
The bare_extraction() Document object

bare_extraction() returns a Document instance (or None) rather than a serialized string. This is the most flexible way to work with metadata in Python because all fields are directly accessible as attributes: 
from trafilatura import bare_extraction

doc = bare_extraction(html, with_metadata=True)

if doc is not None:
    print(doc.title)
    print(doc.author)
    print(doc.date)
    print(doc.url)
    print(doc.hostname)
    print(doc.sitename)
    print(doc.categories)  # list
    print(doc.tags)         # list
    print(doc.description)
    print(doc.license)
    print(doc.language)
    print(doc.fingerprint)

​
Converting to a dictionary

Call .as_dict() to get a plain Python dictionary containing every slot, including body and commentsbody (lxml _Element objects): 
doc_dict = doc.as_dict()
# {'title': '...', 'author': '...', 'date': '...', ...}
The body and commentsbody fields in the dictionary are lxml _Element objects, not strings. Serialize them with lxml.etree.tostring() if you need them as text.

​
Standalone metadata extraction

Use extract_metadata() directly when you only need metadata and no article body: 
from trafilatura.metadata import extract_metadata

doc = extract_metadata(
    html,
    default_url="https://example.org/article",
    extensive=True,
)

print(doc.title)
print(doc.author)
print(doc.date)
​
filecontent
str | HtmlElement
required
The HTML content to parse, either as a string or a pre-parsed lxml element.
​
default_url
str | None
default:"None"
A fallback URL to use when no canonical URL can be found in the HTML. Passing the known URL also helps htmldate find dates embedded in URL paths.
​
date_config
dict | None
default:"None"
A dictionary of parameters forwarded to htmldate. When None, defaults are derived from the EXTENSIVE_DATE_SEARCH config setting.
​
extensive
bool
default:"True"
Enable htmldate’s extensive search mode. Ignored when date_config is provided explicitly.
​
author_blacklist
set[str] | None
default:"None"
A set of author names to suppress from the output.

​
Date extraction

Date extraction is powered by the htmldate library. Trafilatura calls it automatically whenever with_metadata=True (or whenever extract_metadata() is used).

​
Configuration parameters

Pass a dictionary to date_extraction_params in extract() / bare_extraction(), or to the date_config parameter of extract_metadata():
​
extensive_search
bool
default:"True"
Search more aggressively across the full page content for a date. Higher recall, lower precision. Controlled by EXTENSIVE_DATE_SEARCH in settings.cfg when not set explicitly.
​
original_date
bool
default:"True"
Prefer the original publication date over a modification date when both are present.
​
outputformat
str
default:"%Y-%m-%d"
strftime-compatible format string for the returned date. For example "%d/%m/%Y" or "%Y-%m-%dT%H:%M:%S".
​
max_date
str
default:"today"
Upper bound for acceptable dates (format YYYY-MM-DD). Defaults to today’s date to prevent future dates from being accepted. Trafilatura sets this automatically.

​
Example: custom date parameters

from trafilatura import extract

date_params = {
    "extensive_search": False,   # faster, more precise
    "original_date": True,
    "outputformat": "%Y-%m-%d",
    "max_date": "2024-12-31",
}

result = extract(
    html,
    with_metadata=True,
    output_format="json",
    date_extraction_params=date_params,
)

​
Using the Extractor class

from trafilatura import extract
from trafilatura.settings import Extractor, set_date_params

# Start from defaults and override individual keys
my_date_params = set_date_params(extensive=False)
my_date_params["outputformat"] = "%d/%m/%Y"

options = Extractor(
    with_metadata=True,
    output_format="json",
    date_params=my_date_params,
)

result = extract(html, options=options)

​
Passing a URL to improve metadata

Providing the source URL as url= (to extract() / bare_extraction()) or as default_url= (to extract_metadata()) has two benefits:
  1. Hostname extraction: hostname and sitename are resolved from the URL when they cannot be found in the HTML.
  2. Date extraction: htmldate uses the URL to detect dates embedded in paths such as /2024/03/15/article-title/.
from trafilatura import extract

result = extract(
    html,
    url="https://example.org/2024/03/15/my-article/",
    with_metadata=True,
    output_format="json",
)

​
JSON-LD metadata

Trafilatura’s json_metadata.py module handles structured data embedded in <script type="application/ld+json"> blocks. It supports the following Schema.org types among others:
  • Article schemas: Article, BlogPosting, NewsArticle, ScholarlyArticle, SocialMediaPosting, and more.
  • Publisher schemas: Organization, NewsMediaOrganization, WebSite.
  • Page-type schemas: WebPage, ProfilePage, FAQPage, CollectionPage, and others.
The extractor handles nested @graph arrays, liveBlogUpdate lists, and gracefully falls back to regex-based extraction when the JSON is malformed.
JSON-LD data takes priority over OpenGraph and <meta> tags for fields such as author, sitename, categories, and title.

​
Output format differences

How metadata appears in the output depends on the chosen format:
All metadata fields are included as top-level keys in the JSON object alongside the extracted text:
{
  "title": "Example Article",
  "author": "Jane Doe",
  "date": "2024-03-15",
  "url": "https://example.org/article",
  "hostname": "example.org",
  "sitename": "Example Site",
  "categories": ["Technology"],
  "tags": ["python", "scraping"],
  "text": "Article body text..."
}

Build docs developers (and LLMs) love

Get started for freeTalk to us