Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/Amaculus/screaming-frog-api/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

The CLI wrapper functions call the Screaming Frog CLI (ScreamingFrogSEOSpiderCli) as a subprocess. They resolve the CLI path automatically from standard install locations or from the SCREAMINGFROG_CLI environment variable. 
from screamingfrog import start_crawl, export_crawl

# Start a new crawl
start_crawl(
    "https://example.com",
    "./out",
    save_crawl=True,
    export_tabs=["Internal:All", "Response Codes:All"],
)

# Export from an existing .seospider or .dbseospider file
export_crawl(
    "./crawl.seospider",
    "./exports",
    export_tabs=["Internal:All", "Page Titles:Missing"],
)

​
Environment variables

Set SCREAMINGFROG_CLI to the full path of the CLI executable when it is not in a standard install location.
export SCREAMINGFROG_CLI="/opt/screamingfrog/ScreamingFrogSEOSpiderCli"
Standard paths searched automatically:
  • Windows: C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe
  • macOS: /Applications/Screaming Frog SEO Spider.app/Contents/MacOS/ScreamingFrogSEOSpider
  • Linux: /usr/bin/screamingfrogseospider, /usr/local/bin/screamingfrogseospider

​
start_crawl()

Start a new crawl from a URL via the Screaming Frog CLI. 
from screamingfrog import start_crawl

result = start_crawl(
    "https://example.com",
    "./out",
    config="./my.seospiderconfig",
    save_crawl=True,
    export_tabs=["Internal:All", "Response Codes:All"],
    bulk_exports=["All Inlinks"],
)
print(result.returncode)
​
start_url
str
required
The URL to crawl.
​
output_dir
str | Path
required
Directory where exports and crawl files are saved. Created if it does not exist.
​
cli_path
str | None
default:"None"
Path to the CLI executable. Falls back to SCREAMINGFROG_CLI and standard install locations.
​
config
str | Path | None
default:"None"
Path to a .seospiderconfig file to use for this crawl.
​
auth_config
str | Path | None
default:"None"
Path to an auth config file.
​
export_tabs
Sequence[str] | None
default:"None"
Tabs to export (e.g. ["Internal:All", "Page Titles:Missing"]). When None, no tabs are exported unless export_profile is set.
​
bulk_exports
Sequence[str] | None
default:"None"
Bulk exports to run (e.g. ["All Inlinks", "All Outlinks"]).
​
save_reports
Sequence[str] | None
default:"None"
Reports to save.
​
export_format
str
default:"csv"
Export file format passed to --export-format.
​
headless
bool
default:"True"
Run in headless mode (--headless).
​
overwrite
bool
default:"True"
Pass --overwrite to overwrite existing export files.
​
save_crawl
bool
default:"False"
Pass --save-crawl to save the crawl as a .seospider file.
​
timestamped_output
bool
default:"False"
Pass --timestamped-output to suffix output files with a timestamp.
​
task_name
str | None
default:"None"
Task name passed to --task-name.
​
project_name
str | None
default:"None"
Project name passed to --project-name.
​
extra_args
Sequence[str] | None
default:"None"
Additional raw CLI arguments appended to the command.
Returns subprocess.CompletedProcess[str]
Raises RuntimeError if the CLI exits with a non-zero return code.

​
export_crawl()

Export data from an existing crawl file (.seospider or .dbseospider) using --load-crawl. 
from screamingfrog import export_crawl

export_dir = export_crawl(
    "./crawl.seospider",
    "./exports",
    export_tabs=["Internal:All", "Page Titles:Missing"],
    bulk_exports=["All Inlinks"],
)
print(export_dir)
​
load_target
str
required
Path to the crawl file to load (.seospider or .dbseospider).
​
export_dir
str | Path | None
default:"None"
Directory to write exports. A temporary directory is created when None.
​
cli_path
str | None
default:"None"
Path to the CLI executable.
​
export_tabs
Sequence[str] | None
default:"None"
Tabs to export. Defaults to ["Internal:All"].
​
bulk_exports
Sequence[str] | None
default:"None"
Bulk exports to run.
​
save_reports
Sequence[str] | None
default:"None"
Reports to save.
​
export_format
str
default:"csv"
Export file format.
​
headless
bool
default:"True"
Run in headless mode.
​
overwrite
bool
default:"True"
Overwrite existing export files.
​
force
bool
default:"False"
Force re-export even if exported files already exist in export_dir.
​
export_profile
str | None
default:"None"
Named export profile (e.g. "kitchen_sink"). When set, export_tabs and bulk_exports are taken from the profile unless explicitly overridden.
Returns Path — the export directory path.

​
run_cli()

Run the Screaming Frog CLI with arbitrary arguments. 
from screamingfrog.cli import run_cli

result = run_cli(["--version"])
print(result.stdout)
​
args
Sequence[str]
required
CLI arguments. If the first element is not the CLI executable path, it is prepended automatically.
​
cli_path
str | None
default:"None"
Override the CLI executable path.
​
check
bool
default:"True"
Raise RuntimeError on non-zero exit code.
Returns subprocess.CompletedProcess[str]

​
resolve_cli_path()

Resolve the Screaming Frog CLI executable path. Lookup order:
  1. cli_path argument
  2. SCREAMINGFROG_CLI environment variable
  3. Standard install paths for the current platform
  4. PATH (via shutil.which)
from screamingfrog.cli import resolve_cli_path

path = resolve_cli_path()
print(path)
​
cli_path
str | None
default:"None"
Explicit path to the CLI. When provided, this is tried first.
Returns Path
Raises RuntimeError if the CLI cannot be found. Provide cli_path or set SCREAMINGFROG_CLI.

​
resolve_spider_config()

Resolve the spider.config file path used by Screaming Frog. Lookup order:
  1. config_path argument
  2. SCREAMINGFROG_SPIDER_CONFIG environment variable
  3. ~/.ScreamingFrogSEOSpider/spider.config
  4. %APPDATA%/ScreamingFrogSEOSpider/spider.config (Windows)
from screamingfrog.cli import resolve_spider_config

path = resolve_spider_config()
print(path)
​
config_path
str | Path | None
default:"None"
Explicit path to spider.config.
Returns Path

​
ensure_storage_mode()

Context manager that temporarily forces storage.mode to a given value in spider.config. Restores the original content on exit. 
from screamingfrog.cli import ensure_storage_mode

with ensure_storage_mode("DB"):
    # spider.config has storage.mode=DB for this block
    export_crawl("./crawl.seospider", "./exports")

# spider.config is restored to its original state
​
mode
str
default:"DB"
Storage mode value to write (e.g. "DB", "STORE_ON_DISK").
​
config_path
str | Path | None
default:"None"
Explicit path to spider.config. Resolved via resolve_spider_config() when None.
Returns context manager — yields the resolved Path to spider.config.
Used internally by Crawl.from_seospider() when ensure_db_mode=True (the default). You normally do not need to call this directly unless you are managing the CLI yourself.

Build docs developers (and LLMs) love

Get started for freeTalk to us