Documentation Index
Fetch the complete documentation index at: https://mintlify.com/Amaculus/screaming-frog-api/llms.txt
Use this file to discover all available pages before exploring further.
The screamingfrog.cli module provides Python wrappers around the Screaming Frog SEO Spider command-line interface. Use these helpers to start new crawls, export data from existing crawl files, and run arbitrary CLI commands without building argument lists by hand.
Environment variable
Set SCREAMINGFROG_CLI to the full path of the CLI executable when it is not installed in a standard location:
export SCREAMINGFROG_CLI="/opt/screamingfrog/ScreamingFrogSEOSpiderCli"
resolve_cli_path checks the standard install paths for Windows, macOS, and Linux before falling back to PATH.
start_crawl
Crawl a URL and save exports to a directory.
from screamingfrog import start_crawl
start_crawl(
"https://example.com",
"./out",
save_crawl=True,
export_tabs=["Internal:All", "Response Codes:All"],
)
Parameters
| Parameter | Type | Default | Description |
|---|
start_url | str | required | The URL to crawl. |
output_dir | str | Path | required | Directory for exports and crawl output. Created if it does not exist. |
config | str | Path | None | None | Path to a .seospiderconfig file to pass via --config. |
auth_config | str | Path | None | None | Path to an auth config file to pass via --auth-config. |
export_tabs | Sequence[str] | None | None | Tab names to export (e.g. ["Internal:All", "Page Titles:Missing"]). |
bulk_exports | Sequence[str] | None | None | Bulk export names (e.g. ["Links:All Inlinks"]). |
save_reports | Sequence[str] | None | None | Report names to save. |
export_format | str | "csv" | Export file format. |
headless | bool | True | Run the CLI in headless mode. |
overwrite | bool | True | Overwrite existing output files. |
save_crawl | bool | False | Save the crawl as a .seospider file. |
timestamped_output | bool | False | Add a timestamp suffix to the output folder. |
task_name | str | None | None | Task name passed to --task-name. |
project_name | str | None | None | Project name passed to --project-name. |
extra_args | Sequence[str] | None | None | Additional raw CLI arguments appended verbatim. |
cli_path | str | None | None | Override the CLI executable path. |
subprocess.CompletedProcess[str].
export_crawl
Export data from an existing .seospider or .dbseospider crawl file without running a new crawl.
from screamingfrog import export_crawl
export_crawl(
"./crawl.seospider",
"./exports",
export_tabs=["Internal:All", "Page Titles:Missing"],
)
Parameters
| Parameter | Type | Default | Description |
|---|
load_target | str | required | Path to the crawl file to load (--load-crawl). |
export_dir | str | Path | None | None | Destination directory. A temporary directory is used when None. |
export_tabs | Sequence[str] | None | None | Tab names to export. Defaults to ["Internal:All"] when None and no profile is set. |
bulk_exports | Sequence[str] | None | None | Bulk export names to include. |
save_reports | Sequence[str] | None | None | Report names to save. |
export_format | str | "csv" | Export file format. |
export_profile | str | None | None | Named export profile. Use "kitchen_sink" for all tabs and bulk exports. |
headless | bool | True | Run the CLI in headless mode. |
overwrite | bool | True | Overwrite existing output files. |
force | bool | False | Re-export even if internal_all.csv already exists in the destination. |
cli_path | str | None | None | Override the CLI executable path. |
Path.
export_crawl skips the CLI run if internal_all.csv (or equivalent) already exists in export_dir, unless you pass force=True.
run_cli
Low-level passthrough for running arbitrary Screaming Frog CLI commands.
from screamingfrog.cli import run_cli
result = run_cli(["--version"])
print(result.stdout)
args is not the CLI executable, it is prepended automatically.
Parameters
| Parameter | Type | Default | Description |
|---|
args | Sequence[str] | required | CLI arguments. The executable is prepended if not already present. |
cli_path | str | None | None | Override the CLI executable path. |
check | bool | True | Raise RuntimeError on non-zero exit code. |
subprocess.CompletedProcess[str].
resolve_cli_path
Find the Screaming Frog CLI executable. Checked in this order:
- The
cli_path argument, if provided.
- The
SCREAMINGFROG_CLI environment variable.
- Standard install paths for the current platform.
PATH via shutil.which.
from screamingfrog.cli import resolve_cli_path
cli = resolve_cli_path()
print(cli) # e.g. Path('/usr/bin/screamingfrogseospider')
RuntimeError if the executable cannot be found.
Default paths checked per platform:
C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe
C:\Program Files\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe
/Applications/Screaming Frog SEO Spider.app/Contents/MacOS/ScreamingFrogSEOSpider
/usr/bin/screamingfrogseospider
/usr/local/bin/screamingfrogseospider
Kitchen-sink export profile
The "kitchen_sink" export profile contains every tab and bulk export captured from the Screaming Frog UI — 447 tabs and 146 bulk exports. Pass it to export_crawl or Crawl.load to get the broadest possible CSV coverage in one shot.
from screamingfrog import export_crawl
export_crawl(
"./crawl.seospider",
"./exports_kitchen",
export_profile="kitchen_sink",
)
from screamingfrog.config import get_export_profile
profile = get_export_profile("kitchen_sink")
print(len(profile.export_tabs)) # 447
print(len(profile.bulk_exports)) # 146
print(profile.export_tabs[:3])
# ['AI:All', 'AMP:All', 'AMP:Non-200 Response']
get_export_profile returns an ExportProfile dataclass with two fields:
| Field | Type | Description |
|---|
export_tabs | list[str] | All tab names in "Section:Filter" format. |
bulk_exports | list[str] | All bulk export names in "Category:Export" format. |
When loading a .seospider crawl with Crawl.load, pass export_profile="kitchen_sink" together with seospider_backend="csv" to get full GUI-filter parity across all tabs.
ensure_storage_mode context manager
Temporarily forces storage.mode in Screaming Frog’s spider.config for the duration of a with block, then restores the original value.
This is used internally by export_dbseospider_from_seospider and load_seospider_db_project to ensure the CLI writes a DB-mode crawl.
from screamingfrog.cli import ensure_storage_mode
with ensure_storage_mode("DB") as config_path:
# spider.config now has storage.mode=DB
# run CLI commands here ...
pass
# storage.mode is restored to its original value
Parameters
| Parameter | Type | Default | Description |
|---|
mode | str | "DB" | The storage mode to set (e.g. "DB" or "CRAWL"). |
config_path | str | Path | None | None | Path to spider.config. Resolved automatically when None. |
spider.config path. The config file is created if it did not previously exist and removed on exit in that case.
ensure_storage_mode writes to spider.config on disk. It is safe to use inside a with block, but do not call it concurrently from multiple processes pointing at the same config file.