Export, import, and copy data between Silo instances

Silo’s transfer system moves content between instances, environments, and storage drivers. Every export is streaming — the first bytes leave immediately and no copy of your data is staged on disk first. Archives are reproducible tar.gz files ordered by collection and entry id, byte-for-byte identical from identical data. API key hashes are excluded by default so a content export you share with others ships no credentials.

Export

Export writes every project and environment, including empty ones, along with schemas, entries, and media. Use --out for a compressed archive or --dir for an on-disk directory tree.

silo export --dir ./backup                          # on-disk directory tree
silo export --out backup.tar.gz                     # compressed tarball
silo export --out posts.tar.gz --include acme/prod/posts   # one collection

The --dir output and the fs storage driver share the same layout, so an fs-backed instance is already a live export.

Narrowing with —include

--include narrows an export to a project, an environment, or a single collection. Pass it more than once for multiple scopes. Leave it out for the whole instance.

silo export --out site.tar.gz --include site                         # one project
silo export --out prod.tar.gz --include site/prod                    # one environment
silo export --out posts.tar.gz \
  --include site/prod/posts --include site/prod/authors              # two collections

The same syntax applies on the HTTP route, one include parameter per rule:

curl -H "Authorization: Bearer $SILO_KEY" \
  "http://localhost:8090/api/export?include=site/prod/posts&include=blog" \
  -o partial.tar.gz

With no rules, an export requires read permission on everything. With rules, it requires permission only on what the rules name — a key scoped to one project can export that project.

Controlling media with —media

--media (or ?media= on the HTTP route) controls how media library files are handled:

Value	Catalog	Files
`all`	Every asset	Every file in the library
`referenced`	Only assets the moved entries point at	Those files only
`none`	Every asset	No files — catalog only

The default is all for a whole-instance transfer, and referenced once --include narrows the scope. Use none when both instances share the same S3 bucket — the catalog still moves so filenames, folders, and URLs are preserved, but no bytes are transferred.

Exporting API keys

Pass --with-keys to include API key hashes in the archive. Omit it (the default) for a content export you share with others.

Import

Import reads an archive or directory and writes its content into Silo. Every imported entry is validated against the schema it lands under — there is no option to skip validation, and a rejected entry is counted and named in the result without stopping the import.

silo import ./backup --mode merge             # newest updated_at wins
silo import backup.tar.gz --mode replace      # replace per collection
silo import backup.tar.gz --dry-run           # report changes, write nothing

Merge vs replace

merge (default)

Matches on (project, environment, collection, id). Missing entries are inserted. Conflicts are resolved by the newest updated_at, then by the higher rev, then by source instance_id. Pass --prefer local|remote to override the whole rule.

replace

Brings each collection the archive carries to exactly the archive’s state. Every entry in the archive is written. Then every entry the archive does not carry is removed. Collections the archive does not mention are left alone.

Deletions do not merge. Silo keeps no tombstones, so only replace mode reflects a deletion made on another instance.

An import is not atomic. A failure partway leaves earlier writes in place. Treat a failed import as unknown state. A replace that stops partway never leaves a collection empty — it writes first and removes after — so the worst case is extra entries. Run the import again to remove them.

Check an untrusted archive with --dry-run before applying it.

A dry run always reports rejected: 0. It does not write schemas, so it has nothing to compare entries against. Use it to preview adds, updates, and schema conflicts — not to audit validation.

Import result shape

{
  "added": 120,
  "updated": 0,
  "deleted": 0,
  "skipped": 0,
  "rejected": 2,
  "rejections": [
    {
      "project": "acme",
      "env": "prod",
      "collection": "posts",
      "id": "01J...",
      "reason": "validation failed: \"/title\": must be string"
    }
  ]
}

The rejections list holds up to 100 entries. The rejected count is always complete. Read it after every import — a source instance can hold data an older schema accepted, and those rows stop at the door.

Copy between instances

POST /api/copy pulls an export from another running Silo and feeds it through the same importer. The destination uses the source credentials for that one request and never stores them.

curl -X POST http://new-silo:8090/api/copy \
  -H "Authorization: Bearer $DESTINATION_SILO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_url": "http://old-silo:8090",
    "source_api_key": "silo_...",
    "mode": "merge",
    "with_keys": false,
    "dry_run": true,
    "include": ["site/prod"],
    "media": "referenced"
  }'

include and media work the same way as on an export. The destination sends include to the source so the source builds only what you asked for rather than the full instance.

Progress streaming

An import or copy normally says nothing between the request and the answer. Behind a proxy that closes quiet connections, a working transfer can look like a failed one. Send Accept: application/x-ndjson to receive one JSON object per line as work progresses:

curl -X POST "http://new-silo:8090/api/import" \
  -H "Authorization: Bearer $DESTINATION_SILO_KEY" \
  -H "Accept: application/x-ndjson" \
  -H "Content-Type: application/gzip" \
  --data-binary @backup.tar.gz

{"type":"progress","phase":"extract","result":{...}}
{"type":"progress","phase":"entries","result":{"added":200,...}}
{"type":"progress","phase":"media"}
{"type":"result","result":{"added":412,"updated":0,...}}

The stream sends a progress line whenever the importer reports (every 200 entries and at each phase boundary), a bare heartbeat after each second of silence, and a final result or error line. The HTTP status goes out before the work begins — the first byte is the status line, and the outcome is the last line.

Read the last line, not the HTTP status code. The status is sent before work starts, so it is always 200. A failure arrives as an error line:

{"type":"error","status":409,"error":{"code":"conflict","message":"..."}}

A stream that stops with no result line means the connection ended before the transfer finished. The destination is in an unknown state — check it before retrying.

The admin UI’s Data Transfer page uses this stream to show live progress for running imports and copies.

Copying between environments on the same instance

Moving data between two environments on the same instance requires no archive. Promoting dev to staging, or seeding a fresh environment from prod, is a single request:

curl -X POST http://localhost:8090/api/projects/acme/envs/staging/copy \
  -H "Authorization: Bearer $SILO_KEY" \
  -H "Content-Type: application/json" \
  -d '{"from": {"project": "acme", "env": "prod"}, "mode": "merge", "dry_run": true}'

{
  "mode": "merge",
  "dry_run": true,
  "added": 12,
  "updated": 0,
  "deleted": 0,
  "skipped": 3,
  "rejected": 0,
  "rejections": []
}

This route requires no transfer:* claim. It uses the same collection and entry claims as the ordinary CRUD routes:

Direction	Claim required
Source read	`collections:<project>/<env>/:schema:read`, `collections:<project>/<env>/:entries:read`
Destination write	`collections:<project>/<env>/:create`, `collections:<project>/<env>/:schema:update`, `collections:<project>/<env>/:entries:create`, `collections:<project>/<env>/:entries:update`
Replace mode (delete)	`collections:<project>/<env>/:delete`, `collections:<project>/<env>/:entries:delete`

A key scoped to one project can move data between that project’s environments and no others. Copying an environment onto itself is a 400. Media is stored per instance, not per environment, so it is already shared and none is copied. The admin exposes this at Settings > Environment > Data Transfer with a preview-then-apply flow.

HTTP API reference

Action	Endpoint
Export	`GET /api/export`
Import	`POST /api/import`
Copy from another instance	`POST /api/copy`
Copy between environments	`POST /api/projects/{p}/envs/{e}/copy`

Required claims

Operation	Claims required
`transfer:export`	Also needs `collections://:schema:read` and `collections://:entries:read` (or narrowed equivalents)
`transfer:import`	Also needs `entries:create`, `entries:update`, `entries:delete` on the destination scope
`transfer:copy`	Requires both `transfer:export` authority on the source and `transfer:import` authority on the destination

Archive format and on-disk layout

The on-disk layout produced by --dir and the fs storage driver are one and the same format. Every archive and every fs-backed instance carries a format_version — currently "1". Silo refuses to open or import data stamped with a version it does not recognize.

<data-dir>/
  manifest.json                 # format_version, instance_id, last_seq,
                                # selection and media scope if narrowed
  projects/
    acme/
      prod/
        schemas/
          posts.schema.json
        content/
          posts/
            01J8XQ4Z8K9M2P3R5T7V9X1B3D.json
  media/

Each entry file holds the full envelope, pretty-printed with a stable field order so diffs stay small. The same collection name in two environments never collides, on disk or in an archive.

Size limits

Archives arriving over HTTP are checked twice before anything is written:

Setting	Default	What it limits
`[transfer] max_archive_size_mb`	1024 MB	The compressed archive size
`[transfer] max_extracted_size_mb`	4096 MB	The uncompressed size (read from archive headers; every file counts as at least 4 KB)

A request past either limit gets a 413 with code archive_too_large. The message names the setting to raise. Both limits apply to /api/import and /api/copy. A file you name on the command line is not checked. Raise both to copy a large instance. Both are configurable in silo.toml or from Settings > Configuration > Transfers in the admin.

Importing system records

An archive can carry Silo’s own records under _system. A selection cannot name them, so the import checks them after unpacking, against the same claims their own routes require:

Records	Claim needed
`_keys`	`keys:import`
`_media`, `_media_folders`, `_media_folder_moves`	`media:create` (even with `media=none`); `media:delete` too when `replace` would empty them
`_variables` for a project	`create` and `entries:update` on `<project>//`
`_audit`, `_plugins`, `_scope_renames`, any other `_` name	Never imported — the request is a `400`

A missing claim is a 403 that names the records and the claim. The check reads rows, not empty collections, so exporting an empty library loads with content permissions alone. The silo import CLI on the host is trusted and loads all of it.

Cross-driver migration

Export and import speak only the storage interface, so switching from SQLite to the fs driver (or back) is:

silo export --out migration.tar.gz          # from the old instance
silo import migration.tar.gz --mode replace # into the new one

This is also the acceptance test for any third-party storage driver — both drivers must produce identical API results.

Get Started

Core Features

Security & Access

Operations

Integrations

Export, import, and copy data between Silo instances

Export

Narrowing with —include

Controlling media with —media

Exporting API keys

Import

Merge vs replace

merge (default)

replace

Import result shape

Copy between instances

Progress streaming

Copying between environments on the same instance

HTTP API reference

Required claims

Archive format and on-disk layout

Size limits

Importing system records

Cross-driver migration

Build docs developers (and LLMs) love

Get Started

Core Features

Security & Access

Operations

Integrations

Documentation Index

​Export

​Narrowing with —include

​Controlling media with —media

​Exporting API keys

​Import

​Merge vs replace

merge (default)

replace

​Import result shape

​Copy between instances

​Progress streaming

​Copying between environments on the same instance

​HTTP API reference

​Required claims

​Archive format and on-disk layout

​Size limits

​Importing system records

​Cross-driver migration

Build docs developers (and LLMs) love

Export

Narrowing with —include

Controlling media with —media

Exporting API keys

Import

Merge vs replace

Import result shape

Copy between instances

Progress streaming

Copying between environments on the same instance

HTTP API reference

Required claims

Archive format and on-disk layout

Size limits

Importing system records

Cross-driver migration