Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/sandwichfarm/nostr-watch/llms.txt

Use this file to discover all available pages before exploring further.

@nostrwatch/trawler crawls Nostr relays (WebSocket servers that store and forward events) to collect metadata, capabilities, and health data. It uses the nostrawl library as its crawling engine and persists state to a local SQLite database. When paired with relaymon, trawler continuously feeds newly discovered relay URLs into the monitor so that the relay list stays current without manual curation. Configurable batch sizes, concurrency limits, and seed relay sources allow the crawl to be tuned for network conditions. Dependencies are vendored and committed, so the app runs offline after the initial clone.

​
Prerequisites

  • Deno >=1.40

​
Environment variables

All environment variables are optional. They override the corresponding config.yaml values.
VariableRequiredDescriptionExample
TRAWLER_DB_PATHNoOverride the SQLite database file path./data/trawler.db
TRAWLER_DB_WALNoEnable WAL mode for SQLitetrue

​
Installation

@nostrwatch/trawler is a Deno application — no npm install step is required. Clone the monorepo and navigate to the app directory. The vendor/ directory is committed, so Deno dependencies are available offline.
1

Install Deno

Install Deno >=1.40 from deno.land if you have not already.
2

Navigate to the app directory

cd apps/trawler
3

Copy the configuration file

cp config.yaml config.local.yaml
4

Edit config.local.yaml

Set your seed relays, allowed networks, concurrency, and database path.
5

Start crawling

deno task start

​
Quick start

# Copy and edit the configuration file
cp config.yaml config.local.yaml

# Start crawling
deno task start

# Or compile to a standalone binary
deno task compile

​
Available tasks

TaskDescription
deno task startRun the trawler
deno task compileCompile to a standalone binary in dist/
deno task testRun the test suite
deno task force-refreshClear all caches and restart

​
Configuration reference

Trawler is configured via config.yaml. Copy the file and edit as needed. 
logLevel: info

trawler:
  db:
    path: ./trawler.db
    enableWAL: true
  relaysPerBatch: 5
  concurrency: 2
  seed:
    interval: 60000
    sources: ["config"]
    options:
      allowedNetworks: ["clearnet"]
      config:
        - 'wss://relay.nostr.watch'

​
Key configuration options

KeyDefaultDescription
logLevelinfoLogging verbosity: debug, info, warn, error
trawler.db.path./trawler.dbSQLite database file path
trawler.db.enableWALtrueEnable Write-Ahead Logging for better concurrent access
trawler.relaysPerBatch5Number of relays processed per crawl batch
trawler.concurrency2Maximum concurrent relay checks
trawler.seed.interval60000Seed list refresh interval in milliseconds
trawler.seed.sources["config"]Seed sources: config, api, events
trawler.seed.options.allowedNetworks["clearnet"]Network types to crawl: clearnet, tor, i2p
TRAWLER_DB_PATH and TRAWLER_DB_WAL environment variables override the corresponding trawler.db.* config values when set.

​
Connecting trawler to relaymon

When deploying trawler alongside relaymon, configure relaymon’s seed to read from trawler’s database: 
relaymon:
  seed:
    sources:
      - db
    options:
      db:
        path: /path/to/trawler.db
RelayMon opens trawler’s database in read-only mode to avoid SQLite locking conflicts. The Docker Compose stacks handle this path wiring automatically via shared volume mounts.

​
Known limitations

Alpha status: Trawler is in early development. The original README noted incomplete nostrawl library integration; some crawl behaviors may change without notice.
No automated test coverage: The deno task test scaffold exists but test coverage is minimal. Crawl correctness relies on manual verification against known relay sets.

Build docs developers (and LLMs) love

Get started for freeTalk to us