Skip to main content
BOOM’s pipeline consists of three main components that work together to process astronomical alerts. This guide covers how to run each component.

​
Starting the services

Before running BOOM, start the required infrastructure services using Docker Compose: 
docker compose --profile api up -d
This launches:
  • Valkey (in-memory queue)
  • MongoDB (alert storage)
  • Kafka (message streaming)
  • BOOM API server
Verify the containers are running: 
docker ps

​
Building BOOM

Build the Rust binaries with optimizations: 
cargo build --release
Use the --release flag for better performance. Debug builds are significantly slower.

​
Pipeline components

1

Start the Kafka producer (optional)

For testing with archival data, start a Kafka producer to simulate real-time alerts:
cargo run --release --bin kafka_producer <SURVEY> [DATE] [PROGRAMID]

​
Parameters

  • SURVEY: Survey name (ztf, lsst, or decam)
  • DATE: UTC date in YYYYMMDD format (optional, defaults to yesterday)
  • PROGRAMID: Program identifier (ZTF: public, partnership, or caltech)

​
Example: ZTF public alerts

cargo run --release --bin kafka_producer ztf 20240617 public
This downloads and produces ZTF public alerts from June 17, 2024.
Leave the producer running in a separate terminal while you start the consumer and scheduler.

​
Additional options

View all available options:
cargo run --release --bin kafka_producer -- --help
Options include:
  • --limit: Limit the number of alerts produced
  • --server-url: Override the Kafka broker URL (default: localhost:9092)
2

Start the Kafka consumer

The consumer reads alerts from Kafka topics and transfers them to Valkey queues:
cargo run --release --bin kafka_consumer <SURVEY> [DATE] --programids [PROGRAMIDS]

​
Parameters

  • SURVEY: Survey name to consume from
  • DATE: UTC date in YYYYMMDD format (optional, defaults to today)
  • --programids: Comma-separated program IDs (default: public)

​
Example: ZTF consumer

cargo run --release --bin kafka_consumer ztf 20240617 --programids public

​
Advanced options

cargo run --release --bin kafka_consumer ztf 20240617 \
  --config /path/to/config.yaml

​
Consumer options

OptionDescriptionDefault
--configPath to configuration fileconfig.yaml
--processesNumber of parallel Kafka readers1
--max-in-queueMaximum alerts in Valkey queue15000
--clearClear the Valkey queue before startingfalse
--exit-on-eofExit when topic is empty (testing only)false
--topics-overrideOverride default topic namesNone
--instance-idUUID for this consumer instanceAuto-generated
3

Start the scheduler

The scheduler manages worker pools that process alerts:
cargo run --release --bin scheduler <SURVEY> [--config CONFIG_PATH]

​
Parameters

  • SURVEY: Survey to process alerts for (ztf, lsst, or decam)
  • --config: Path to configuration file (default: config.yaml)

​
Example: ZTF scheduler

cargo run --release --bin scheduler ztf
The scheduler spawns three types of workers based on your configuration:
  1. Alert workers: Ingest alerts from Valkey, format to BSON, perform crossmatches, and write to MongoDB
  2. Enrichment workers: Run classification models on alerts and update MongoDB
  3. Filter workers: Execute user-defined filters and send matching alerts to Kafka

​
Scheduler output

You should see messages like:
Processed alert with candid: 2462315415115010042, queueing for classification
received alerts len: 100
heartbeat: workers running alert=3/3 enrichment=3/3 filter=3/3
The number of workers for each type is configured in config.yaml under the workers section.

​
Worker configuration

Edit config.yaml to adjust worker counts:
config.yaml
workers:
  ztf:
    command_interval: 500
    alert:
      n_workers: 3
    enrichment:
      n_workers: 3
    filter:
      n_workers: 3

​
Stopping BOOM

To gracefully shut down BOOM:
  1. Stop the consumer with Ctrl+C
  2. Stop the scheduler with Ctrl+C
The scheduler will attempt to gracefully stop all workers by sending interrupt signals.
Graceful shutdown is still being improved. You may see error handling in the logs during shutdown.
To stop all Docker services: 
docker compose down

​
Clearing Kafka topics

If you need to clear a Kafka topic before restarting the producer: 
docker exec -it broker /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server broker:9092 \
  --delete \
  --topic ztf_YYYYMMDD_programid
Replace YYYYMMDD with the date and programid with your program ID.

​
Next steps

Processing alerts

Learn how alerts flow through the pipeline

Creating filters

Write custom filters to identify interesting alerts

Monitoring

Monitor pipeline performance with Prometheus

Logging

Configure logging and debugging output

Build docs developers (and LLMs) love

Get started for freeTalk to us