Skip to main content
Phisherman requires Redis for several critical subsystems. Without a working Redis connection, the server will not be able to serve feed data, cache scan results, or enforce rate limits.

​
Why Redis is required

Redis is used for:
  • Feed data storage — URLHaus, OpenPhish, PhishTank, and PhishStats feeds are loaded into Redis Sets on startup and refreshed on a schedule.
  • Scan result cache — Completed scan results are stored in a Redis Hash (keyed by SHA-256 of the URL) with a 5-minute TTL to avoid re-running all checkers for repeat requests.
  • WHOIS cache — WHOIS lookup results are cached in a Redis Hash for 24 hours to avoid repeated external lookups.
  • DNS cache — DNS resolution results are cached in a Redis Hash for 1 hour.
  • Google Safe Browsing cache — API responses are cached in a Redis Hash for 1 hour (15 minutes for error responses).
  • Rate limiting — Each client IP has a Redis String key that tracks request count within a 15-minute window.

​
Provisioning Upstash Redis

Phisherman uses the @upstash/redis client, which communicates over HTTP REST — no persistent TCP connection is needed.
1

Create an Upstash account

Go to upstash.com and sign up for a free account.
2

Create a Redis database

In the Upstash console, click Create Database. Choose a region close to where you will deploy Phisherman to minimize round-trip latency. Select Regional (not Global) unless you are running a multi-region deployment.
3

Copy your credentials

After the database is created, open the REST API tab in the Upstash dashboard. Copy the REST URL and REST Token values shown there.
4

Set environment variables

Add the credentials to your .env file:
UPSTASH_REDIS_REST_URL=https://<your-instance>.upstash.io
UPSTASH_REDIS_REST_TOKEN=<your-token>
Phisherman initialises the Redis client in src/utils/redis.ts using these two variables:
const redis = new Redis({
  url: process.env.UPSTASH_REDIS_REST_URL!,
  token: process.env.UPSTASH_REDIS_REST_TOKEN!,
});

​
Redis key inventory

The table below lists every key Phisherman writes to Redis.
KeyTypeDescriptionRefresh / TTL
urlhaus_blacklistSetURLHaus active malware URLsRefreshed every 5 min
openphish_urlsSetOpenPhish full URLsRefreshed every 15 min
openphish_hostsSetOpenPhish hostnamesRefreshed every 15 min
phishtank_urlsSetPhishTank verified phishing URLsRefreshed every 1 hr
phishstats_urlsSetPhishStats URLs (last 20,000 entries)Refreshed every 90 min
phishstats_hostsSetPhishStats hostnamesRefreshed every 90 min
scan_resultsHashCached scan results, SHA-256 keyed5-minute TTL per entry
scan_results_expiryZSETExpiry tracking for scan_resultsCleaned up on each CacheManager cycle
whois_dataHashWHOIS lookup results per registered domain24-hour TTL per entry
whois_expiryZSETExpiry tracking for whois_dataCleaned up on each CacheManager cycle
gsb_cache_hashHashGoogle Safe Browsing results per URL1-hour TTL (15-min TTL for error responses)
gsb_cache_expiryZSETExpiry tracking for gsb_cache_hashCleaned up on each CacheManager cycle
gwr_cache_hashHashGoogle Web Risk results per URL (checker currently disabled)1-hour TTL (15-min TTL for errors)
gwr_cache_expiryZSETExpiry tracking for gwr_cache_hashCleaned up on each CacheManager cycle
dns_cache_hashHashDNS resolution results per hostname1-hour TTL per entry
dns_cache_expiryZSETExpiry tracking for dns_cache_hashCleaned up on each CacheManager cycle
ratelimit:<ip>StringRequest count for a client IP address15-minute window (900-second expiry)
The Upstash free tier provides 10,000 commands per day and 256 MB of storage. This is sufficient for personal projects and low-traffic deployments. For production workloads with high scan volume, monitor your command usage in the Upstash dashboard.
Choose a Redis region that is geographically close to your API server. Every scan involves multiple Redis reads (feed lookups, cache checks, rate limit increment), so even a few hundred milliseconds of cross-region latency will noticeably increase response times.

Build docs developers (and LLMs) love

Get started for freeTalk to us