Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/maravento/blackweb/llms.txt

Use this file to discover all available pages before exploring further.

After downloading BlackWeb, it’s crucial to verify the file’s integrity to ensure it hasn’t been corrupted during download or tampered with.

​
Why Verify Checksums?

Checksum verification ensures:
  • File integrity: The download wasn’t corrupted during transfer
  • Authenticity: The file matches the official release
  • Security: The file hasn’t been tampered with by a third party
BlackWeb is a large file (118.8 MB with 4.7+ million domains), making verification especially important.

​
Quick Verification

Run this one-liner to download, extract, and verify BlackWeb: 
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.txt.sha256
LOCAL=$(sha256sum blackweb.txt | awk '{print $1}'); REMOTE=$(awk '{print $1}' blackweb.txt.sha256); echo "$LOCAL" && echo "$REMOTE" && [ "$LOCAL" = "$REMOTE" ] && echo OK || echo FAIL

​
Step-by-Step Verification

1

Download BlackWeb

Download and extract the BlackWeb archive:
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -
2

Download Checksum File

Download the official SHA256 checksum:
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.txt.sha256
3

Calculate Local Checksum

Compute the SHA256 hash of your downloaded file:
LOCAL=$(sha256sum blackweb.txt | awk '{print $1}')
echo "Local checksum: $LOCAL"
4

Extract Remote Checksum

Extract the official checksum from the downloaded file:
REMOTE=$(awk '{print $1}' blackweb.txt.sha256)
echo "Remote checksum: $REMOTE"
5

Compare Checksums

Compare both checksums:
[ "$LOCAL" = "$REMOTE" ] && echo "✓ OK - File verified" || echo "✗ FAIL - Checksum mismatch"

​
Understanding the Output

​
Successful Verification

47e3f8a9b2c1d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0
47e3f8a9b2c1d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0
OK
The two hashes match, indicating the file is authentic and intact.

​
Failed Verification

47e3f8a9b2c1d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0
12a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3
FAIL
The hashes don’t match. Do not use this file. Re-download BlackWeb.

​
Manual Verification

If you prefer manual verification:

​
1. Generate Local Hash

sha256sum blackweb.txt

​
2. View Official Hash

cat blackweb.txt.sha256

​
3. Compare Visually

Ensure both hashes are identical.

​
Automated Verification Script

Create a reusable verification script:
verify-blackweb.sh
#!/bin/bash

echo "[*] Verifying BlackWeb integrity..."

if [ ! -f "blackweb.txt" ]; then
    echo "[!] Error: blackweb.txt not found"
    exit 1
fi

if [ ! -f "blackweb.txt.sha256" ]; then
    echo "[!] Downloading checksum file..."
    wget -q -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.txt.sha256
fi

LOCAL=$(sha256sum blackweb.txt | awk '{print $1}')
REMOTE=$(awk '{print $1}' blackweb.txt.sha256)

echo "[+] Local checksum:  $LOCAL"
echo "[+] Remote checksum: $REMOTE"

if [ "$LOCAL" = "$REMOTE" ]; then
    echo "[✓] Verification successful - File is authentic"
    exit 0
else
    echo "[✗] Verification failed - Checksum mismatch!"
    echo "[!] Do not use this file. Please re-download."
    exit 1
fi
Make it executable and run: 
chmod +x verify-blackweb.sh
./verify-blackweb.sh

​
Testing Blocked Domains

After verification and deployment, test that blocking is working correctly.

​
Using cURL

Test through your Squid proxy: 
curl -x http://your-proxy:3128 -I http://example-blocked-domain.com
Expected output for blocked domain: 
HTTP/1.1 403 Forbidden
Server: squid
Expected output for allowed domain: 
HTTP/1.1 200 OK

​
Using Squid Logs

Monitor real-time blocking: 
sudo tail -f /var/log/squid/access.log | grep "TCP_DENIED"
You should see entries for denied requests: 
1234567890.123 1 192.168.1.100 TCP_DENIED/403 3456 GET http://blocked-domain.com/ - HIER_NONE/- text/html

​
Check Specific Domain

Verify if a domain is in the blocklist: 
grep -i "example.com" /etc/acl/blackweb.txt
If found, it will be blocked when accessed through Squid.

​
Troubleshooting

Possible causes:
  1. Corrupted download: Re-download the file
  2. Wrong file version: Ensure you downloaded both files from the same release
  3. Modified file: If you edited blackweb.txt, checksums will not match
Solution:
rm blackweb.txt blackweb.tar.gz blackweb.txt.sha256
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.tar.gz && cat blackweb.tar.gz* | tar xzf -
wget -q -c -N https://raw.githubusercontent.com/maravento/blackweb/master/blackweb.txt.sha256
Install the necessary package:Ubuntu/Debian:
sudo apt-get install coreutils
CentOS/RHEL:
sudo yum install coreutils
macOS:
# Use shasum instead
shasum -a 256 blackweb.txt
Verification steps:
  1. Confirm domain is in blocklist: 
    grep -i "suspicious-domain.com" /etc/acl/blackweb.txt
  2. Check Squid configuration: 
    squid -k parse
  3. Ensure Squid was reloaded: 
    sudo systemctl reload squid
  4. Verify client is using the proxy: 
    curl -x http://proxy:3128 http://suspicious-domain.com
  5. Check access logs: 
    sudo tail -f /var/log/squid/access.log

​
Best Practices

Always verify checksums when:
  • Performing initial installation
  • Updating to a new BlackWeb release
  • Files are downloaded over untrusted networks
  • Implementing in production environments
If checksum verification fails, never use the file. A mismatched checksum could indicate:
  • Download corruption
  • Network tampering
  • Man-in-the-middle attack
Always re-download from the official source.

​
Next Steps

After successful verification:
  1. Deploy to production: Move the verified file to /etc/acl/blackweb.txt
  2. Configure Squid: Follow the Squid Configuration guide
  3. Set up automation: Consider automating updates with checksum verification
  4. Monitor effectiveness: Review Squid logs regularly to ensure proper blocking

Build docs developers (and LLMs) love

Get started for freeTalk to us