Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/jbms/finance-dl/llms.txt

Use this file to discover all available pages before exploring further.

finance-dl configuration files are plain Python modules, which means the credentials dict you pass to each scraper can be populated at runtime — not just from string literals. This page covers the credential shapes each scraper category expects and three practical approaches for keeping secrets out of your config file.

​
Credential Shapes by Scraper Type

Most Selenium-based scrapers expect exactly two keys in the credentials dict:
KeyUsed by
usernameamazon, comcast, ebmud, google_purchases, healthequity, mint, paypal, pge, schwab, stockplanconnect, usbank, venmo, and others
passwordSame scrapers as username
tokenwaveapps (WaveApps API token)
api_key / api_secretgemini (Gemini crypto exchange REST API)
Always check the individual scraper module’s documentation for the exact keys it requires before filling in your config.
Never hardcode real credentials as string literals in your config file. Configuration files are easy to accidentally commit to version control, share, or back up to cloud storage. Any secret stored as a plain string is one copy-paste away from being compromised.

​
Approach 1 — Interactive Prompt

The simplest safe approach is to call input() for the username and getpass() for the password. Python will pause and ask for each value at the terminal when the configuration function is evaluated, so the secrets live only in memory and are never written to disk. 
from getpass import getpass
import os

def CONFIG_paypal():
    return dict(
        module='finance_dl.paypal',
        credentials={
            'username': input('PayPal username: '),
            'password': getpass('PayPal password: '),
        },
        output_directory=os.path.join(data_dir, 'paypal'),
    )
getpass() suppresses terminal echo so the password is not visible as you type. This approach works well for occasional manual runs but is unsuitable for fully automated (unattended) execution.

​
Approach 2 — Environment Variables

For automated runs, populate credentials from environment variables set outside the config file: 
import os

def CONFIG_mint():
    return dict(
        module='finance_dl.mint',
        credentials={
            'username': os.environ['MINT_USERNAME'],
            'password': os.environ['MINT_PASSWORD'],
        },
        output_directory=os.path.join(data_dir, 'mint'),
        profile_dir=os.path.join(profile_dir, 'mint'),
    )
Set the variables in your shell session, a .env file loaded by a process manager, or a secrets manager that injects environment variables at runtime. The config file itself never contains a secret value.

​
Approach 3 — External Password Manager

Any tool that can write a secret to stdout can feed credentials into your config. Wrap the call in subprocess.check_output() and decode the result: 
import os
import subprocess

def CONFIG_amazon():
    def secret(name):
        return subprocess.check_output(
            ['pass', 'show', f'finance-dl/{name}']
        ).decode().strip()

    return dict(
        module='finance_dl.amazon',
        credentials={
            'username': secret('amazon/username'),
            'password': secret('amazon/password'),
        },
        output_directory=os.path.join(data_dir, 'amazon'),
        profile_dir=os.path.join(profile_dir, 'amazon'),
    )
The example above uses pass, but the same pattern works with gpg, 1password, bitwarden, secret-tool, or any other CLI password manager. The subprocess call happens at the moment the CONFIG_ function is executed, not at import time, so credentials are fetched on demand.

​
Using profile_dir to Reduce MFA Prompts

Set profile_dir to a persistent Chrome user-data directory for any scraper that uses multi-factor authentication. Chrome will save your session cookies and authentication state between runs, so you typically only need to complete the MFA challenge once — the first time you run the scraper with a new profile directory. Subsequent runs reuse the saved session automatically.
def CONFIG_healthequity():
    return dict(
        module='finance_dl.healthequity',
        credentials={
            'username': os.environ['HEALTHEQUITY_USERNAME'],
            'password': os.environ['HEALTHEQUITY_PASSWORD'],
        },
        output_directory=os.path.join(data_dir, 'healthequity', '1234567'),
        # Saves session state so MFA is only required on the first run.
        profile_dir=os.path.join(profile_dir, 'healthequity'),
    )
Give each scraper its own subdirectory under profile_dir so their browser profiles do not interfere with one another.

Build docs developers (and LLMs) love

Get started for freeTalk to us