Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/jbms/finance-dl/llms.txt

Use this file to discover all available pages before exploring further.

finance-dl is driven entirely by a plain Python configuration module that you write once and reuse across every scrape. This page shows you how to create that config file, add scraper configurations for both an OFX-based source (Vanguard) and a Selenium-based source (Amazon), and invoke the CLI to start downloading data to your local filesystem.
finance-dl appends the current working directory to sys.path at startup, so your config module is found automatically as long as you run the finance-dl command (or python -m finance_dl.cli) from the same directory that contains your config file. No PYTHONPATH changes or package installs are needed for the config module itself.
1

Create your config module

Create a new Python file named finance_dl_config.py in a directory of your choice. This file will hold all of your scraper configurations.Start with the common boilerplate — a directory for saved browser profiles (avoids re-entering MFA codes) and a root data directory for downloaded files:
# finance_dl_config.py
import os

# Persistent Chrome profile directory — saves session state between runs.
profile_dir = os.path.join(os.getenv('HOME'), '.cache', 'finance_dl')

# Root directory where all downloaded data will be written.
data_dir = '/path/where/data/will/be/saved'
Replace /path/where/data/will/be/saved with an absolute path on your machine, for example os.path.expanduser('~/finance-data').
2

Add an OFX configuration (Vanguard example)

For institutions that support OFX Direct Connect, define a CONFIG_<name>() function that returns a dictionary with module='finance_dl.ofx' and an ofx_params block.
def CONFIG_vanguard():
    # To find `id`, `org`, and `url` for your institution,
    # search at https://www.ofxhome.com/
    ofx_params = {
        'id': '15103',
        'org': 'Vanguard',
        'url': 'https://vesnc.vanguard.com/us/OfxDirectConnectServlet',
        'username': 'XXXXXX',
        'password': 'XXXXXX',
    }
    return dict(
        module='finance_dl.ofx',
        ofx_params=ofx_params,
        output_directory=os.path.join(data_dir, 'vanguard'),
    )
OFX scrapers connect directly to the institution’s server — no browser or ChromeDriver required. The output_directory will be created automatically on first run, and downloaded OFX files are written there atomically.
3

Add a Selenium configuration (Amazon example)

For institutions without an OFX endpoint, define a CONFIG_<name>() function that returns a dictionary pointing to the appropriate Selenium scraper module. Include a profile_dir to persist browser session state across runs:
def CONFIG_amazon():
    return dict(
        module='finance_dl.amazon',
        credentials={
            'username': 'XXXXXX',
            'password': 'XXXXXX',
        },
        output_directory=os.path.join(data_dir, 'amazon'),
        # profile_dir is optional but recommended — saves cookies between runs.
        profile_dir=os.path.join(profile_dir, 'amazon'),
    )
Selenium scrapers run headless by default. If the first attempt fails (for example, due to an MFA prompt), finance-dl automatically retries with the browser window visible so you can complete the login manually.
4

Run a scraper from the command line

Open a terminal, navigate to the directory containing finance_dl_config.py, and run:
python -m finance_dl.cli --config-module finance_dl_config --config vanguard
Or, if you installed via pip, use the installed entry point:
finance-dl --config-module finance_dl_config --config vanguard
finance-dl will:
  1. Import finance_dl_config and call CONFIG_vanguard()
  2. Load the finance_dl.ofx module
  3. Connect to the Vanguard OFX endpoint and download transactions
  4. Write output files to the output_directory you configured
Output example:
2024-01-15 10:23:01 ofx.py:88 [INFO] Connecting to Vanguard OFX endpoint
2024-01-15 10:23:03 ofx.py:112 [INFO] Downloaded 47 transactions
2024-01-15 10:23:03 ofx.py:145 [INFO] Writing /path/where/data/will/be/saved/vanguard/transactions.ofx
Downloaded files accumulate in the output_directory. Existing files are never overwritten — new data is written atomically alongside them.
5

Understand the CLI flags

The full set of CLI flags available via python -m finance_dl.cli:
FlagShortDescription
--config-modulePython module name defining CONFIG_<name> functions
--config-cName of the configuration to run (calls CONFIG_<name>())
--spec-sJSON configuration specification as a literal string; mutually exclusive with --config
--interactive-iLaunch an IPython shell for manual inspection and debugging
--visibleRun the browser in a visible window (implied by --interactive)
--logLog level: DEBUG, INFO, WARNING, ERROR (default: INFO)
Using --spec instead of --config:If you want to run a one-off scrape without a config file, pass the full configuration as a JSON string:
finance-dl --spec '{"module": "finance_dl.ofx", "ofx_params": {...}, "output_directory": "/tmp/test"}'
--config and --spec are mutually exclusive — you must use one or the other.
6

Debug with interactive mode

When a scraper fails or you want to explore its internals, add the -i flag to drop into an IPython shell after the browser has launched:
python -m finance_dl.cli --config-module finance_dl_config --config amazon -i
Interactive mode:
  • Forces the browser to run visibly (not headless), so you can watch and interact with the Chrome window
  • Opens an IPython shell with the scraper module’s namespace loaded, giving you direct access to all scraper functions and the live browser session
  • Enables %autoreload so edits to your scraper code take effect without restarting
This is the recommended workflow for diagnosing login failures, MFA prompts, or site layout changes.

​
Tips for Credentials Management

Hard-coding usernames and passwords in your config file works but is not recommended for sensitive accounts. Instead, prompt at runtime or pull from a password manager: 
from getpass import getpass

def CONFIG_paypal():
    return dict(
        module='finance_dl.paypal',
        credentials={
            'username': input('PayPal username: '),
            'password': getpass('PayPal password: '),
        },
        output_directory=os.path.join(data_dir, 'paypal'),
    )

​
Next Steps

Introduction

Learn about all 18+ supported data sources and how finance-dl’s two scraping mechanisms work.

Installation

Review ChromeDriver setup and version-matching requirements for Selenium scrapers.

Build docs developers (and LLMs) love

Get started for freeTalk to us