File I/O, Downloads, and Integrity Checks in ztfquery

The ztfquery.io module is the backbone of ztfquery’s data-access layer. It translates ZTF filenames into local paths, authenticates with IRSA (the primary ZTF data archive), downloads missing files on demand, and verifies the integrity of everything on disk. All other ztfquery modules that touch files — queries, lightcurves, alerts — ultimately route through the functions defined here.

Storage Configuration

ztfquery stores all downloaded data under a configurable root directory.

LOCALSOURCE defaults to ./Data/ in the current working directory. Set the ZTFDATA environment variable to point to any local path before importing ztfquery:

export ZTFDATA="/data/ztf/"

Users on the CC-IN2P3 cluster have direct access to the raw archive at /sps/ztf/data/, which is exposed as the CCIN2P3_SOURCE constant. No download is needed in that environment.

from ztfquery import io

print(io.LOCALSOURCE)      # e.g. /data/ztf/  (or ./Data/)
print(io.CCIN2P3_SOURCE)   # /sps/ztf/data/

Authentication

ZTF science and calibration images hosted on IRSA require an authenticated session. ztfquery stores credentials (base64-encoded) in ~/.ztfquery.

`set_account`

Set or update stored credentials for any ZTF-related service.

io.set_account(which, username=None, password=None, token=None,
               test=True, force=False)

which

str

required

The service to configure. Supported values: "irsa", "fritz", "marshal", "pharos", "skyvision". Use "fritz" for Fritz/SkyPortal (token-based).

username

str | None

IRSA username. If None, you will be prompted interactively.

password

str | None

Account password. If None, a secure getpass prompt is shown.

test

bool

default:"True"

Validate credentials against the live service before saving. Set False for offline use or when testing without network access.

force

bool

default:"False"

Save credentials even if the validation test fails. Use with caution.

from ztfquery import io

# Interactive: prompts for IRSA username and password
io.set_account("irsa")

# Scripted (not recommended for shared environments)
io.set_account("irsa", username="myuser", password="mypassword")

# Fritz/SkyPortal uses a token instead of username/password
io.set_account("fritz")  # will prompt: "Enter your fritz token:"

Credentials are stored with base64 encoding in ~/.ztfquery. This is obfuscation, not encryption. Do not use this on a shared system where other users can read your home directory.

`get_cookie`

Obtain a session cookie from the IPAC login service.

io.get_cookie(username=None, password=None, session=None, update=False)

If username and password are None, the stored credentials from ~/.ztfquery are used automatically.

from ztfquery import io

cookies = io.get_cookie()  # uses stored credentials

`open_irsa_session`

Open a requests.Session pre-loaded with IRSA authentication cookies.

io.open_irsa_session(auth=None, incl_cookies=True)

from ztfquery import io

session = io.open_irsa_session()
# re-use this session for multiple downloads to avoid repeated logins

`test_irsa_account`

Verify that stored (or supplied) IRSA credentials are accepted by the server.

from ztfquery import io

is_ok = io.test_irsa_account()
print("IRSA login valid:", is_ok)

Resolving and Downloading Files

`get_file`

The primary entry point for obtaining a local file path. If the file is not already cached locally, it is downloaded from IRSA automatically.

io.get_file(
    filename,
    suffix=None,
    session=None,
    downloadit=True,
    check_suffix=True,
    dlfrom="irsa",
    overwrite=False,
    maxnprocess=4,
    exist=True,
    test_file=True,
    squeeze=True,
    show_progress=True,
    client=None,
    wait=None,
    fill_notexist="None",
)

filename

str | list[str]

required

One or more ZTF filenames (science, raw, or calibration). Accepted forms are ZTF basenames such as ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits.

suffix

str | None

Override the product suffix to retrieve an associated file. For example, if filename is a sciimg.fits, passing suffix="mskimg.fits" returns the corresponding mask image path. For raw files, suffix is used as the imgtypecode.

session

requests.Session | None

A pre-authenticated session (from open_irsa_session). If None, a new session is created per call.

downloadit

bool

default:"True"

Whether to download the file if it is not found locally. Set False to only resolve the local path without triggering a download.

exist

bool

default:"True"

If False, return the local paths without checking whether the files exist or triggering any download.

test_file

bool

default:"True"

If True, FITS files that exist locally are tested for corruption; corrupted files are re-downloaded.

dlfrom

str

default:"\"irsa\""

Download source. Only "irsa" and "ccin2p3" are currently supported.

overwrite

bool

default:"False"

Re-download and overwrite the file even if it already exists locally.

maxnprocess

int

default:"4"

Maximum number of parallel download processes when filename is a list.

squeeze

bool

default:"True"

If True and only a single file is requested, return a plain string instead of a one-element list.

show_progress

bool

default:"True"

Display a progress bar during download.

client

dask.distributed.Client | None

A Dask distributed client. When provided, downloads are submitted as Dask futures.

wait

str | float | None

Throttle the download rate when using a Dask client. None auto-computes a wait based on queue length; "None" disables throttling; a float sets an absolute wait time in seconds.

fill_notexist

str

default:"\"None\""

Controls what to return for files that do not exist after the download attempt. "None" (string) leaves the path as-is; "remove" drops missing entries from the returned list; any other string is substituted as a placeholder.

from ztfquery import io

scifile = "ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits"

# Get local path, downloading if necessary
local_path = io.get_file(scifile)
print(local_path)

# Get the associated mask image instead
mask_path = io.get_file(scifile, suffix="mskimg.fits")

# Download multiple files in parallel (up to 8 processes)
filenames = [scifile, scifile.replace("zi", "zg")]
paths = io.get_file(filenames, maxnprocess=8, squeeze=False)

# Only resolve path, no download
path_only = io.get_file(scifile, downloadit=False)

`download_from_filename`

Lower-level function to explicitly download a file by its ZTF filename, bypassing the local-cache check.

io.download_from_filename(filename, session=None, suffix=None,
                           overwrite=False, auth=None, nodl=False,
                           host="irsa", maxnprocess=4, show_progress=True,
                           check_suffix=True, client=None, wait=None)

from ztfquery import io

io.download_from_filename(
    "ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits",
    host="irsa",
    overwrite=True,
)

`bulk_get_file`

Download a large list of files using Dask for parallelism. Returns either delayed objects, futures, or computed results depending on as_dask.

io.bulk_get_file(filenames, client=None, suffix=None, as_dask="delayed", **kwargs)

filenames

list[str]

required

List of ZTF filenames to download.

client

dask.distributed.Client | None

A Dask distributed client for cluster-level parallelism. If None, local Dask threads are used.

as_dask

str

default:"\"delayed\""

Controls the return type:

"delayed" — returns (list_of_delayed, session).
"computed" — blocks and returns the list of local file paths.
"futures" — submits to client and returns (futures, session).
"gathered" — submits to client, waits, and returns the file paths.

from ztfquery import io
from dask.distributed import Client

client = Client(n_workers=4)

filenames = [
    "ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits",
    "ztf_20190918123456_000700_zg_c05_o_q1_sciimg.fits",
]

# Non-blocking: get dask futures back
futures, session = io.bulk_get_file(filenames, client=client, as_dask="futures")

# Blocking: wait for all downloads and get local paths
paths = io.bulk_get_file(filenames, as_dask="computed")

Parsing Filenames

ZTF filenames encode rich metadata — date, field, filter, CCD, quadrant, and product type — in a structured naming convention.

`parse_filename`

Decompose a ZTF filename into its constituent parts.

io.parse_filename(filename, as_serie=True)

filename

str

required

A ZTF basename or full path. Works for science (sciimg), raw, calibration, and reference filenames.

as_serie

bool

default:"True"

If True (default), return a pandas.Series. If False, return a plain dict.

from ztfquery import io

fname = "ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits"
parsed = io.parse_filename(fname)
print(parsed)
# filefracday    20190917468333
# paddedfield             000698
# filtercode                  zi
# ccdid                        3
# imgtypecode                  o
# qid                          2
# suffix               sciimg.fits
# dtype: object

`filename_to_kind`

Determine whether a filename corresponds to a science, raw, calibration, or reference product.

io.filename_to_kind(filename)

from ztfquery import io

kind = io.filename_to_kind("ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits")
print(kind)   # "sci"

kind = io.filename_to_kind("ztf_20190917_000001_zg_c01_q1_refimg.fits")
print(kind)   # "ref"

# Works on lists too
kinds = io.filename_to_kind(["ztf_...sciimg.fits", "ztf_...raw.fits"])

`get_filedataframe`

Parse a list of ZTF filenames into a combined DataFrame that includes parsed metadata and a column indicating whether each file exists locally.

io.get_filedataframe(filenames)

from ztfquery import io

filenames = [
    "ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits",
    "ztf_20190918123456_000700_zg_c05_o_q1_sciimg.fits",
]
df = io.get_filedataframe(filenames)
print(df.columns.tolist())
# ['filefracday', 'paddedfield', 'filtercode', 'ccdid',
#  'imgtypecode', 'qid', 'suffix', 'isfile', 'filename']

# Filter to only the files already on disk
local = df[df["isfile"]]

Raw Data Paths

`filefracday_to_local_rawdata`

Glob for all local raw FITS files matching a given filefracday identifier and optional CCD ID.

io.filefracday_to_local_rawdata(filefracday, ccdid="*")

from ztfquery import io

# Find all raw files for a given fractional-day identifier
raw_files = io.filefracday_to_local_rawdata("20190917468333")

# Restrict to CCD 3
raw_ccd3 = io.filefracday_to_local_rawdata("20190917468333", ccdid=3)

File Integrity Checking

ztfquery can validate every cached file and optionally remove or re-download corrupted entries.

`run_full_filecheck`

Scan all files under LOCALSOURCE (or a custom path) and report corrupted or unreadable files.

io.run_full_filecheck(extension="*", startpath=None, erasebad=True,
                       redownload=False, nprocess=4, show_progress=True)

extension

str

default:"\"*\""

Only check files with this extension (e.g. "fits", "txt"). Leading dots are stripped automatically.

startpath

str | None

Root directory to scan. Defaults to LOCALSOURCE ($ZTFDATA).

erasebad

bool

default:"True"

Automatically delete corrupted files from disk.

redownload

bool

default:"False"

Re-download corrupted files from IRSA after erasing them.

nprocess

int

default:"4"

Number of parallel worker processes for checking.

from ztfquery import io

# Check all FITS files, erase bad ones, and re-download them
bad_files = io.run_full_filecheck(
    extension="fits",
    erasebad=True,
    redownload=True,
    nprocess=8,
)
print(f"Found {len(bad_files)} corrupted files")

`test_files`

Test a specific list of files rather than scanning an entire directory.

io.test_files(filename, erasebad=True, nprocess=1, show_progress=True, redownload=False)

from ztfquery import io

local_paths = [
    "/data/ztf/sci/.../ztf_...sciimg.fits",
    "/data/ztf/sci/.../ztf_...mskimg.fits",
]
bad = io.test_files(local_paths, erasebad=False)
print("Corrupted:", bad)

`get_localfiles`

List all ztfquery-managed files of a given extension under a directory tree.

io.get_localfiles(extension="*", startpath=None)

from ztfquery import io

all_fits = io.get_localfiles(extension="fits")
print(f"{len(all_fits)} FITS files under {io.LOCALSOURCE}")

# Custom path
txt_files = io.get_localfiles(extension="txt", startpath="/tmp/ztf_test/")

`calculate_hash`

Compute the MD5 hash of a file for integrity verification.

io.calculate_hash(fname)

from ztfquery import io

digest = io.calculate_hash("/data/ztf/sci/.../ztf_...sciimg.fits")
print(digest)   # e.g. "d41d8cd98f00b204e9800998ecf8427e"

ztfquery automatically writes .md5 sidecar files alongside downloaded data when write_hash=True is passed to download_single_url. Subsequent calls to _test_file_ skip the full read if a matching hash file already exists, significantly speeding up bulk integrity checks.

Complete Workflow Examples

Set up credentials

from ztfquery import io

# Run once; credentials are saved to ~/.ztfquery
io.set_account("irsa")

Verify your account

assert io.test_irsa_account(), "IRSA login failed — check your credentials"

Download a science image

from ztfquery import io

scifile = "ztf_20190917468333_000698_zi_c03_o_q2_sciimg.fits"
local_path = io.get_file(scifile)
print("Saved to:", local_path)

Parse its metadata

meta = io.parse_filename(scifile)
print(meta)

kind = io.filename_to_kind(scifile)
print("Product type:", kind)  # "sci"

Download the associated mask

mask_path = io.get_file(scifile, suffix="mskimg.fits")
print("Mask at:", mask_path)

Run an integrity check on everything downloaded

bad = io.run_full_filecheck(extension="fits", erasebad=True, redownload=True)
if not bad:
    print("All files OK")

Getting Started

Core Modules

Field & Sky Tools

Examples & Tutorials

File I/O, Downloads, and Integrity Checks in ztfquery

Storage Configuration

Authentication

`set_account`

`get_cookie`

`open_irsa_session`

`test_irsa_account`

Resolving and Downloading Files

`get_file`

`download_from_filename`

`bulk_get_file`

Parsing Filenames

`parse_filename`

`filename_to_kind`

`get_filedataframe`

Raw Data Paths

`filefracday_to_local_rawdata`

File Integrity Checking

`run_full_filecheck`

`test_files`

`get_localfiles`

`calculate_hash`

Complete Workflow Examples

Build docs developers (and LLMs) love

Getting Started

Core Modules

Field & Sky Tools

Examples & Tutorials

Documentation Index

​Storage Configuration

​Authentication

​set_account

​get_cookie

​open_irsa_session

​test_irsa_account

​Resolving and Downloading Files

​get_file

​download_from_filename

​bulk_get_file

​Parsing Filenames

​parse_filename

​filename_to_kind

​get_filedataframe

​Raw Data Paths

​filefracday_to_local_rawdata

​File Integrity Checking

​run_full_filecheck

​test_files

​get_localfiles

​calculate_hash

​Complete Workflow Examples

Build docs developers (and LLMs) love

Storage Configuration

Authentication

`set_account`

`get_cookie`

`open_irsa_session`

`test_irsa_account`

Resolving and Downloading Files

`get_file`

`download_from_filename`

`bulk_get_file`

Parsing Filenames

`parse_filename`

`filename_to_kind`

`get_filedataframe`

Raw Data Paths

`filefracday_to_local_rawdata`

File Integrity Checking

`run_full_filecheck`

`test_files`

`get_localfiles`

`calculate_hash`

Complete Workflow Examples