Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/DedalusProject/dedalus_conda/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

The Dedalus conda build scripts support linking to custom MPI, FFTW, and HDF5 libraries instead of using conda-forge versions. This is essential for:
  • High-performance computing (HPC) clusters with vendor-optimized MPI implementations
  • Systems requiring specific MPI flavors (Intel MPI, Cray MPICH, etc.)
  • Environments with pre-existing optimized library installations
  • Clusters where conda-installed MPI may conflict with system configurations
Custom library configurations are more complex and should only be used when the conda-forge build is insufficient for your needs. For laptops and workstations, use the standard conda-forge installation.

​
When to Use Custom Libraries

​
Use Custom Libraries When:

  • Your HPC cluster requires a specific MPI implementation (e.g., Intel MPI, Cray MPICH)
  • You need vendor-optimized libraries for maximum performance
  • System administrators provide pre-built libraries tuned for your hardware
  • The conda-installed MPI conflicts with your cluster’s job scheduler

​
Use Conda Libraries When:

  • Installing on a laptop or workstation
  • You have full control over the system
  • Simplicity and reproducibility are priorities
  • You don’t have access to system-specific optimized libraries

​
Configuration Options

Both conda_install_dedalus2.sh and conda_install_dedalus3.sh provide three main configuration flags at the top of the script: 
# Install OpenMPI from conda, otherwise MPI_PATH must be set
INSTALL_MPI=1
#export MPI_PATH=/path/to/custom/mpi

# Install FFTW from conda, otherwise FFTW_PATH must be set
INSTALL_FFTW=1
#export FFTW_PATH=/path/to/custom/fftw

# Install HDF5 from conda, otherwise HDF5_DIR must be set
INSTALL_HDF5=1
#export HDF5_DIR=/path/to/custom/hdf5
#export HDF5_MPI="ON"

​
Using Custom MPI

1
Locate Your MPI Installation
2
Find the prefix directory of your system MPI installation:
3
which mpicc
# Example output: /usr/local/mpi/bin/mpicc
# MPI_PATH would be: /usr/local/mpi
4
Common MPI locations on HPC systems:
5
  • /opt/intel/mpi
  • /usr/local/mpi
  • /sw/mpi
  • Loaded via environment modules: module show openmpi
    • 6
    Configure the Build Script
    7
    Edit the script before running:
    8
    INSTALL_MPI=0
export MPI_PATH=/usr/local/mpi  # Your MPI prefix
    9
    What Happens
    10
  • The script will NOT install OpenMPI from conda
  • mpi4py will be built from source using pip
  • The script adds ${MPI_PATH}/bin to PATH to ensure mpicc is available
  • Dedalus will be compiled with CC=mpicc to link against your custom MPI
    • FTTW from conda is compiled with OpenMPI support. If you use a custom non-OpenMPI implementation, you should also use custom FFTW to avoid compatibility issues.

    ​
    Using Custom FFTW

    1
    Locate Your FFTW Installation
    2
    Find your FFTW prefix directory:
    3
    find /usr/local -name "libfftw3*.so" 2>/dev/null
# Example: /usr/local/fftw/lib/libfftw3.so
# FFTW_PATH would be: /usr/local/fftw
    4
    Configure the Build Script
    5
    INSTALL_FFTW=0
export FFTW_PATH=/usr/local/fftw
    6
    FFTW Linking Behavior
    7
    The script automatically adjusts FFTW linking based on your BLAS choice:
    8
  • OpenBLAS: Dynamic linking (FFTW_STATIC=0)
  • MKL: Static linking (FFTW_STATIC=1) to avoid symbol conflicts with MKL
    • Ensure your custom FFTW is compiled with the same MPI implementation you’re using. MPI-enabled FFTW built with OpenMPI will not work with Intel MPI.

    ​
    Using Custom HDF5

    1
    Locate Your HDF5 Installation
    2
    Find your HDF5 prefix and determine if it has parallel support:
    3
    which h5pcc  # If this exists, you have parallel HDF5
# Find prefix:
/path/to/h5pcc -show | grep -o '\-L[^ ]*' | head -1
    4
    Configure the Build Script
    5
    For parallel HDF5 (recommended for HPC):
    6
    INSTALL_HDF5=0
export HDF5_DIR=/usr/local/hdf5-parallel
export HDF5_MPI="ON"
    7
    For serial HDF5:
    8
    INSTALL_HDF5=0
export HDF5_DIR=/usr/local/hdf5
# Do not set HDF5_MPI or set to "OFF"
    9
    What Happens
    10
  • The script will NOT install HDF5 from conda
  • h5py will be built from source using pip with --no-binary=h5py
  • If HDF5_MPI="ON", h5py is built with CC=mpicc for parallel support
  • h5py will link against your custom HDF5 library
    • Parallel HDF5 must be built with the same MPI implementation you’re using for Dedalus. If your HDF5 was built with Intel MPI, Dedalus must also use Intel MPI.

    ​
    Complete Custom Build Example

    Here’s a complete example for an HPC cluster using Intel MPI and vendor-provided libraries: 
    # Load required modules (example)
module load intel-mpi/2021.5
module load fftw/3.3.10-intel-mpi
module load hdf5/1.12.1-intel-mpi

# Get library paths from modules
export MPI_PATH=$I_MPI_ROOT
export FFTW_PATH=/sw/fftw/3.3.10-intel-mpi
export HDF5_DIR=/sw/hdf5/1.12.1-intel-mpi
export HDF5_MPI="ON"

# Edit conda_install_dedalus3.sh:
INSTALL_MPI=0
INSTALL_FFTW=0
INSTALL_HDF5=0

# Run the installation
bash conda_install_dedalus3.sh

    ​
    Validation

    After installation, verify your custom libraries are properly linked: 
    conda activate dedalus3
python3 -c "import mpi4py; print(mpi4py.get_config())"
# Should show your custom MPI paths

    ​
    Platform-Specific Considerations

    • Always use environment modules to load consistent library versions
    • Check with system administrators for recommended MPI implementations
    • Many clusters use Cray MPICH, Intel MPI, or MVAPICH2
    • Ensure your libraries are compiled for the correct CPU architecture (e.g., Haswell, Skylake)
    • Custom MPI builds on macOS can be complex due to Accelerate framework
    • Conda-forge builds are generally recommended for macOS
    • If using custom libraries, ensure they’re compiled for the same architecture (x86_64 or arm64)
    • Note: FFTW from conda with custom MPI only works well with OpenMPI-compatible MPIs
    • The script defaults to x86_64 builds via Rosetta (APPLE_SILICON_BUILD_ARM=0)
    • Native arm64 builds may have issues with certain library versions
    • Custom libraries on Apple Silicon require careful architecture matching
    • Consider using conda-forge builds unless you have specific arm64-optimized libraries

    ​
    Troubleshooting Custom Builds

    Error:
    ERROR: mpicc: command not found
    Solution:
    • Verify MPI_PATH points to the correct prefix
    • Ensure ${MPI_PATH}/bin/mpicc exists
    • Check that environment modules are loaded before running the script
    Error:
    ld: library not found for -lfftw3
    Solution:
    • Verify library paths are correct
    • Add library paths to LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (macOS): 
      export LD_LIBRARY_PATH=${FFTW_PATH}/lib:${HDF5_DIR}/lib:${MPI_PATH}/lib:$LD_LIBRARY_PATH
    Symptoms:
    • Segmentation faults during parallel runs
    • MPI initialization errors
    Solution:
    • Ensure ALL components (mpi4py, h5py, FFTW, Dedalus) use the same MPI implementation
    • Do not mix OpenMPI and Intel MPI
    • Rebuild all components if you change MPI implementations

    ​
    Next Steps

    Build docs developers (and LLMs) love

    Get started for freeTalk to us