5. Advanced Users

5.1. Bfrescoxpro

The Bfrescoxpro Python package is designed to be as similar to the Bfrescox package as possible while still allowing users to install the Bfrescoxpro package with a custom build of the package’s Frescox binary. At present, users can specify that the package build its binary with

  • MPI distributed parallelism,

  • OpenMP shared parallelism, or

  • MPI+OpenMP hybrid parallelism.

The package and its distribution scheme have been designed to allow users to build and use custom binaries on machines ranging from laptops to leadership class machines that have MPI implementations provided and optimized by experts for each particular machine.

This customization, however, requires that the Bfrescoxpro package be distributed as a source distribution. Since users are required to install the package and have the package build its Frescox binary from scratch, they are also responsible for setting up the software stack needed to build and use their custom Frescox binary before installing Bfrescoxpro and every time that they use the package’s installation. The extra thought and care required to use this customization is one motivation for providing build customization through a separate package with a different name.

Note that while Bfrescoxpro might be needed to acquire data, the Bfrescox package can be used on any machine to load and analyze data acquired with Bfrescoxpro.

5.1.1. Requirements

Before installing Bfrescoxpro, users must provide a Fortran compiler that supports all requirements for building Frescox. At present, installations require either the use of the

  • GCC compiler suite (gfortran) or

  • an Intel compiler suite (ifort or ifx).

If OpenMP parallelization is desired, ensure that the compiler supports OpenMP compilation. Please consult the Frescox documentation for more information on its build requirements.

Todo

The distribution is encoded with the version of Frescox that will be used. This information is effectively hidden from the users. Therefore, they won’t know what version of the Frescox documentation to refer to. See Issue 30.

If MPI parallelization is desired, then the user must also provide an MPI installation that is compatible with the Fortran compiler.

The package’s build system uses Meson to automatically detect external dependencies, such as the compiler and MPI installation, and to build the binary. Both Meson and its backend ninja are temporarily installed automatically during package installation.

Note

Testing on some systems has revealed that the Bfrescoxpro build system can fail to correctly use properly installed non-Open MPI installations (e.g., MPICH). This is presently under investigation (See Issue 17) and a potential workaround is to insist that MPI be used in the build

$ cd /path/to/Bfrescox/bfrescoxpro_pypkg
$ BFRESCOX_USE_MPI=enabled python -m pip install .

5.1.2. Installation

While we intend for this package to eventually be distributed by PyPI for direct installation via pip, during this alpha development phase, users must install the package directly from a local clone of the Bfrescox repository.

After

  • installing a local clone,

  • checking out the desired commit or tag,

  • and setting up your target Python environment as desired,

execute

$ cd /path/to/Bfrescox/bfrescoxpro_pypkg
$ python -m pip install .

This will build a binary with OpenMP if the compiler that is found by Meson supports OpenMP compilation. In addition, it will build with MPI if an MPI implementation is found by Meson. If both are found, then an MPI+OpenMP binary is built. If neither is found, then please update your software stack or use the Bfrescox package.

Note that one can inspect the external dependencies found by the build system and the progress of the build using -v in the above.

Users can also override customization by setting the BFRESCOX_USE_MPI and BFRESCOX_USE_OMP environment variables to

  • disabled - build will not include the associated feature

  • enabled - build requires the associated feature and its dependencies

  • auto - include associated feature if it and its dependencies are found

For instance, to build a Frescox binary that must use OpenMP but that should not use MPI despite the fact that MPI is installed, use

$ BFRESCOX_USE_MPI=disabled BFRESCOX_USE_OMP=enabled python -m pip install .

By default, auto is enabled.

UNOFFICIAL & UNTESTED CUSTOMIZATIONS

If a user would like to build Frescox using a local installation of BLAS/LAPACK (Issue 15), then they can set BFRESCOX_USE_LAPACK=enabled at installation.

Users can also build Frescox with extra functionality by setting

at installation.

5.1.3. Testing

The Bfrescoxpro Python package has a minimal, automated test suite integrated in the package that can be run to test an installation. After installing the package, the installation can be tested by executing

$ python
>>> import bfrescoxpro
>>> bfrescoxpro.__version__
<version>
>>> bfrescoxpro.print_information()
    ...
>>> bfrescoxpro.test()
    ...

While users are encouraged to perform extra testing of all Bfrescox and Bfrescoxpro installations, the customizability of Bfrescoxpro installations likely merits more extensive additional testing.

5.1.4. Troubleshooting

The automatic detection of the compiler and MPI implementation can be influenced by standard build system environment variables such as FC and MPIFC.

If the build system does not automatically discover the compiler, ensure that the compiler is in the PATH. If it still fails, try

$ FC=/path/to/compiler python -m pip install .

Note that this could also be used to override the choice of compiler made by the build system.

If the build system does not automatically discover the desired MPI installation, ensure that at least one of the installation’s mpif90, mpifort, mpiifort, etc. compiler wrappers is in the PATH. If it still fails, try

$ MPIFC=/path/to/wrapper python -m pip install .

5.1.5. Programmatic interface

While the documentation in Section 4 is geared toward Bfrescox, it is generally useful for Bfrescoxpro as well. The only differences is that in Bfrescoxpro the bfrescox.run_simulation() function has the additional required mpi_setup argument.

The following example demonstrates the use of mpi_setup by running an MPI+OpenMP Frescox simulation using the given standard Fortran NML configuration file with 5 OpenMP threads for each of 2 MPI processes.

import os

from pathlib import Path

import bfrescoxpro

os.environ["OMP_NUM_THREADS"] = 5

result = bfrescoxpro.run_simulation(
    configuration=bfrescoxpro.Configuration.from_NML("simulation.in"),
    filename=Path.cwd().joinpath("test.out"),
    mpi_setup={bfrescoxpro.N_MPI_PROCESSES: 2}
)

5.2. Using MPI and OpenMP with Bfrescoxpro

The Frescox parallelization manual for MPI and OpenMP usage is at github.com/llnl/Frescox/blob/master/man/parallel/openmp-f5a.pdf

5.3. Custom Frescox binary

Todo

  • Write this once we have basic functionality in the package.