From 6c314b32de1f02b4c40df52623ce275112ce846f Mon Sep 17 00:00:00 2001 From: Joshua Lampert Date: Sat, 4 Nov 2023 14:56:36 +0100 Subject: [PATCH 1/6] update docs on system-provided MPI including HDF5 --- docs/src/parallelization.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/docs/src/parallelization.md b/docs/src/parallelization.md index fa6fc1a5d32..9d46cc12b12 100644 --- a/docs/src/parallelization.md +++ b/docs/src/parallelization.md @@ -53,23 +53,24 @@ a system-provided MPI installation with Trixi.jl can be found in the following s ### [Using a system-provided MPI installation](@id parallel_system_MPI) -When using Trixi.jl with a system-provided MPI backend the underlying -[`p4est`](https://github.com/cburstedde/p4est) and [`t8code`](https://github.com/DLR-AMR/t8code) -libraries need to be compiled with the same MPI installation. Therefore, you also need to -use system-provided `p4est` and `t8code` installations (for notes on how to install `p4est` -and `t8code` see e.g. [here](https://github.com/cburstedde/p4est/blob/master/README) and -[here](https://github.com/DLR-AMR/t8code/wiki/Installation), use the configure option +When using Trixi.jl with a system-provided MPI backend, the underlying +[`p4est`](https://github.com/cburstedde/p4est), [`t8code`](https://github.com/DLR-AMR/t8code) +and [`hdf5`](https://github.com/HDFGroup/hdf5) libraries need to be compiled with the same MPI +installation. Therefore, you also need to use system-provided `p4est`, `t8code`, `hdf5` installations +(for notes on how to install `p4est` and `t8code` see e.g. [here](https://github.com/cburstedde/p4est/blob/master/README) +and [here](https://github.com/DLR-AMR/t8code/wiki/Installation), use the configure option `--enable-mpi`). Note that `t8code` already comes with a `p4est` installation, so it suffices -to install `t8code`. In addition, [P4est.jl](https://github.com/trixi-framework/P4est.jl) and -[T8code.jl](https://github.com/DLR-AMR/T8code.jl) need to be configured to use the custom -installations. Follow the steps described -[here](https://github.com/DLR-AMR/T8code.jl/blob/main/README.md#installation) and -[here](https://github.com/trixi-framework/P4est.jl/blob/main/README.md#installation) for the -configuration. The paths that point to `libp4est.so` (and potentially to `libsc.so`) need to be +to install `t8code`. In addition, [P4est.jl](https://github.com/trixi-framework/P4est.jl), +[T8code.jl](https://github.com/DLR-AMR/T8code.jl) and [HDF5.jl](https://github.com/JuliaIO/HDF5.jl) +need to be configured to use the custom installations. Follow the steps described +[here](https://github.com/DLR-AMR/T8code.jl/blob/main/README.md#installation), +[here](https://github.com/trixi-framework/P4est.jl/blob/main/README.md#installation) and +[here](https://trixi-framework.github.io/Trixi.jl/stable/parallelization/#Using-parallel-input-and-output) +for the configuration. The paths that point to `libp4est.so` (and potentially to `libsc.so`) need to be the same for P4est.jl and T8code.jl. This could e.g. be `libp4est.so` that usually can be found in `lib/` or `local/lib/` in the installation directory of `t8code`. In total, in your active Julia project you should have a LocalPreferences.toml file with sections -`[MPIPreferences]`, `[T8code]` and `[P4est]` as well as an entry `MPIPreferences` in your +`[MPIPreferences]`, `[T8code]`, `[P4est]` and `[HDF5]` as well as an entry `MPIPreferences` in your Project.toml to use a custom MPI installation. A `LocalPreferences.toml` file created as described above might look something like the following: ```toml From 1cc618aa6da0368b5d8ac8cb5496742f7bfd40e7 Mon Sep 17 00:00:00 2001 From: Joshua Lampert Date: Sat, 4 Nov 2023 14:59:33 +0100 Subject: [PATCH 2/6] fix typo --- docs/src/parallelization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/parallelization.md b/docs/src/parallelization.md index 9d46cc12b12..079661fc579 100644 --- a/docs/src/parallelization.md +++ b/docs/src/parallelization.md @@ -56,7 +56,7 @@ a system-provided MPI installation with Trixi.jl can be found in the following s When using Trixi.jl with a system-provided MPI backend, the underlying [`p4est`](https://github.com/cburstedde/p4est), [`t8code`](https://github.com/DLR-AMR/t8code) and [`hdf5`](https://github.com/HDFGroup/hdf5) libraries need to be compiled with the same MPI -installation. Therefore, you also need to use system-provided `p4est`, `t8code`, `hdf5` installations +installation. Therefore, you also need to use system-provided `p4est`, `t8code` and `hdf5` installations (for notes on how to install `p4est` and `t8code` see e.g. [here](https://github.com/cburstedde/p4est/blob/master/README) and [here](https://github.com/DLR-AMR/t8code/wiki/Installation), use the configure option `--enable-mpi`). Note that `t8code` already comes with a `p4est` installation, so it suffices From 49e02995b930a4922528abcd53e470810559d51c Mon Sep 17 00:00:00 2001 From: Joshua Lampert <51029046+JoshuaLampert@users.noreply.github.com> Date: Sun, 5 Nov 2023 11:06:51 +0100 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: Michael Schlottke-Lakemper --- docs/src/parallelization.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/src/parallelization.md b/docs/src/parallelization.md index 079661fc579..6bd55ddd7d4 100644 --- a/docs/src/parallelization.md +++ b/docs/src/parallelization.md @@ -55,23 +55,23 @@ a system-provided MPI installation with Trixi.jl can be found in the following s When using Trixi.jl with a system-provided MPI backend, the underlying [`p4est`](https://github.com/cburstedde/p4est), [`t8code`](https://github.com/DLR-AMR/t8code) -and [`hdf5`](https://github.com/HDFGroup/hdf5) libraries need to be compiled with the same MPI -installation. Therefore, you also need to use system-provided `p4est`, `t8code` and `hdf5` installations -(for notes on how to install `p4est` and `t8code` see e.g. [here](https://github.com/cburstedde/p4est/blob/master/README) +and [`HDF5`](https://github.com/HDFGroup/hdf5) libraries need to be compiled with the same MPI +installation. Therefore, you also need to use system-provided `p4est`, `t8code` and `HDF5` installations +(for notes on how to install `p4est` and `t8code` see, e.g., [here](https://github.com/cburstedde/p4est/blob/master/README) and [here](https://github.com/DLR-AMR/t8code/wiki/Installation), use the configure option `--enable-mpi`). Note that `t8code` already comes with a `p4est` installation, so it suffices to install `t8code`. In addition, [P4est.jl](https://github.com/trixi-framework/P4est.jl), -[T8code.jl](https://github.com/DLR-AMR/T8code.jl) and [HDF5.jl](https://github.com/JuliaIO/HDF5.jl) +[T8code.jl](https://github.com/DLR-AMR/T8code.jl), and [HDF5.jl](https://github.com/JuliaIO/HDF5.jl) need to be configured to use the custom installations. Follow the steps described [here](https://github.com/DLR-AMR/T8code.jl/blob/main/README.md#installation), -[here](https://github.com/trixi-framework/P4est.jl/blob/main/README.md#installation) and +[here](https://github.com/trixi-framework/P4est.jl/blob/main/README.md#installation), and [here](https://trixi-framework.github.io/Trixi.jl/stable/parallelization/#Using-parallel-input-and-output) for the configuration. The paths that point to `libp4est.so` (and potentially to `libsc.so`) need to be the same for P4est.jl and T8code.jl. This could e.g. be `libp4est.so` that usually can be found in `lib/` or `local/lib/` in the installation directory of `t8code`. -In total, in your active Julia project you should have a LocalPreferences.toml file with sections -`[MPIPreferences]`, `[T8code]`, `[P4est]` and `[HDF5]` as well as an entry `MPIPreferences` in your -Project.toml to use a custom MPI installation. A `LocalPreferences.toml` file +In total, in your active Julia project you should have a `LocalPreferences.toml` file with sections +`[MPIPreferences]`, `[T8code]`, `[P4est]`, and `[HDF5]` as well as an entry `MPIPreferences` in your +`Project.toml` to use a custom MPI installation. A `LocalPreferences.toml` file created as described above might look something like the following: ```toml [HDF5] From 9b0089170e6124e377152eb5f118e7e826de408d Mon Sep 17 00:00:00 2001 From: Joshua Lampert Date: Sun, 5 Nov 2023 11:46:26 +0100 Subject: [PATCH 4/6] put LocalPreferences.toml in code font --- docs/src/parallelization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/parallelization.md b/docs/src/parallelization.md index 6bd55ddd7d4..9041a494542 100644 --- a/docs/src/parallelization.md +++ b/docs/src/parallelization.md @@ -219,7 +219,7 @@ julia> HDF5.API.set_libraries!("/path/to/your/libhdf5.so", "/path/to/your/libhdf ``` For more information see also the [documentation of HDF5.jl](https://juliaio.github.io/HDF5.jl/stable/mpi/). In total, you should -have a file called LocalPreferences.toml in the project directory that contains a section +have a file called `LocalPreferences.toml` in the project directory that contains a section `[MPIPreferences]`, a section `[HDF5]` with entries `libhdf5` and `libhdf5_hl`, a section `[P4est]` with the entry `libp4est` as well as a section `[T8code]` with the entries `libt8`, `libp4est` and `libsc`. From af508a6a176975b88ef05019fa88cd49a854d35c Mon Sep 17 00:00:00 2001 From: Joshua Lampert Date: Tue, 7 Nov 2023 15:15:35 +0100 Subject: [PATCH 5/6] add notes when which preferences need to be set --- docs/src/parallelization.md | 49 ++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/docs/src/parallelization.md b/docs/src/parallelization.md index 9041a494542..dc4c3d6c91e 100644 --- a/docs/src/parallelization.md +++ b/docs/src/parallelization.md @@ -56,21 +56,28 @@ a system-provided MPI installation with Trixi.jl can be found in the following s When using Trixi.jl with a system-provided MPI backend, the underlying [`p4est`](https://github.com/cburstedde/p4est), [`t8code`](https://github.com/DLR-AMR/t8code) and [`HDF5`](https://github.com/HDFGroup/hdf5) libraries need to be compiled with the same MPI -installation. Therefore, you also need to use system-provided `p4est`, `t8code` and `HDF5` installations +installation. If you want to use `p4est` (via the `P4estMesh`) or `t8code` (via the `T8codeMesh`) +from Trixi.jl, you also need to use system-provided `p4est` or `t8code` installations (for notes on how to install `p4est` and `t8code` see, e.g., [here](https://github.com/cburstedde/p4est/blob/master/README) and [here](https://github.com/DLR-AMR/t8code/wiki/Installation), use the configure option -`--enable-mpi`). Note that `t8code` already comes with a `p4est` installation, so it suffices -to install `t8code`. In addition, [P4est.jl](https://github.com/trixi-framework/P4est.jl), -[T8code.jl](https://github.com/DLR-AMR/T8code.jl), and [HDF5.jl](https://github.com/JuliaIO/HDF5.jl) -need to be configured to use the custom installations. Follow the steps described -[here](https://github.com/DLR-AMR/T8code.jl/blob/main/README.md#installation), -[here](https://github.com/trixi-framework/P4est.jl/blob/main/README.md#installation), and -[here](https://trixi-framework.github.io/Trixi.jl/stable/parallelization/#Using-parallel-input-and-output) +`--enable-mpi`). Otherwise, there will be warnings that no preference is set for P4est.jl and +T8code.jl that can be ignored if you do not use these libraries from Trixi.jl. Note that +`t8code` already comes with a `p4est` installation, so it suffices to install `t8code`. +In order to use system-provided `p4est` and `t8code` installations, [P4est.jl](https://github.com/trixi-framework/P4est.jl) +and [T8code.jl](https://github.com/DLR-AMR/T8code.jl) need to be configured to use the custom +installations. Follow the steps described [here](https://github.com/DLR-AMR/T8code.jl/blob/main/README.md#installation) and +[here](https://github.com/trixi-framework/P4est.jl/blob/main/README.md#installation). for the configuration. The paths that point to `libp4est.so` (and potentially to `libsc.so`) need to be -the same for P4est.jl and T8code.jl. This could e.g. be `libp4est.so` that usually can be found -in `lib/` or `local/lib/` in the installation directory of `t8code`. +the same for P4est.jl and T8code.jl. This could, e.g., be `libp4est.so` that usually can be found +in `lib/` or `local/lib/` in the installation directory of `t8code`. The preferences for +[HDF5.jl](https://github.com/JuliaIO/HDF5.jl) always need to be set, even if you do not want to +use `HDF5` from Trixi.jl, see also https://github.com/JuliaIO/HDF5.jl/issues/1079. To set the +preferences for HDF5.jl, follow the instructions described +[here](https://trixi-framework.github.io/Trixi.jl/stable/parallelization/#Using-parallel-input-and-output). + In total, in your active Julia project you should have a `LocalPreferences.toml` file with sections -`[MPIPreferences]`, `[T8code]`, `[P4est]`, and `[HDF5]` as well as an entry `MPIPreferences` in your +`[MPIPreferences]`, `[T8code]` (only needed if `T8codeMesh` is used), `[P4est]` (only needed if +`P4estMesh` is used), and `[HDF5]` as well as an entry `MPIPreferences` in your `Project.toml` to use a custom MPI installation. A `LocalPreferences.toml` file created as described above might look something like the following: ```toml @@ -78,6 +85,10 @@ created as described above might look something like the following: libhdf5 = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so" libhdf5_hl = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5_hl.so" +[HDF5_jll] +libhdf5_hl_path = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5_hl.so" +libhdf5_path = "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so" + [MPIPreferences] __clear__ = ["preloads_env_switch"] _format = "1.0" @@ -98,6 +109,22 @@ libsc = "/home/mschlott/hackathon/libtrixi/t8code/install/lib/libsc.so" libt8 = "/home/mschlott/hackathon/libtrixi/t8code/install/lib/libt8.so" ``` +This file is created with the following sequence of commands: +```julia +julia> using MPIPreferences +julia> MPIPreferences.use_system_binary() +``` +Restart the Julia REPL +```julia +julia> using P4est +julia> P4est.set_library_p4est!("/home/mschlott/hackathon/libtrixi/t8code/install/lib/libp4est.so") +julia> P4est.set_library_sc!("/home/mschlott/hackathon/libtrixi/t8code/install/lib/libsc.so") +julia> using T8code +julia> T8code.set_libraries_path!("/home/mschlott/hackathon/libtrixi/t8code/install/lib/") +julia> using HDF5 +julia> HDF5.API.set_libraries!("/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so", "/usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5_hl.so") +``` +After the preferences are set, restart the Julia REPL again. ### [Usage](@id parallel_usage) From 6992e07b1c2f9fce822badc8832b554985caf457 Mon Sep 17 00:00:00 2001 From: Joshua Lampert Date: Sun, 19 Nov 2023 18:51:07 +0100 Subject: [PATCH 6/6] T8codeMesh does not support MPI yet --- docs/src/parallelization.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/src/parallelization.md b/docs/src/parallelization.md index dc4c3d6c91e..3cce7c381b2 100644 --- a/docs/src/parallelization.md +++ b/docs/src/parallelization.md @@ -69,10 +69,11 @@ installations. Follow the steps described [here](https://github.com/DLR-AMR/T8co [here](https://github.com/trixi-framework/P4est.jl/blob/main/README.md#installation). for the configuration. The paths that point to `libp4est.so` (and potentially to `libsc.so`) need to be the same for P4est.jl and T8code.jl. This could, e.g., be `libp4est.so` that usually can be found -in `lib/` or `local/lib/` in the installation directory of `t8code`. The preferences for -[HDF5.jl](https://github.com/JuliaIO/HDF5.jl) always need to be set, even if you do not want to -use `HDF5` from Trixi.jl, see also https://github.com/JuliaIO/HDF5.jl/issues/1079. To set the -preferences for HDF5.jl, follow the instructions described +in `lib/` or `local/lib/` in the installation directory of `t8code`. Note that the `T8codeMesh`, however, +does not support MPI yet. +The preferences for [HDF5.jl](https://github.com/JuliaIO/HDF5.jl) always need to be set, even if you +do not want to use `HDF5` from Trixi.jl, see also https://github.com/JuliaIO/HDF5.jl/issues/1079. +To set the preferences for HDF5.jl, follow the instructions described [here](https://trixi-framework.github.io/Trixi.jl/stable/parallelization/#Using-parallel-input-and-output). In total, in your active Julia project you should have a `LocalPreferences.toml` file with sections