CUDA Docker Container #1141

chapman39 · 2024-06-12T21:26:08Z

This PR adds a new docker container with CUDA, so we can test GPU support in Azure. Due to space limitations of Azure VMs, TPLs are built with +shared and serac is built with -DBUILD_SHARED_LIBS=ON for Docker containers.

Fixes #1117

scripts/spack/packages/serac/package.py

…cuda-docker-container

chapman39 · 2024-07-19T22:05:59Z

scripts/spack/packages/serac/package.py

+    # Only propagate shared if not CUDA
+    depends_on("umpire build_type=Debug".format(dep), when="+umpire build_type=Debug".format(dep))
+    depends_on("umpire+shared".format(dep), when="+umpire+shared~cuda".format(dep))
+    depends_on("umpire~shared".format(dep), when="+umpire~shared".format(dep))


Looks a bit ugly, but umpire needed it's own section, due to this conflict: https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/umpire/package.py#L293

chapman39 · 2024-07-23T23:01:24Z

@samuelpmishLLNL @white238 I'm getting a weird NVCC error. It's complaining about a null pointer dereference in the stdlib.

https://dev.azure.com/llnl-serac/serac/_build/results?buildId=12183&view=logs&j=6e1d03e6-cc5b-563e-720e-c51be027141d&t=f6d84462-d4c0-58c0-06af-1f9ba7b544b8&l=1563

In file included from /usr/include/c++/12/functional:59,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/gtest-matchers.h:43,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/internal/gtest-death-test-internal.h:47,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/gtest-death-test.h:43,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/gtest.h:65,
                 from /home/serac/serac/src/serac/physics/tests/thermal_finite_diff.cpp:10:
In member function ‘bool std::_Function_base::_M_empty() const’,
    inlined from ‘_Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = serac::tuple<mfem::Vector&, serac::Functional<serac::H1<1>(serac::H1<1, 2>, serac::H1<1>, serac::H1<1>), serac::ExecutionSpace::CPU>::Gradient&>; _ArgTypes = {double}]’ at /usr/include/c++/12/bits/std_function.h:589:14,
    inlined from ‘serac::FiniteElementDual& serac::HeatTransfer<order, dim, serac::Parameters<parameter_space ...>, std::integer_sequence<int, parameter_indices ...> >::computeTimestepSensitivity(size_t) [with int order = 1; int dim = 2; parameter_space = {}; int ...parameter_indices = {}]’ at /home/serac/serac/src/serac/infrastructure/../../serac/physics/heat_transfer.hpp:964:78:
/usr/include/c++/12/bits/std_function.h:247:37: error: null pointer dereference [-Werror=null-dereference]
  247 |     bool _M_empty() const { return !_M_manager; }
      |                                     ^~~~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [src/serac/physics/tests/CMakeFiles/thermal_finite_diff.dir/build.make:76: src/serac/physics/tests/CMakeFiles/thermal_finite_diff.dir/thermal_finite_diff.cpp.o] Error 1
make[2]: Leaving directory '/home/serac/serac/_serac_build_and_test_2024_07_23_15_30_44/[email protected]_cuda'
make[1]: *** [CMakeFiles/Makefile2:3058: src/serac/physics/tests/CMakeFiles/thermal_finite_diff.dir/all] Error 2

host-configs/docker/[email protected]_cuda.cmake

white238 · 2024-07-24T01:08:57Z

@samuelpmishLLNL @white238 I'm getting a weird NVCC error. It's complaining about a null pointer dereference in the stdlib.

https://dev.azure.com/llnl-serac/serac/_build/results?buildId=12183&view=logs&j=6e1d03e6-cc5b-563e-720e-c51be027141d&t=f6d84462-d4c0-58c0-06af-1f9ba7b544b8&l=1563

In file included from /usr/include/c++/12/functional:59,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/gtest-matchers.h:43,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/internal/gtest-death-test-internal.h:47,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/gtest-death-test.h:43,
                 from /home/serac/serac/cmake/blt/thirdparty_builtin/googletest/googletest/include/gtest/gtest.h:65,
                 from /home/serac/serac/src/serac/physics/tests/thermal_finite_diff.cpp:10:
In member function ‘bool std::_Function_base::_M_empty() const’,
    inlined from ‘_Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = serac::tuple<mfem::Vector&, serac::Functional<serac::H1<1>(serac::H1<1, 2>, serac::H1<1>, serac::H1<1>), serac::ExecutionSpace::CPU>::Gradient&>; _ArgTypes = {double}]’ at /usr/include/c++/12/bits/std_function.h:589:14,
    inlined from ‘serac::FiniteElementDual& serac::HeatTransfer<order, dim, serac::Parameters<parameter_space ...>, std::integer_sequence<int, parameter_indices ...> >::computeTimestepSensitivity(size_t) [with int order = 1; int dim = 2; parameter_space = {}; int ...parameter_indices = {}]’ at /home/serac/serac/src/serac/infrastructure/../../serac/physics/heat_transfer.hpp:964:78:
/usr/include/c++/12/bits/std_function.h:247:37: error: null pointer dereference [-Werror=null-dereference]
  247 |     bool _M_empty() const { return !_M_manager; }
      |                                     ^~~~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [src/serac/physics/tests/CMakeFiles/thermal_finite_diff.dir/build.make:76: src/serac/physics/tests/CMakeFiles/thermal_finite_diff.dir/thermal_finite_diff.cpp.o] Error 1
make[2]: Leaving directory '/home/serac/serac/_serac_build_and_test_2024_07_23_15_30_44/[email protected]_cuda'
make[1]: *** [CMakeFiles/Makefile2:3058: src/serac/physics/tests/CMakeFiles/thermal_finite_diff.dir/all] Error 2

My recommendation is to turn off warnings as errors for this build and log an issue. Off-hand I can't figure out where that is actually coming from w/o a deeper look.

chapman39 · 2024-07-26T22:50:24Z

On the codevelop azure pipelines built with BUILD_SHARED_LIBS=ON, I got some errors while running serac tests:

https://dev.azure.com/llnl-serac/serac/_build/results?buildId=12196&view=logs&j=6120c41f-dd84-5658-817e-72df38d78194&t=c1c85309-f09f-566e-0fcd-ac74c434141f&l=2675

5: [ERROR in line 974 of file /home/serac/serac/src/serac/numerics/equation_solver.cpp]
5: MESSAGE=AMGX requested in non-GPU build

and

7: HDF5-DIAG: Error detected in HDF5 (1.8.23) thread 0:
7:   #000: /home/serac/serac_tpls/build_stage/spack-stage-hdf5-1.8.23-chknnutjvno3stawugxsj4u4a6nci7s6/spack-src/src/H5L.c line 1131 in H5Literate(): link iteration failed
7:     major: Symbol table
7:     minor: Iteration failed
7:   #001: /home/serac/serac_tpls/build_stage/spack-stage-hdf5-1.8.23-chknnutjvno3stawugxsj4u4a6nci7s6/spack-src/src/H5Gint.c line 812 in H5G_iterate(): error iterating over links
7:     major: Symbol table
7:     minor: Iteration failed
7:   #002: /home/serac/serac_tpls/build_stage/spack-stage-hdf5-1.8.23-chknnutjvno3stawugxsj4u4a6nci7s6/spack-src/src/H5Gobj.c line 661 in H5G__obj_iterate(): can't iterate over dense links
7:     major: Symbol table
7:     minor: Iteration failed
7:   #003: /home/serac/serac_tpls/build_stage/spack-stage-hdf5-1.8.23-chknnutjvno3stawugxsj4u4a6nci7s6/spack-src/src/H5Gdense.c line 1020 in H5G__dense_iterate(): iteration operator failed
7:     major: Symbol table
7:     minor: Can't move to next iterator location
7:   #004: /home/serac/serac_tpls/build_stage/spack-stage-hdf5-1.8.23-chknnutjvno3stawugxsj4u4a6nci7s6/spack-src/src/H5Glink.c line 478 in H5G__link_iterate_table(): iteration operator failed
7:     major: Symbol table
7:     minor: Can't move to next iterator location

among others. I was having some trouble figuring out why. The AMGX one is especially weird since I've checked the CMakeCache.txt and MFEM_USE_AMGX is false. For now, I simply set build shared off for codevelop, but we might want to figure this out.

.gitignore

scripts/spack/packages/serac/package.py

white238 · 2024-07-30T19:50:02Z

Thanks for sticking through this @chapman39 ! I know it was not a minor feat.

Setup for cuda docker container

feb7ebf

chapman39 added WIP Work in progress CI Continuous Integration gpu GPU related labels Jun 12, 2024

chapman39 self-assigned this Jun 12, 2024

chapman39 commented Jun 12, 2024

View reviewed changes

scripts/spack/packages/serac/package.py Show resolved Hide resolved

chapman39 added 24 commits July 5, 2024 15:11

Merge remote-tracking branch 'origin/develop' into feature/chapman39/…

487397e

…cuda-docker-container

Improve docker script error-checking

0a6e3b8

Update cuda container to use gcc12

cd790e1

Add cuda dockerfile to github actions yml

59f377c

Merge remote-tracking branch 'origin/develop' into feature/chapman39/…

de6dcee

…cuda-docker-container

Remove installing texlive and doxygen in cuda as a test

8ea6c98

Show disk space in other containers

59c11f3

Add df's, clang12, remove make test from cuda

ccd3e53

Try maximize build space github action

730b2b0

Update printing of df

cdecd46

update github action with section

be7c2de

Try moving maximize build space github action and doing default options

e923c91

remove unused packages (dotnet, android, ghc)

0294314

Delete more stuff

f8fdf4b

Add cuda docker host config

e14d003

Merge remote-tracking branch 'origin/develop' into feature/chapman39/…

8a89207

…cuda-docker-container

Merge remote-tracking branch 'origin/develop' into feature/chapman39/…

82b5037

…cuda-docker-container

See size before build src in azure

63d37c7

Print in right spot

21d1eb6

Adjust dockerfile generator script for CUDA

03d1a49

remove df

eb574dd

Build shared libs in azure

c9423ac

Merge remote-tracking branch 'origin/develop' into feature/chapman39/…

22001ae

…cuda-docker-container

Ignore local TPL build symlink

833625b

chapman39 added 2 commits July 18, 2024 16:45

Force sundials to be built static

f858be5

Don't propagate +shared to umpire if +cuda

7a023b1

chapman39 commented Jul 19, 2024

View reviewed changes

chapman39 added 3 commits July 19, 2024 16:46

Fix minor spack mistake

224504d

Remove static propagation based on shared variant for sundials

6768ac9

Update docker host configs

9ea3336

Skip unit tests for CUDA azure

17244b7

chapman39 commented Jul 24, 2024

View reviewed changes

host-configs/docker/[email protected]_cuda.cmake Show resolved Hide resolved

chapman39 added 3 commits July 24, 2024 09:13

Feedback and type-o fix

7a51b4f

Resolve more azure failures

dc9b2a5

See if turning off shared libs on codevelop resolves failing unit tests

0da5a62

chapman39 requested a review from white238 July 29, 2024 20:31

chapman39 changed the title ~~WIP: CUDA Docker Container~~ CUDA Docker Container Jul 29, 2024

chapman39 requested review from btalamini, tupek2 and zatkins-work July 29, 2024 20:32

chapman39 removed the WIP Work in progress label Jul 29, 2024

white238 reviewed Jul 29, 2024

View reviewed changes

.gitignore Show resolved Hide resolved

Merge branch 'develop' into feature/chapman39/cuda-docker-container

1d108df

white238 reviewed Jul 29, 2024

View reviewed changes

scripts/spack/packages/serac/package.py Show resolved Hide resolved

white238 approved these changes Jul 29, 2024

View reviewed changes

improve umpire comment

9ff0b78

btalamini approved these changes Jul 30, 2024

View reviewed changes

chapman39 merged commit e27f6a1 into develop Jul 30, 2024
2 checks passed

chapman39 deleted the feature/chapman39/cuda-docker-container branch July 30, 2024 20:58

white238 mentioned this pull request Aug 7, 2024

AWS tutorial Docker image update LLNL/axom#1388

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA Docker Container #1141

CUDA Docker Container #1141

chapman39 commented Jun 12, 2024 •

edited

Loading

chapman39 Jul 19, 2024 •

edited

Loading

chapman39 commented Jul 23, 2024

white238 commented Jul 24, 2024

chapman39 commented Jul 26, 2024

white238 commented Jul 30, 2024

CUDA Docker Container #1141

CUDA Docker Container #1141

Conversation

chapman39 commented Jun 12, 2024 • edited Loading

chapman39 Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

chapman39 commented Jul 23, 2024

white238 commented Jul 24, 2024

chapman39 commented Jul 26, 2024

white238 commented Jul 30, 2024

chapman39 commented Jun 12, 2024 •

edited

Loading

chapman39 Jul 19, 2024 •

edited

Loading