From ef97b788b860018c3d2fa74efa7f80c65126c419 Mon Sep 17 00:00:00 2001 From: pelesh Date: Thu, 4 Jan 2024 02:44:08 -0500 Subject: [PATCH] Update user guide. (#133) * Update user guide. * Reorganize documentation directories * Add docs on HPCToolkit profiling * Add rocprof instructions to the documentation. --- LICENSE | 25 +-- README.md | 6 +- docs/conf.py | 2 +- docs/index.rst | 59 ++++-- .../index.rst | 0 docs/sphinx/developer_guide/index.rst | 83 +++++++- docs/sphinx/developer_guide/profiling.rst | 190 ++++++++++++++++++ docs/sphinx/license.rst | 5 + docs/sphinx/licenses.rst | 5 - .../{guide.rst => user_guide/index.rst} | 50 +++-- resolve/Doxygen.hpp | 3 +- 11 files changed, 359 insertions(+), 69 deletions(-) rename docs/sphinx/{coding_guide => coding_guidelines}/index.rst (100%) create mode 100644 docs/sphinx/developer_guide/profiling.rst create mode 100644 docs/sphinx/license.rst delete mode 100644 docs/sphinx/licenses.rst rename docs/sphinx/{guide.rst => user_guide/index.rst} (72%) diff --git a/LICENSE b/LICENSE index 871c34b9..b8f645f9 100644 --- a/LICENSE +++ b/LICENSE @@ -2,9 +2,10 @@ Copyright (c) 2023, UT-Battelle, LLC and Battelle Memorial Institute. All rights reserved. ReSolve version 1.0 - + + Open Source License: - + Subject to the conditions of this License, UT-Battelle, LLC and Battelle Memorial Institute hereby grant, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), a @@ -26,16 +27,16 @@ endorse or promote products derived from this Software without their specific prior written permission. 3. The Software, with or without modification, must include the following -acknowledgment: - "This product includes software produced by UT-Battelle, LLC under Contract -No. DE-AC05-00OR22725 and Battelle Memorial Institute under Contract -No. DE-AC05-76RL01830 with the Department of Energy.” - - - -****************************************************************************** -DISCLAIMER +acknowledgment: "This product includes software produced by UT-Battelle, LLC +under Contract No. DE-AC05-00OR22725 and Battelle Memorial Institute under +Contract No. DE-AC05-76RL01830 with the Department of Energy.” + + +------------------------------------------------------------------------------- + +DISCLAIMER: + UT-BATTELLE, LLC, BATTELLE MEMORIAL INSTITUTE, AND THE GOVERNMENT MAKE NO REPRESENTATIONS AND DISCLAIM ALL WARRANTIES, BOTH EXPRESSED AND IMPLIED. THERE ARE NO EXPRESS OR IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR @@ -45,4 +46,4 @@ WILL ACCOMPLISH THE INTENDED RESULTS OR THAT THE SOFTWARE OR ITS USE WILL NOT RESULT IN INJURY OR DAMAGE. THE USER ASSUMES RESPONSIBILITY FOR ALL LIABILITIES, PENALTIES, FINES, CLAIMS, CAUSES OF ACTION, AND COSTS AND EXPENSES, CAUSED BY, RESULTING FROM OR ARISING OUT OF, IN WHOLE OR IN PART -THE USE, STORAGE OR DISPOSAL OF THE SOFTWARE. \ No newline at end of file +THE USE, STORAGE OR DISPOSAL OF THE SOFTWARE. diff --git a/README.md b/README.md index 36a6310a..e7e27277 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,10 @@ # ReSolve -ReSolve is a library of GPU-resident linear solver. It contains iterative and direct linear solvers designed to run on NVIDIA and AMD GPUs, as well as on CPU devices. +ReSolve is a library of GPU-resident linear solvers. It contains iterative and direct solvers designed to run on NVIDIA and AMD GPUs, as well as on CPU devices. + +The User Guide and developer's documentation is available [online](https://resolve.readthedocs.io/en/version-1.0/), +including Doxygen-generated [source code documentation](https://resolve.readthedocs.io/en/version-1.0/doxygen/html/index.html). -The User Guide and developer's documentation is available [online](https://resolve.readthedocs.io/en/version-1.0/) ## Getting started diff --git a/docs/conf.py b/docs/conf.py index 97d0b66b..283d0eb0 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -199,7 +199,7 @@ # dir menu entry, description, category) texinfo_documents = [ (master_doc, 'ReSolve', u'ReSolve Documentation', - 'ReSolve Team', 'ReSolve', 'ReSolve is a library of GPU-resident linear solver.', + 'ReSolve Team', 'ReSolve', 'ReSolve is a library of GPU-resident linear solvers.', 'Miscellaneous'), ] diff --git a/docs/index.rst b/docs/index.rst index e07793cf..7a319e71 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -2,23 +2,28 @@ ReSolve ******* -ReSolve is a library of GPU-resident linear solver. It contains iterative and direct linear solvers designed to run on NVIDIA and AMD GPUs, as well as on CPU devices. +ReSolve is an open-source library that provides GPU-resident linear solvers. +It contains iterative and direct solvers designed to run on NVIDIA and AMD +GPUs, as well as on CPU devices. + +ReSolve source code and documentation are available at +`GitHub `_. -To learn how to get start with resolve please checkout our `User Guide `_ -============= Documentation -============= +------------- + +To get started, please check our `User Guide `_. +Source code documentation generated in Doxygen is also +`linked `_ to this site. -Source code documentation are also linked on this site. -`ReSolve Github Project `_. -`Source documentation `_ Contributing ------------ -For all contributions to ReSolve please follow the `developer -guidelines `__ +For all contributions to ReSolve please consult +`Developer Guide `_ and follow the +`Coding Style Guidelines `_. Authors and acknowledgment -------------------------- @@ -29,7 +34,7 @@ Primary authors of this project are: * Slaven Peles peless@ornl.gov (ORNL) ReSolve project would not be possible without significant contributions -from (in alphabetic order): +from (in alphabetic order): * Maksudul Alam (ORNL) * Ryan Danehy (PNNL) @@ -38,18 +43,37 @@ from (in alphabetic order): * Phil Roth (ORNL) * Cameron Rutherford (PNNL) -Development of this code was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations—the Office of Science and the National Nuclear Security Administration—responsible for the planning and preparation of a capable exascale ecosystem—including software, applications, hardware, advanced system engineering, and early testbed platforms—to support the nation’s exascale computing imperative. +Development of this code was supported by the Exascale Computing Project (ECP), +Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations +— the Office of Science and the National Nuclear Security Administration — +responsible for the planning and preparation of a capable exascale ecosystem +— including software, applications, hardware, advanced system engineering, and +early testbed platforms — to support the nation’s exascale computing +imperative. License ------- +ReSolve is a free software distributed under a BSD-style license. See +the `LICENSE `__ and `NOTICE `__ +for more details. All new contributions to ReSolve must be made under the +same licensing terms. + +**Please Note:** If you are using ReSolve with any third party libraries linked +in (e.g., KLU), be sure to review the respective license of the package as that +license may have more restrictive terms than the ReSolve license. + Copyright © 2023, UT-Battelle, LLC, and Battelle Memorial Institute. -ReSolve is a free software distributed under a BSD-style license. -See the `LICENSE `__ and `NOTICE `__ files for details. All -new contributions to ReSolve must be made under the same licensing -terms. +.. toctree:: + :maxdepth: 2 + :hidden: + :caption: User Resources + + sphinx/user_guide/index + sphinx/license + sphinx/notice .. toctree:: @@ -65,8 +89,7 @@ terms. :hidden: :caption: Developer Resources - sphinx/coding_guide/index + sphinx/coding_guidelines/index doxygen/index - sphinx/licenses - sphinx/notice sphinx/developer_guide/index + sphinx/developer_guide/profiling diff --git a/docs/sphinx/coding_guide/index.rst b/docs/sphinx/coding_guidelines/index.rst similarity index 100% rename from docs/sphinx/coding_guide/index.rst rename to docs/sphinx/coding_guidelines/index.rst diff --git a/docs/sphinx/developer_guide/index.rst b/docs/sphinx/developer_guide/index.rst index 239190c7..7581d682 100644 --- a/docs/sphinx/developer_guide/index.rst +++ b/docs/sphinx/developer_guide/index.rst @@ -18,11 +18,18 @@ We also export under the ``ReSolve::`` namespace in our installed CMake configur Spack Package --------------- -Our current spack package is introduced in https://github.com/spack/spack/pull/40871, and contains support for building Re::Solve with CUDA and HIP/ROCm support. +Our current Spack package has been introduced +`upstream `_, and contains support +for building Re::Solve with CUDA and HIP/ROCm support. -We also have a custom ``spack`` folder/installation that contains our spack submodule located in ``buildsystem/spack/spack``. This is used to build Re::Solve on CI platforms, as well as support development of the spack package as neccessary. +We also have a custom ``spack`` folder/installation that contains our spack +submodule located in ``buildsystem/spack/spack``. This is used to build +Re::Solve on CI platforms, as well as support development of the spack package +as neccessary. -See the Quik-How-To section below for more information on how to update the spack package and typical workflows for building Re::Solve with spack on CI platforms for testing. +See the Quik-How-To section below for more information on how to update the +spack package and typical workflows for building Re::Solve with spack on CI +platforms for testing. GitHub Actions @@ -83,6 +90,76 @@ Deception specific CI. Incline specific CI. + +Writing Documentation +--------------------- + +Re::Solve uses Sphynx for the documentation. To write and preview the +documentation on your local machine use e.g. ``pip`` tool to install following +Python packages: + +.. code:: shell + + pip install sphinx docutils sphinx_rtd_theme sphinxcontrib-jquery m2r2 + +If you prefer using Anaconda utilities, getting these packages is +slightly different: + +.. code:: shell + + conda install sphinx docutils sphinx_rtd_theme + conda install -c conda-forge sphinxcontrib-jquery m2r2 + + +Once you have all the required packages, you can build the HTML docs by + +.. code:: shell + + git clone git@github.com:ORNL/ReSolve.git + sphinx-build -M html ReSolve/docs/ ./build + +This will generate HTML documentation and place it in ``build`` +subdirectory in your current directory. + + +Using Dev Container for Writing Documentation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In case you cannot install Sphynx and other dependencies on your machine, +Re::Solve provides scripts for building development container with all +tools required for Sphynx documentation generation. To create container +for documentation development follow these straightforward steps: + +Prerequisites +""""""""""""" + +#. install Docker Desktop and launch the app +#. install the "Remote Development" extension in VSCode +#. open your local clone of resolve in VSCode + + +Build Container +""""""""""""""" + +The build info for this container is in `.devcontainer/`. There is a Dockerfile and +json file associated with the configuration. + +#. if connected, disconnect from the PNNL VPN +#. launch the container build + + * `cmd shift p` to open the command pallette in vscode + * click `> Dev Container: rebuild and reopen container` + * this will start building the container, taking about 40 minutes + * click on the pop up with `(show log)` to view the progress + +#. Open new terminal within Visual Studio Code and run the renderDocs.sh (note this takes a minute) +#. Open the link that was served to you after step 3 + +.. note:: Pushing/pulling from git is not supported in a devcontainer, + and should be done separately. + + + Devcontainer Configuration ---------------------------- diff --git a/docs/sphinx/developer_guide/profiling.rst b/docs/sphinx/developer_guide/profiling.rst new file mode 100644 index 00000000..fcfd4686 --- /dev/null +++ b/docs/sphinx/developer_guide/profiling.rst @@ -0,0 +1,190 @@ +Profiling +========= + + + +########## +HPCToolkit +########## + +`HPCToolkit `_ can profile any code built with debug +symbols, it does not require special code instrumenting. In CMake, one simply +needs to select ``RelWithDebInfo`` as the build type. + +Consult HPCToolkit `documentation `_ +to find out how to obtain and install all the required profiling software. +Hopefully, your system administrator will install HPCToolkit for you and you +would only need to load appropriate modules. + +.. note:: On Frontier supercomputer all you need to get are ``ums``, + ``ums023`` and ``hpctoolkit`` modules. + +Once you built your code with debug symbols and got the working HPCToolkit +installation, you run profiling with + +.. code:: shell + + hpcrun -t -e CPUTIME -e gpu=amd -o out.m my_executable.exe + +`hpcrun `_ is a profiling tool that +collects call path profiles of program executions using statistical sampling +of hardware counters, software counters, or timers. In the example above: + + * Flag ``-t`` tells profiler to generate a call path trace in addition to a + call path profile. + * Flag ``-e`` selects events or sampling periods for profiling. + + * ``-e CPUTIME`` will tell profiler to use Linux system timer to set the + sampling rate. + * ``-e gpu=amd`` will instruct profiler to collect comprehensive + operation-level measurements for HIP programs on AMD GPU. + + * Flag ``-o`` specifies the directory where the output data will be stored. + * In this example, ``my_executable.exe`` is the binary being profiled. + +It is often helpful to map profiling data to specific CPU and GPU code. This +can be done by invoking `hpcstruct `_ +command as + +.. code:: shell + + hpcstruct out.m + +where ``out.m`` is the directory with profiling measurements collected by +``hpcrun``. This step is optional. + +`hpcprof `_ analyzes profile +performance measurements and attributes them to static source code structure. +One can simply call + +.. code:: shell + + hpcprof -o out.d out.m + +This will generate profiling information using measurements in directory +``out.m`` and store it in directory ``out.d``. The data in ``out.d`` can be +viewed using ``hpcviewer`` tool. + +It is recommended to download and install graphical version of +`HPCToolkit viewer `_ on your local +machine and analyze profiling data there. The graphical user interface provides +many productivity features but often runs slowly over SSH connection. + +When running the profiler on a machine with a scheduler, it is best to use +a script. When using SLURM, for example, a script to run the profiler would +look something like this: + +.. code:: shell + + #!/bin/bash + + #SBATCH -A MyAllocation + #SBATCH -J resolve_profile + #SBATCH -o %x-%j.out + #SBATCH -t 00:30:00 + #SBATCH -N 1 + + EXE=resolve_executable.exe + OUT=hpctoolkit_resolve_profile + + # Profile ReSolve code on a single GPU + echo "`date` Starting run" + srun -N 1 -n 1 -c 1 -G 1 \ + hpcrun -t -e CPUTIME -e gpu=amd -o ${OUT}.m ${EXE} + echo "`date` Finished run" + + # Use 56 cores to process profiling measurements + srun -N 1 -n 1 -c 56 hpcstruct ${OUT}.m + srun -N 1 -n 1 -c 56 hpcprof -o ${OUT}.d ${OUT}.m + +Any arguments that the executable takes can be simply added as when the +executable is called locally. Similar script could be written for LSF or MOAB +scheduler. + +########### +ROCProfiler +########### + +Unlike HPCToolkit, `ROCProfiler `_ +requires code to be instrumented using `ROC Tracer `_ +library. Both, ROCProfiler and ROC Tracer are part of the ROCm library, so +no additional software needs to be installed once you obtain ROCm. + +To build your instrumented code, you need to link your Re::Solve build to +ROC Tracer library: + +.. code:: cmake + + target_include_directories(ReSolve SYSTEM PUBLIC ${HIP_PATH}/roctracer/include ${HIP_PATH}/include ) + target_link_libraries(ReSolve PUBLIC "-L${HIP_PATH}/roctracer/lib -lroctracer64" "-L${HIP_PATH}/roctracer/lib -lroctx64" ) + +Next, you need to annotate events you want to trace in your code execution. +This could be done in a straightforward manner using ROC Tracer push and pop +commands: + +.. code:: c++ + + // some include files ... + + #include + + // some code ... + + roctxRangePush("My Event"); + + // my event code ... + + roctxRangePop(); + roctxMarkA("My Event"); + +The string label is an optional argument to the annotation code. + +.. note:: At this time, current Re::Solve code does not have any profiling + annotations. If you want to profile it, you need to add annotations + to Re::Solve sources relevant to your profiling and rebuild the code. + +Once your instrumented code is built, it can be profiled by calling the +``rocprof`` tool like this: + +.. code:: shell + + rocprof --stats --hip-trace --roctx-trace -o out.csv ./my_executable.exe + +In this example + + * Flag ``-o`` specifies the output file in comma separated values format, + in this case ``out.csv``. + * File ``my_executable.exe`` is the binary being profiled. + * Flag ``--stats`` enables kernel execution stats. + * Flag ``--hip-trace`` includes HIP API timelines in profiling data. + * Flag ``--roctx-trace`` enables rocTX application code annotation trace. + +The profiler will create several files with name ``out`` but with different +extensions. To visualize output, one can upload the ``out.json`` file to +`Perfetto `_. + +When running ROCProfiler on a machine with a scheduler, it is a good idea +to write a profiling script. Here is an example for a SLURM scheduler: + +.. code:: shell + + #!/bin/bash + + #SBATCH -A CSC359 + #SBATCH -J resolve_test + #SBATCH -o %x-%j.out + #SBATCH -t 00:30:00 + #SBATCH -N 1 + + EXE=build/examples/klu_rocsolverrf_fgmres.exe + OUT=rocprof-resolve25k + ARGS="" + + echo "`date` Starting run" + srun -N 1 -n 1 -c 1 -G 1 \ + rocprof --stats --hip-trace --roctx-trace -o ${OUT}.csv \ + ${EXE} ${ARGS} + echo "`date` Finished run" + + + diff --git a/docs/sphinx/license.rst b/docs/sphinx/license.rst new file mode 100644 index 00000000..ba8dc9d6 --- /dev/null +++ b/docs/sphinx/license.rst @@ -0,0 +1,5 @@ +******* +License +******* + +.. include:: ../../LICENSE \ No newline at end of file diff --git a/docs/sphinx/licenses.rst b/docs/sphinx/licenses.rst deleted file mode 100644 index a1e3eb08..00000000 --- a/docs/sphinx/licenses.rst +++ /dev/null @@ -1,5 +0,0 @@ -******* -License -******* - -.. mdinclude:: ../../LICENSE \ No newline at end of file diff --git a/docs/sphinx/guide.rst b/docs/sphinx/user_guide/index.rst similarity index 72% rename from docs/sphinx/guide.rst rename to docs/sphinx/user_guide/index.rst index a3fe6da8..049ae386 100644 --- a/docs/sphinx/guide.rst +++ b/docs/sphinx/user_guide/index.rst @@ -3,12 +3,17 @@ User Guide Getting started --------------- -ReSolve is maintained and developed on the -`ReSolve Github Project `_. +ReSolve is maintained and developed at `Github `_. +It has following build dependencies: -Dependencies: - KLU, AMD and COLAMD libraries from SuiteSparse - CUDA >= 11.4 - CMake >= 3.22 +* C++11 compliant compiler +* CMake >= 3.22 +* KLU, AMD and COLAMD libraries from SuiteSparse >= 5.0 (optional) +* CUDA >= 11.4 (optional) +* HIP/ROCm >= 5.6 (optional) -To build it: + +To acquire and build Re::Solve it is as easy as: .. code:: shell @@ -17,6 +22,10 @@ To build it: $ cmake ../resolve $ make +Note if you don't include any of the optional dependencies, there will be +little functionality provided. You might want to use tool such as ``ccmake`` +to enable features you need. + To install the library ---------------------- @@ -28,7 +37,10 @@ In the directory where you built the library run To change the install location please use the CMAkePresets.json file as mentioned in `test and deploy <#test-and-deploy>`__ -To run it, download `test linear systems `__ and then edit script ```runResolve`` `__ to match locations of your linear systems and binary installation. The script will emulate nonlinear solver calling the linear solver repeatedly. +To run it, download `test linear systems `__ +and then edit script |runResolve|_ +to match locations of your linear systems and binary installation. +The script will emulate nonlinear solver calling the linear solver repeatedly. To use the ReSolve library in your own project ---------------------------------------------- @@ -119,31 +131,15 @@ call the preset flag in the cmake build step. cmake -B build --preset cluster -Developing Documentation Using Dev Container -------- - -Prerequisites - -#. install Docker Desktop and launch the app -#. install the "Remote Development" extension in VSCode -#. open your local clone of resolve in VSCode - -Build Container -The build info for this container is in `.devcontainer/`. There is a Dockerfile and json file associated with the configuration. -#. if connected, disconnect from the PNNL VPN -#. launch the container build - * `cmd shift p` to open the command pallette in vscode - * click `> Dev Container: rebuild and reopen container` - * this will start building the container, taking about 40 minutes - * click on the pop up with `(show log)` to view the progress - -#. Open new terminal within VSCODe and run the renderDocs.sh (note this takes a minute) -#. Open the link that was served to you after step 3 - -Note - pushing/pulling from git is not supported in a devcontainer, and should be done independently. +Getting Help +------------ For any questions or to report a bug please submit a `GitHub issue `__. + + +.. |runResolve| replace:: ``runResolve`` +.. _runResolve: https://github.com/ORNL/ReSolve/blob/develop/runResolve diff --git a/resolve/Doxygen.hpp b/resolve/Doxygen.hpp index df23b3cc..c3228b2d 100644 --- a/resolve/Doxygen.hpp +++ b/resolve/Doxygen.hpp @@ -9,7 +9,8 @@ * and direct linear solvers designed to run on NVIDIA and AMD GPUs, as well as * on CPU devices. This is the main page of source code documentation intended * for developers who want to contribute to ReSolve code. General documentation - * is available at readthedocs. + * is available at readthedocs. The + * ReSolve project is hosted on GitHub. * * * @section name_sec Name