From fd305a85dca2b061c95e83fd5b901c8ef4b73d84 Mon Sep 17 00:00:00 2001 From: Jan Janssen Date: Wed, 1 May 2024 15:59:45 -0600 Subject: [PATCH] Use jupyter book for documentation --- .readthedocs.yml | 10 +++- README.md | 4 +- binder/postBuild | 9 ++- docs/Makefile | 20 ------- docs/_config.yml | 14 +++++ docs/_static/pyiron_logo.ico | Bin 21238 -> 0 bytes docs/_toc.yml | 9 +++ docs/{source => }/advanced.md | 0 docs/{source => }/command.md | 0 docs/{source => }/debug.md | 0 docs/{_static => images}/pyiron-logo.png | Bin docs/{source => }/installation.md | 0 docs/make.bat | 35 ----------- docs/{source => }/queue.md | 0 docs/source/conf.py | 35 ----------- docs/source/index.rst | 41 ------------- docs/source/python.md | 71 ----------------------- 17 files changed, 42 insertions(+), 206 deletions(-) delete mode 100644 docs/Makefile create mode 100644 docs/_config.yml delete mode 100644 docs/_static/pyiron_logo.ico create mode 100644 docs/_toc.yml rename docs/{source => }/advanced.md (100%) rename docs/{source => }/command.md (100%) rename docs/{source => }/debug.md (100%) rename docs/{_static => images}/pyiron-logo.png (100%) rename docs/{source => }/installation.md (100%) delete mode 100644 docs/make.bat rename docs/{source => }/queue.md (100%) delete mode 100644 docs/source/conf.py delete mode 100644 docs/source/index.rst delete mode 100644 docs/source/python.md diff --git a/.readthedocs.yml b/.readthedocs.yml index c2d96ee1..10a2a19f 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -9,10 +9,16 @@ build: os: "ubuntu-22.04" tools: python: "mambaforge-22.9" + jobs: + pre_build: + # Generate the Sphinx configuration for this Jupyter Book so it builds. + - "cp README.md docs" + - "cp notebooks/*.ipynb docs" + - "jupyter-book config sphinx docs/" # Build documentation in the docs/ directory with Sphinx sphinx: - configuration: docs/source/conf.py + builder: html # Optionally build your docs in additional formats such as PDF and ePub formats: [] @@ -26,3 +32,5 @@ python: install: - method: pip path: . + extra_requirements: + - sphinx \ No newline at end of file diff --git a/README.md b/README.md index bb689c63..e6017142 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ -# pysqa - a simple queue adapter for python +# pysqa [![Python package](https://github.com/pyiron/pysqa/workflows/Python%20package/badge.svg)](https://github.com/pyiron/pysqa/actions) [![Documentation Status](https://readthedocs.org/projects/pysqa/badge/?version=latest)](https://pysqa.readthedocs.io/en/latest/?badge=latest) [![Coverage Status](https://coveralls.io/repos/github/pyiron/pysqa/badge.svg?branch=main)](https://coveralls.io/github/pyiron/pysqa?branch=main) -[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/pysqa/HEAD?labpath=notebooks%2Fexample.ipynb) +[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/pysqa/HEAD?labpath=example.ipynb) High-performance computing (HPC) does not have to be hard. In this context the aim of `pysqa` is to simplify the submission of calculation to an HPC cluster as easy as starting another subprocess locally. This is achieved based on the assumption that even though modern HPC queuing systems offer a wide range of different configuration options, most users submit the majority of their jobs with very similar parameters. diff --git a/binder/postBuild b/binder/postBuild index a8681249..c335ad04 100644 --- a/binder/postBuild +++ b/binder/postBuild @@ -3,4 +3,11 @@ mkdir -p /home/jovyan/.local/share/jupyter/kernels/flux cp binder/kernel.json /home/jovyan/.local/share/jupyter/kernels/flux # install pympipool -pip install . --no-deps --no-build-isolation \ No newline at end of file +pip install . --no-deps --no-build-isolation + +# copy notebooks +mv notebooks/*.ipynb . +rm -rf notebooks + +# clean up +rm -rf .ci_support .github binder docs notebooks pysqa pysqa.egg-info tests .coveralls.yml .gitignore .readthedocs.yml CODE_OF_CONDUCT.md LICENSE MANIFEST.in README.md pyproject.toml setup.py build \ No newline at end of file diff --git a/docs/Makefile b/docs/Makefile deleted file mode 100644 index d0c3cbf1..00000000 --- a/docs/Makefile +++ /dev/null @@ -1,20 +0,0 @@ -# Minimal makefile for Sphinx documentation -# - -# You can set these variables from the command line, and also -# from the environment for the first two. -SPHINXOPTS ?= -SPHINXBUILD ?= sphinx-build -SOURCEDIR = source -BUILDDIR = build - -# Put it first so that "make" without argument is like "make help". -help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -.PHONY: help Makefile - -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/_config.yml b/docs/_config.yml new file mode 100644 index 00000000..22ee0346 --- /dev/null +++ b/docs/_config.yml @@ -0,0 +1,14 @@ +title: pysqa +author: Jan Janssen +logo: images/pyiron-logo.png + +execute: + execute_notebooks : off + +repository: + url : https://github.com/pyiron/pysqa + path_to_book : "" + +launch_buttons: + notebook_interface : jupyterlab + binderhub_url : https://mybinder.org \ No newline at end of file diff --git a/docs/_static/pyiron_logo.ico b/docs/_static/pyiron_logo.ico deleted file mode 100644 index f5270a0c8742724905f1bd7947fe7b8c7003f5a4..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 21238 zcmeI44{TLcp2uH-MMGF5hPcKMXjnpsF{baey{&|;4U4QYEFvOQMP-pe>^MXbu>6r$ z2aQ1l#${Q;ILIn8NYHVdu5DOG5gA2g5E&QI09GW3$gs5_md>MRKi~WAd2`?Gdwuue ztFvx=$*;fj=XZX8&iS4H=Qa{4h;)k-7Dm8@k&S&Lks*;tq_Q#*A37ou*-hF|(jbYD z?9zck{l-P3u^Xb%qEN(LKs2tXXfg8Zx`1^kmN$6v(V;KUONC5hSvR1Euw~NT=qN22 z_h>EwyJg&R4Q@G?PWdWh%V)x#{U0Oy1K7oPI=vQh(|8pXk~ccL3^a2_VS)JvzH z%T59VBp*D|G*EvH4rQIJ@?}4L;23i61zsh-3K&lK#4oCg$HVUgd@ch^{!Y{A)L_>Y z)c;Z9vw>&G-vaLD$rs&X;4pMq+aEa&Fbb#vW&k~bF@WRQH!>m58I8;d@Lfo}7jQZ8 zMaZsJJVEP7-?2a)X?f|>@R1+O#|BGAzJAnsH*8!)`!AHuo$zK^){}O*I2*0y z1D$+n4g4qAG(RR!^0xs7zPq8j!k@3ObiA>+c=BWS-~Y~YD_6Gt=GkY>hE=QLOJ>jB zbz@1%?cxzELs}0&zH|mKz>`558z-+9up0TI6a6g7wE5#>vB~dl-)`F5+f(3=@4jnx zz43;*skC&er4@$H5bW3iAMtAifw z`70ahN;KxVuztbyDf0cc2g7ePHmdE01$pKYpHLiKc;+`6Hxu%oo%@gG+_`h1bU54E zYO1cfYH=>|g0w%mCi;|>EqvgYrDomZkC;OrePoV)`l9}*jm4Qg_}1!X2+5e^Xeld=6Ar>#U-Zkp%SxtPL=u7@#86a%)RZl4Y|k* z(%w=YeJuW;f2seVF8=s8FFR)a6A!1zzw1XoS{>w>Db9GVIcYWadn9~!! zsj}-AO*Cz7ZKkEU*(_hM;3)Dh_REELZ=ed=mw;9Pxy~k)O+rK29&~c+fbnZ2JT@`@ ztOc)}@(m{c*GWI6rBxG(RUK2=D9ZaF@ zY#6FHmt*76Km5GZeDN#Cs4h;R&wu|ri;%40X_p`~yK6((m=l_jvZCPAY(D=%Y5O2CfDA1ZgRM8M3u5wDx+~ z^&5BXZp%PZ3B*%r9pX+3ou$u)u}$mX9jW{hdYt~Xvy8TJ*Uo-E(AK4)SHJA-rwbFt zcDoJ_+P;-}MRW4rARn*H=2RV=%}Ni_hKW<|O6pJhCaZ(it(PFXH*iTh9?;4cwt8jO z(0*ChOX>L7H006moptbQA{=TyhOQ?t05~tZt?V$^@H^!}^T9ViIrYqLto7=*zG@ygFg~Tv)$N77YfYrUBp_ni*&bR@+VV(2>jjZW)1!sG zE9<88seHY8;6tw))4j>GcWVc8GJRtwv)Uwe>4UfKz8f6D{8Gdt=e=@pp5qON4>lfkIynMp3?qqTAthA z>+eVWIw_3p8VA)c4*TMI%`@BZ)ExdGeUw!zLq((D-M-W`$G zpR^+$?ThhO9No{z*LTyWZNGKx#}4`W6|x&}$4~GYm%h>2PS`zTVl3;4jjIxNIlIB` zxIn&Vf7t=T?im4iT%vs0al*;*Lv?t*KWy=j|MGzo*uD#B_4>tL(lizdb^(TYQs?VB z6B8T;)F#uV1K~vfciYlEhtQp#dp1TrT}&H(AbQ|5W$<}NwkUoQNcsg~+yBM;IPeZ> znh!<0?tJYHmctJ1fA#>fZ45Sl0M?kVJBamwyWaS979ct1p_kf3bJ!+)o3?f) z65a!-O?97#{(rgmLifJFlcd{uP-CdhDu(!VBY!qPy_G!-G!o`a$32hI9OC+ppFUq< z?AAG8FWTSr=Un)rXNpfM4)mf(`2dpXRHjfaWT#O-=*un$gl^ z!_aqwkH&ku=Mg>*j0bYol`=XTo&{)bIB}EXn5m;inQ7NvYpR9~6Ymy4bH`ldraeE- zrnB-XKMdFp`1RI&+~G_<8$a}r@1G3vBEFWqzS+tv-zd%ptlC6(YG2oq3ao1%}4$=ckP$&lBYLMmxa+cNJF_n z@O{VQldMzXTK6T_w83&V4&6P~*I#cAA2^V_zYN}|2jhni9yBTg|5=t#XBfI+=$O*x zK7Ch_pYt7yUmoQzg#QX~(tbD0c?ft3vO4TlRGz#(ylH8XJwY3TaozoIT)o-_g?0Zv zsl2=ux$e5yua6Cr7wqSL{d7k=8VKz}mTx8uFYR^H^#`pDJIv{%jjFlhj;yw;F806w zep2`Sd-k|xmd~HBw0qL&LweeNPP+Eun}L*lS~?!*lLr4mw6}ZaAz3~@(_UaU{Wdwb zkw5Nd_uP}{)MrJ*#{*0 zLs5~&*p)yHpf>0Mj92?-vse8+9Xz#UiF{f7dVuvNZ!+O~fz9}to!d1=%o0sk+M0h|f*y8gL%+_arN8w&L9>)^h9X5N%3>K9KTw@dqZn7TuU!Bu6^ z)+5iRzSm*@ZFWA9us@ss=bHcZsZ-|1*Ig%Hs0^cHSJI%v3f{-){G|hQmHn!(-#+)8 z#)geuX6`C-e6R9>>pQn!exUxLyP$ua zR%~i-ikU?dV&=gcV`ky_m|1W`%*+Sojf6J#(`YZ_OB^5(?`V2mKR>=bl9k4?Zi58Ojv#MO^5x1weGQfJ)gK~U8%XP zVz~R>I8$GkGkLPwKkFGpzU{XBklmAYZYys-yQB`5+!iwpn>Jku=YdlNb z*>+q5HOCR`u+(X8+`SA3-P(YR>0OxJ#sso?PIsz zRoMH+&TP|N{LpOmP5$7^La}v9I#*a zQC09hA(=jTlB<2-WWefU^M|9OTDSbSNs4PAPX{#FO@WwwPPXqoI8uo&Bl@_{#sL;scuh6ZKzIBwu8~ zR5m}f?TJ3WNIX?XszGCw<_AHic`5~~Fv981XK>jG(-r57{+eWQxL&`Ag?<fZNy8>c^Qib@Pt) zeJ%9;FuX;hy3*V~u{7F3y$(&s+omDE2Q(wlsSvR5g+j@K{!z<&De>mokt4hs(G2XAKE;f2M zNSWP$WBGK3(L-+ocZJd0N3)f39ekFwLDa(;A6>?}gf@9M)0uwC!xs6=RA4o*1=t51 z!9R|HwPx5!J8lHeCr#r^*18p6_l;EIjdVsm)7WwZ-7~)BAa6bprVr`8S+f5~pYQsI z{OQy|7+pOYr;Re5ek;eAzR_Mjt-`8X$@7COYwto%E#xtM!=%x3*VRUQ3m-QBI9KP?a+rXj5 zZus-2H2t^E{<>$(_yF3J^MribnvNIaP9JpG;MXHv7`>9dPTOmrIuRUdevw?tUjpAw z_7e{OtEepNn3B$hZ<~f(^#k?OOmlX|^ z%fa`PfA3YFEd|%zSzgq?yD=+1UtL~tK~gM%_q>S3ZZv+iP-|^`plKRtnrquHFm)H0 zZMCMmiF7x0@N170w8O7yQKTRazvhZaLGwV0^obNSQMi2oaq-v|iTtf_U4(-CXcOUf zF#l>HU-+E(!ml~O|I=-?uDpV}+R8}eqV|HiiUE;GbG)E#ILoA_nu5AZ74KWH?ZS#k zBwhqw#G-aeLDS!0m)Ta(l(4gVb9dPvxu8|Ln#e`15m)Dd@kp&P@yOq`N3ItBdk^1b Z6mN&9j(pYAHjv@JM0_gznf|2w{ukjmYlQ#+ diff --git a/docs/_toc.yml b/docs/_toc.yml new file mode 100644 index 00000000..26b947cd --- /dev/null +++ b/docs/_toc.yml @@ -0,0 +1,9 @@ +format: jb-book +root: README +chapters: +- file: installation.md +- file: queue.md +- file: example.ipynb +- file: command.md +- file: advanced.md +- file: debug.md \ No newline at end of file diff --git a/docs/source/advanced.md b/docs/advanced.md similarity index 100% rename from docs/source/advanced.md rename to docs/advanced.md diff --git a/docs/source/command.md b/docs/command.md similarity index 100% rename from docs/source/command.md rename to docs/command.md diff --git a/docs/source/debug.md b/docs/debug.md similarity index 100% rename from docs/source/debug.md rename to docs/debug.md diff --git a/docs/_static/pyiron-logo.png b/docs/images/pyiron-logo.png similarity index 100% rename from docs/_static/pyiron-logo.png rename to docs/images/pyiron-logo.png diff --git a/docs/source/installation.md b/docs/installation.md similarity index 100% rename from docs/source/installation.md rename to docs/installation.md diff --git a/docs/make.bat b/docs/make.bat deleted file mode 100644 index dc1312ab..00000000 --- a/docs/make.bat +++ /dev/null @@ -1,35 +0,0 @@ -@ECHO OFF - -pushd %~dp0 - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set SOURCEDIR=source -set BUILDDIR=build - -%SPHINXBUILD% >NUL 2>NUL -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.https://www.sphinx-doc.org/ - exit /b 1 -) - -if "%1" == "" goto help - -%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% -goto end - -:help -%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% - -:end -popd diff --git a/docs/source/queue.md b/docs/queue.md similarity index 100% rename from docs/source/queue.md rename to docs/queue.md diff --git a/docs/source/conf.py b/docs/source/conf.py deleted file mode 100644 index 9b8bd7ab..00000000 --- a/docs/source/conf.py +++ /dev/null @@ -1,35 +0,0 @@ -# Configuration file for the Sphinx documentation builder. -# -# For the full list of built-in configuration values, see the documentation: -# https://www.sphinx-doc.org/en/master/usage/configuration.html - -# -- Project information ----------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information - -project = 'pysqa' -copyright = '2023, Jan Janssen' -author = 'Jan Janssen' -release = '0.0.22' - -# -- General configuration --------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration - -extensions = ["myst_parser"] - -templates_path = ['_templates'] -exclude_patterns = [] - - - -# -- Options for HTML output ------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output - -try: - import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_logo = "../_static/pyiron-logo.png" - html_favicon = "../_static/pyiron_logo.ico" -except ImportError: - html_theme = 'alabaster' - -html_static_path = ['_static'] diff --git a/docs/source/index.rst b/docs/source/index.rst deleted file mode 100644 index 046fd046..00000000 --- a/docs/source/index.rst +++ /dev/null @@ -1,41 +0,0 @@ -========================================= -pysqa - a simple queue adapter for python -========================================= - -:Author: Jan Janssen -:Contact: janssen@mpie.de - -High-performance computing (HPC) does not have to be hard. In this context the aim of pysqa is to simplify the submission of calculation to an HPC cluster as easy as starting another subprocess locally. This is achieved based on the assumption that even though modern HPC queuing systems offer a wide range of different configuration options, most users submit the majority of their jobs with very similar parameters. - -Therefore, in pysqa users define submission script templates once and reuse them to submit many different calculations or workflows. These templates are defined in the jinja2 template language, so current submission scripts can be easily extended to templates. In addition to the submission of new jobs to the queuing system pysqa also allows the users to track the progress of their jobs, delete them or enable reservations using the built-in functionality of the queuing system. - -Features --------- -The core feature of pysqa is the communication to an HPC queuing system. This includes: - -* Submission of new calculation to the queuing system. -* List of calculation currently waiting or running on the queuing system. -* Deleting calculation which are currently waiting or running on the queuing system. -* List of available queue templates created by the user. -* Restriction of templates to a specific number of cores, run time or other computing resources. With integrated checks if a given calculation follows these restrictions. - -In addition to these core features, pysqa is continously extended to support more usecases for a larger group of users. These new features include the support for remote queuing systems: - -* Remote connection via the secure shell protocol to access remote HPC clusters. -* Transfer of file to and from remote HPC clusters, based on a predefined mapping of the remote file system into the local file system. -* Support for both individual connections as well as continous connections depending on the network availability. - -Finally, there is current work in progress to support a combination of multiple local and remote queuing systems from within pysqa, which are represented to the user as a single resource. - -Documentation -------------- - -.. toctree:: - :maxdepth: 2 - - installation - queue - python - command - advanced - debug diff --git a/docs/source/python.md b/docs/source/python.md deleted file mode 100644 index 63ea03eb..00000000 --- a/docs/source/python.md +++ /dev/null @@ -1,71 +0,0 @@ -# Python Interface -The `pysqa` package primarily defines one class, that is the `QueueAdapter`. It loads the configuration from a configuration directory, initializes the corrsponding adapter for the specific queuing system and provides a high level interface for users to interact with the queuing system. The `QueueAdapter` can be imported using: -``` -from pysqa import QueueAdapter -``` -After the initial import the class is initialized using the configuration directory specificed by the `directory` parameter which defaults to `"~/.queues"`: -``` -qa = QueueAdapter(directory="~/.queues") -``` -Another optional parameter of the `QueueAdapter` class is the `execute_command`, still this is primarily used for testing purposes to call the underlying shell commands. - -## List available queues -List available queues as list of queue names: -``` -qa.queue_list -``` -List available queues in an pandas dataframe: -``` -qa.queue_view -``` - -## Submit job to queue -Submit a job to the queue - if no queue is specified it is submitted to the default queue defined in the queue configuration: -``` -qa.submit_job( - queue=None, - job_name=None, - working_directory=None, - cores=None, - memory_max=None, - run_time_max=None, - dependency_list=None, - command=‘python test.py’, - **kwargs -) -``` -The only required parameter is: -* `command` the command that is executed as part of the job - -Additional options for the submission of the job are: -* `queue` the queue the job is submitted to. If this option is not defined the `primary_queue` defined in the configuration is used. -* `job_name` the name of the job submitted to the queuing system. -* `working_directory` the working directory the job submitted to the queuing system is executed in. -* `cores` the number of cores used for the calculation. If the cores are not defined the minimum number of cores defined for the selected queue are used. -* `memory_max` the memory used for the calculation. -* `run_time_max` the run time for the calculation. If the run time is not defined the maximum run time defined for the selected queue is used. -* `dependency_list` other jobs the calculation depends on. -* `**kwargs` allows writing additional parameters to the job submission script if they are available in the corresponding template. - -## Show jobs in queue -Get status of all jobs currently handled by the queuing system: -``` -qa.get_queue_status() -``` -With the additional parameter `user` a specific user can be defined to only list the jobs of this specific user. - -In analogy the jobs of the current user can be listed with: -``` -qa.get_status_of_my_jobs() -``` - -Finally, the status of a specific job with the queue id `1234` can be received from the queuing system using: -``` -qa.get_status_of_job(process_id=1234) -``` - -## Delete job from queue -Delete a job with the queue id `1234` from the queuing system: -``` -qa.delete_job(process_id=1234) -``` \ No newline at end of file