diff --git a/Developing.md b/Developing.md index 2c51a59..0916088 100644 --- a/Developing.md +++ b/Developing.md @@ -31,22 +31,26 @@ running with https, you need to set `OAUTHLIB_INSECURE_TRANSPORT=1` in your shell. Command: + ```bash export OAUTHLIB_INSECURE_TRANSPORT=1 cray config get core.hostname ``` Output: + ```text http://localhost:8080 ``` Command: + ```bash cray mymodule things list ``` Output: + ```text [] ``` @@ -59,8 +63,8 @@ To test against a real hardware system, build the source as a wheel and install in a virtual environment on the target system. ```bash -pip install wheel -python setup.py bdist_wheel +python -m pip install build +python -m build --wheel scp ./dist/cray-.whl ssh python3 -m venv venv @@ -86,70 +90,142 @@ nox -s tests cover ## Installation for Development -For development, we recommend the container below or a virtualenv with python3: +For development, we recommend a virtualenv with python3: -```bash -virtualenv -p python3 cli -cd cli -source bin/activate -git clone https://github.com/Cray-HPE/craycli.git -cd craycli -python -m pip install . -``` +> ***NOTE*** It is recommended to install the highest version of Python supported by the application \(e.g. Python 3.10\) +> For MacOS users we recommend using [`pyenv`](https://github.com/pyenv/pyenv#homebrew-in-macos), which can be installed via +> [Homebrew (`brew`)](https://brew.sh/). Using `pyenv` enables installing every version of Python there is, and avoids +> interactions with the system Python. For help using `pyenv`, see [setting up python](#setting-up-python). -### Dev Container +### Setting up Python -We created a container that includes all dependencies required to run nox and -develop for the cli: +These steps are specific to MacOS. For other distros, please ask in the CrayCLI Slack channel. -```bash -git clone https://github.com/Cray-HPE/craycli.git -cd craycli -utils/devenv.sh -``` +[`pyenv`](https://github.com/pyenv/pyenv) is a Python version manager for MacOS, it allows users to +install multiple versions of Python. It is not recommended to use the system's Python, since it often requires root +privileges and is a fragile dependency of the OS itself. `pyenv` installs Python versions into the user directory on +the system, keeping the Python versions local to the user session. + +1. Install bew using the steps outlined on [Homebrew (`brew`)](https://brew.sh/)'s homepage. + +1. Install `pyenv`, using the directions on their MacOS page [here](https://github.com/pyenv/pyenv#homebrew-in-macos). + + > ***NOTE*** There are automatic installers mention on the linked page above. + +1. Open a new shell, and invoke `pyenv` to make sure it works. +1. Install Python 3.10 + + ```bash + # Find latest Python 3.10 + pyenv install -l | grep 3.10 + + pyenv install 3.10.10 # or w/e the latest one was + ``` + +1. Install `virtualenv` + + > ***NOTE*** Optionally you can run `pyenv global 3.10.10` and then `python -m pip install virtualenv`. The step below + > uses the installed path instead because not all users wnat to change their global Python version for their user account. + + ```bash + ~/.pyenv/versions/3.10.10/bin/python -m pip install virtualenv + ``` + +1. Create a `virtualenv` for `craycli` + + ```bash + mkdir -p ~/.virtualenvs + ~/.pyenv/versions/3.10.10/bin/python -m virtualenv ~/.virtualenvs/craycli + ``` + +### Installing craycli + +1. Load the `virtualenv` (the example below uses the virtualenv created in [setting up python](#setting-up-python)). + + ```bash + source ~/.virtualenvs/craycli/bin/activate + ``` + +1. Clone `craycli` (if it isn't already cloned somewhere) + + ```bash + git clone https://github.com/Cray-HPE/craycli.git + cd craycli + ``` + +1. Install `craycli` + + ```bash + python -m pip install . + cray --version + ``` ## Building -We are using pyinstaller to generate a binary and wrapping into an RPM. -This is integrated into the DST jenkins pipeline. +We are using `pyinstaller` to generate a binary, and then installing it on systems using an RPM. ## Bugs -If you find a bug in the craycli framework, feel free to open a bug in the -casmcloud project. We'll triage it within a day or two. +If you find a bug in the `craycli` framework, feel free to open a bug in the +CASMCLOUD project. We'll triage it within a day or two. [File Bug](https://github.com/Cray-HPE/craycli/issues/new) ## How to update your swagger Assuming you're using the .remote option for your module: (from the root of the forked project) +#### Convert to Swagger -__Generate the swagger__ +> ***NOTE*** Remember to activate your `virtualenv`, or create one following +> [installation for development](#installation-for-development). -``` -$> ./utils/devenv.sh -bash-4.4$ nox -s swagger -- my_service_name -nox > Running session swagger -nox > Creating virtualenv using python3.6 in /work/.nox/swagger -... -nox > Session swagger was successful. -``` +- Install CI tools. -__Run normal tests__ + ```bash + python -m pip install .[ci] + ``` -``` -bash-4.4$ nox -s tests -nox > Running session tests-2.7 -nox > Creating virtualenv using python2.7 in /work/.nox/tests-2-7 -nox > pip install --upgrade -r requirements-test.txt -nox > pip install --upgrade -e . -nox > py.test --quiet --cov=cray --cov=tests --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=95 tests -... [100%] -Required test coverage of 95% reached. Total coverage: 95.60% -460 passed in 110.78 seconds -nox > Session tests-2.7 was successful. -bash-4.4$ exit -$> -``` +- Generate swagger. + + ```bash + nox -s swagger -- my_service_name path/to/api/file + ``` + + Potential output: + + ```text + nox > Running session swagger + nox > Creating virtual environment (virtualenv) using python3 in .nox/swagger + nox > /bin/bash utils/convert.sh cray/modules/ims swagger3.json + ... + Wrote /Users/rusty/gitstuffs/cray-shasta/craycli/cray/modules/my_service_name/swagger3.json + nox > Session swagger was successful. + ``` + +#### Running `nox` (unit tests) + +- Install CI tools. + + ```bash + python -m pip install .[ci] + ``` + +- Run unit tests. + + ```bash + nox -s tests + ``` + + Potential output: -If they pass open the PR and we’ll merge it asap. + ```text + nox > Running session tests + nox > Creating virtual environment (virtualenv) using python3 in .nox/tests + nox > python -m pip install '.[test]' + nox > python -m pip install . + nox > pytest --quiet --cov=cray --cov-append --cov-config=.coveragerc --cov-report= --cov-fail-under=85 cray + ... [100%] + Required test coverage of 85% reached. Total coverage: 91.83% + 523 passed in 211.76s (0:03:31) + nox > Session tests was successful. + ``` diff --git a/Makefile b/Makefile index 9c6871e..b4da298 100644 --- a/Makefile +++ b/Makefile @@ -45,7 +45,7 @@ export VERSION := $(shell python3 -m setuptools_scm 2>/dev/null | tr -s '-' '~' endif ifeq ($(VERSION),) -$(error VERSION not set! Verify setuptools_scm[toml] is installed and try again.) +$(warning VERSION not set! Verify setuptools_scm[toml] is installed and try again.) endif ############################################################################# @@ -123,6 +123,7 @@ help: @echo ' help Show this help screen.' @echo ' clean Remove build files.' @echo + @echo ' image Build and publish the swagger testing image.' @echo ' rpm Build a YUM/SUSE RPM.' @echo ' all Build all production artifacts.' @echo @@ -132,6 +133,10 @@ help: @echo ' rpm_build Builds the RPM.' @echo ' rpm_build_source Builds the SRPM.' @echo ' rpm_package_source Creates the RPM source tarball.' + @echo + @echo ' image_login Logs into the Docker registry for pulling and publishing images.' + @echo ' image_build Builds the swagger testing image.' + @echo ' image_publish Builds and publishes the testing image.' @echo '' clean: @@ -172,3 +177,18 @@ rpm_build: snyk: $(MAKE) -s image | xargs --verbose -n 1 snyk container test + +############################################################################# +# RPM targets +############################################################################# + +image_login: + docker login artifactory.algol60.net + +image_build: + docker build -t artifactory.algol60.net/csm-docker/stable/craycli/swagger2openapi:latest utils/ + +image_publish: image_build + docker push artifactory.algol60.net/csm-docker/stable/craycli/swagger2openapi:latest + +image: image_publish diff --git a/README.md b/README.md index a5363cb..f7113fb 100644 --- a/README.md +++ b/README.md @@ -67,21 +67,21 @@ The version is derived from Git by the `setuptools_scm` Python module. ```bash # Format {tag}.post1.dev{distance}+{scm letter}{revision hash} - canu, version 1.6.28.post1.dev14+g818da8a + cray, version 1.6.28.post1.dev14+g818da8a ``` - **(unstable/dev)** No distance and not clean; the build came from a commit that has a git-tag, and the repository had uncommitted changes. ```bash # Format {tag}.dYYYYMMDD - canu, version 1.6.28.d20230123 + cray, version 1.6.28.d20230123 ``` - **(unstable/dev)** Distance and not clean; the build came from one or more commits after a git-tag, and the repository had uncommitted changes. ```bash # Format {tag}.post1.dev{distance}+{scm letter}{revision hash}.dYYYYMMDD - canu, version 1.6.28.post1.dev3+g3071655.d20230123 + cray, version 1.6.28.post1.dev3+g3071655.d20230123 ``` The `setuptools_scm` module is configured by `pyproject.toml`. diff --git a/cray/cli.py b/cray/cli.py index e89cd1c..4eabe77 100644 --- a/cray/cli.py +++ b/cray/cli.py @@ -27,7 +27,6 @@ import os import re -import sys import click from cray.auth import AuthUsername @@ -51,9 +50,12 @@ 'config': Config('', '', raise_err=False), 'token': None, 'auth': None - }, 'auto_envvar_prefix': NAME.upper() + }, + 'auto_envvar_prefix': NAME.upper(), + 'help_option_names': ['-h', '--help'], } +CONTEXT_SETTINGS = {} def rsa_required(config): """Get the value for 'auth.login.rsa_required' from the CLI @@ -84,6 +86,7 @@ def rsa_required(config): context_settings=CONTEXT_SETTING ) # pragma: NO COVER @click.pass_context +@click.version_option() def cli(ctx, *args, **kwargs): """ Cray management and workflow tool""" pass @@ -212,17 +215,3 @@ def cli_cb(ctx, result, **kwargs): # Use click echo instead of our logging because we always want to echo # our results click.echo(format_result(result, ctx.obj['globals'].get('format'))) - - -if getattr(sys, 'frozen', False): - version = None - if '--version' in sys.argv: - # Only bother opening the file if actually asking for version - path = os.path.join(os.path.dirname(__file__), 'build_version') - if os.path.isfile(path): - with open(path, encoding='utf-8') as v: - version = v.read() - click.version_option(version)(cli) - cli(sys.argv[1:]) -else: - click.version_option()(cli) diff --git a/noxfile.py b/noxfile.py index 3e651dc..101a167 100644 --- a/noxfile.py +++ b/noxfile.py @@ -59,7 +59,7 @@ def generate(session): """ if len(session.posargs) != 2: - msg = 'Usage: nox -s generate -- [module name] [stash link to swagger]' + msg = 'Usage: nox -s generate -- [module name] [path to API file]' global ERROR_ON_GENERATE if ERROR_ON_GENERATE: raise Exception(msg) @@ -82,7 +82,6 @@ def generate(session): is_local_file = os.path.exists(swagger_file) - if not os.path.exists(module_path): os.makedirs(module_path) if not os.path.isfile(init_file): @@ -107,7 +106,7 @@ def generate(session): convert_file(session, module_path, convert_file_name) -@nox.session(python=None) +@nox.session(python='3') def swagger(session): """Run each swagger file through the converter in case anything changed. This should be run before running unit tests""" @@ -130,7 +129,7 @@ def swagger(session): swagger_file = None remote_file = None for f in files: - if f.startswith('swagger.'): + if f.startswith('swagger'): swagger_file = f break if f == '.remote': diff --git a/utils/swagger2openapi.Dockerfile b/utils/Dockerfile similarity index 75% rename from utils/swagger2openapi.Dockerfile rename to utils/Dockerfile index f490857..8d4f0b1 100644 --- a/utils/swagger2openapi.Dockerfile +++ b/utils/Dockerfile @@ -1,18 +1,18 @@ # # MIT License -# -# (C) Copyright [2020-2022] Hewlett Packard Enterprise Development LP -# +# +# (C) Copyright 2020-2023 Hewlett Packard Enterprise Development LP +# # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), # to deal in the Software without restriction, including without limitation # the rights to use, copy, modify, merge, publish, distribute, sublicense, # and/or sell copies of the Software, and to permit persons to whom the # Software is furnished to do so, subject to the following conditions: -# +# # The above copyright notice and this permission notice shall be included # in all copies or substantial portions of the Software. -# +# # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL @@ -20,15 +20,15 @@ # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR # OTHER DEALINGS IN THE SOFTWARE. -FROM alpine:3.15 as base +FROM artifactory.algol60.net/docker.io/library/alpine:3.17 as base -RUN apk add --update npm +RUN apk add --update npm -RUN mkdir -p /converter +RUN mkdir -p /converter -WORKDIR /converter +WORKDIR /converter -RUN npm install -g swagger2openapi +RUN npm install -g 'swagger2openapi@~7.0.8' -USER 1000 -ENTRYPOINT ["swagger2openapi"] +USER 1000 +ENTRYPOINT ["swagger2openapi"] diff --git a/utils/convert.sh b/utils/convert.sh index c771c9b..6729596 100755 --- a/utils/convert.sh +++ b/utils/convert.sh @@ -1,19 +1,19 @@ #!/bin/bash # # MIT License -# -# (C) Copyright [2020] Hewlett Packard Enterprise Development LP -# +# +# (C) Copyright 2020-2023 Hewlett Packard Enterprise Development LP +# # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), # to deal in the Software without restriction, including without limitation # the rights to use, copy, modify, merge, publish, distribute, sublicense, # and/or sell copies of the Software, and to permit persons to whom the # Software is furnished to do so, subject to the following conditions: -# +# # The above copyright notice and this permission notice shall be included # in all copies or substantial portions of the Software. -# +# # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL @@ -43,6 +43,10 @@ if [ ! -z $APP ]; then $APP $OPTS -o $DIRNAME/swagger3.json cd $CUR else - docker pull arti.dev.cray.com/csm-internal-docker-stable-local/craycli/swagger2openapi:latest - docker run -v $DIRNAME:/converter arti.dev.cray.com/csm-internal-docker-stable-local/craycli/swagger2openapi:latest $OPTS -o swagger3.json + docker pull artifactory.algol60.net/csm-docker/stable/craycli/swagger2openapi:latest + docker run \ + -v $DIRNAME:/converter \ + artifactory.algol60.net/csm-docker/stable/craycli/swagger2openapi:latest \ + $OPTS -o swagger3.json + echo "Wrote $DIRNAME/swagger3.json" fi diff --git a/utils/craycli-devenv.Dockerfile b/utils/craycli-devenv.Dockerfile deleted file mode 100644 index 5dd3624..0000000 --- a/utils/craycli-devenv.Dockerfile +++ /dev/null @@ -1,19 +0,0 @@ -FROM alpine:3.15 as app - -RUN apk add --no-cache bash -RUN apk add --no-cache python3 -RUN apk add --no-cache python3-dev -RUN apk add --no-cache py3-pip -RUN apk add --no-cache build-base -RUN apk add --no-cache docker -RUN apk add --no-cache --repository http://nl.alpinelinux.org/alpine/edge/main libuv -RUN apk add --no-cache --repository http://dl-cdn.alpinelinux.org/alpine/edge/community nodejs-current-npm - -RUN pip3 install --upgrade pip -RUN pip3 install nox - -RUN npm install -g swagger2openapi - -RUN mkdir -p /work -VOLUME /work -WORKDIR /work diff --git a/utils/devenv.sh b/utils/devenv.sh deleted file mode 100755 index afd8262..0000000 --- a/utils/devenv.sh +++ /dev/null @@ -1,33 +0,0 @@ -# -# MIT License -# -# (C) Copyright [2020] Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. - -DIR=${1:-$PWD} -CUR=$PWD -IMAGE="arti.dev.cray.com/csm-internal-docker-stable-local/craycli/craycli-devenv:latest" - -cd $DIR -find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf -docker pull $IMAGE -docker run -u ${UID}:${GID} -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/work -it $IMAGE /bin/bash -cd $CUR -