From 0e0335e89ef7992cc3c6126a5c172eea35da1ae3 Mon Sep 17 00:00:00 2001 From: Kunal Mehta Date: Thu, 8 Feb 2024 13:16:18 -0500 Subject: [PATCH] Fix building of wheels in monorepo and update docs The main issue is that our previous hack to detect the component name from the folder no longer works, since it'll now be an unprefixed "client", so have the Makefile add the prefixing in. Documentation has been updated as well. --- Makefile | 4 ++-- README.md | 31 +++++++++++++++---------------- 2 files changed, 17 insertions(+), 18 deletions(-) diff --git a/Makefile b/Makefile index b307d33a..9af02afa 100644 --- a/Makefile +++ b/Makefile @@ -57,9 +57,9 @@ requirements: ## Creates requirements files for the Python projects .PHONY: build-wheels build-wheels: ## Builds the wheels and adds them to the wheels subdirectory - ./scripts/verify-sha256sum-signature $$(basename ${PKG_DIR}) + ./scripts/verify-sha256sum-signature securedrop-$$(basename ${PKG_DIR}) ./scripts/build-sync-wheels - ./scripts/sync-sha256sums $$(basename ${PKG_DIR}) + ./scripts/sync-sha256sums securedrop-$$(basename ${PKG_DIR}) @echo Done! .PHONY: test diff --git a/README.md b/README.md index cd005683..e51c3b0a 100644 --- a/README.md +++ b/README.md @@ -66,8 +66,8 @@ _If you don't need to deploy a package and just want to test locally, you can st ## Updating our bootstrapped build tools -We use [build](https://pypa-build.readthedocs.io/en/latest/) toolchain to build our reproducible wheels. -If we have to update the tool, use the following steps +We use the [build](https://pypa-build.readthedocs.io/en/latest/) toolchain to build our reproducible wheels. +If we have to update the tools, use the following steps ```shell # Ensure you are running in a cleanly boostrapped virtual environment @@ -95,35 +95,31 @@ Make sure that your GPG public key is stored in `pubkeys/`, so CI can verify the ## Updating Python wheels -Maintainers of `securedrop-client` and `securedrop-proxy` must ensure that -the requirements files which are used for build of these packages (`build-requirements.txt`) -using `make requirements` are kept up to date in latest `main` of those repositories. - -If new dependencies were added in the `build-requirements.txt` of that -repo that are not in the `wheels` subdirectory for the package in this repository, -then the maintainer needs to do the following (we are taking `securedrop-client` project -as an example): +When adding a new production dependency to a component, new wheels will need to be built +plus updates to `build-requirements.txt`. This should be done after you have updated the +dependencies in the component's `pyproject.toml` and `poetry.lock` files. ### 0. Enable the virtualenv -You can create a fresh virtualenv and install the build tools from our bootstrapped wheels. +Create a fresh virtualenv and install the build tools from our bootstrapped wheels. ```shell rm -rf .venv make install-deps ``` -Remember that the following steps needs to be done from the same virtual environment. +The following steps needs to be done from the same virtual environment. ### 1. Create updated build-requirements.txt for the project -From the `securedrop-builder` directory, +From the `securedrop-builder` directory, run the following, where `` +is what you're trying to update dependencies for, e.g. "client", "proxy", etc. ```shell -PKG_DIR=/home/user/code/securedrop-client make requirements +PKG_DIR=/home/user/code/securedrop-client/ make requirements ``` -This will create the proper `build-requirements.txt` file in the project directory along with the binary wheel +This will create/update the `build-requirements.txt` file in the project directory along with the binary wheel hashes from our own Python package index server. If we are missing any wheels from our cache/build/server, it will let you know with a following message. @@ -151,7 +147,10 @@ PKG_DIR=/home/user/code/securedrop-client make build-wheels This above command will let you know about any new wheels + sources. It will build/download sources from PyPI (by verifying it against the sha256sums from -the `requirements.txt` of the project). +the `poetry.lock` of the project). + +If your package contains compiled code (e.g. C or Rust extensions), it must be +built for all Debian versions we support. ### 3. Commit changes to the wheels directory (if only any update of wheels)