Make the Python3 pipeline work on Kubernetes

[marcomicera]

Prerequisites

Basic understanding of how Quartermaster works (see this presentation).

Problem

The Python3 pipeline is supposed to:

1. git clone and build a Python3 repository;
2. detect its licenses;
3. report its findings back.

Currently, the pipeline for Python3 is broken when being run locally, and it has never worked on Kubernetes.

Goal

Make it work on Kubernetes.
Personally, I wouldn't make the effort to try to make it run locally, but I would concentrate on making it run in Docker containers, as we ultimately want to deploy Quartermaster on Kubernetes.

A possible workflow

See how the pipeline for Maven works on Kubernetes

1. Start by deploying Quartermaster for Maven on Kubernetes.
   Worth mentioning:
   • The main Quartermaster Job manifest.
   • You can inspect what's been deployed by running:

     kubectl kustomize qmstr

     from the deploy folder.
2. Observe its results by inspecting the DGraph database instance's content.
   • The database schema is defined here as a proto definition file.
     • Might be useful to generate a diagram out of it with protodot.
     • Detected licenses should be contained in DataNodes, having:
       • type = license, and
       • data = the detected license.

Inspect the Python3 local demos

The demos submodule contains attempts of license detection of all sorts of projects, including C ones.
This paragraph contains instructions on how to run these demos locally:

Running the demos is Makefile based.
Running just make it will compile all the projects located in the demos/ directory with Quartermaster:

$ make
...

Should you want to compile a specific project run make followed by the name of the project you want to compile e.g.:

$ make curl
...

For more information and updates you can visit the Quartermaster project.

This is a tedious, but maybe necessary step

Several problems arise when trying to run those locally, though: these demos require you to build the qmstr/master (huge and inefficient) image via the Quartermaster's main Makefile (last step):

$ make --dry-run
echo /usr/local/bin/docker
go test ./clients/qmstrctl
go build  -ldflags "-X github.com/QMSTR/qmstr/lib/go-qmstr/cli.QmstrVersion=0.5" -o out/qmstrctl ./clients/qmstrctl
go test ./clients/qmstr
go build -o out/qmstr ./clients/qmstr
go test ./modules/builders/qmstr-wrapper
go build -o out/builders/qmstr-wrapper ./modules/builders/qmstr-wrapper
docker build -f masterserver/Dockerfile -t qmstr/master --target master  --build-arg QMSTR_VERSION=0.5 --build-arg GRPCIO_VERSION=1.20.0 --build-arg DGRAPH_VERSION=1.1.0 --build-arg HUGO_VERSION=0.39 --build-arg SCANCODE_VERSION=3.1.1 --build-arg GO_VERSION=1.12.4 .

Against all good practices, its (huge) Dockerfile masterserver/Dockerfile contains several big components inside, such as the DGraph database itself: this makes the image build take forever to complete.
I'm afraid, though, that this step is necessary to be able to run these demos in a local environment.

That big image is just for demos

Please note that in production, however, a complete different set of images is being used, which reside under the docker folder.

How the Kubernetes version should look like

The qmstr-client is the image that is in charge of building the project (note the mvn package command):

qmstr/deploy/qmstr/base/job.yaml

Lines 25 to 37 in 77f2904

	- name: qmstr-client
	image: endocodeci/qmstr-client-mvn
	command: [ "sh", "-c", "mvn package -Dmaven.test.skip=true && qmstrctl analyze --verbose" ]
	env:
	- name: QMSTRADDRENV
	value: "localhost:50051"
	- name: BUILDROOT
	value: /var/qmstr/buildroot
	- name: QMSTR_MASTER
	value: "localhost:50051"
	volumeMounts:
	- name: buildroot-volume
	mountPath: /var/qmstr/buildroot

This means that it is also language-dependent (note its name endocodeci/qmstr-client-mvn, which should really just be a tag in my opinion, i.e., endocodeci/qmstr-client:mvn), as the Maven version should contain the mvn binary, the Python3 version the python3 binary, and so on.
For that matter, the qmstr-client Dockerfile should contain a target specifically for Python3 projects (this is the Maven target for reference).

How does the build system interact with Quartermaster

As you can see from this example, the build command python3 setup.py qmstr is wrapped by the qmstr binary, that makes this communication possible (the qmstrctl --verbose spawn qmstr/python-pillowdemo part only spawns a container locally and runs all the commands that follow inside it).

The build command python3 setup.py qmstr also contains a last qmstr keyword at the end: I forgot what was that about, but I believe it is essential for Quartermaster to be able to fetch build information.

Please note that this does not happen with Maven:

qmstr/deploy/qmstr/base/job.yaml

Line 27 in 77f2904

command: [ "sh", "-c", "mvn package -Dmaven.test.skip=true && qmstrctl analyze --verbose" ]

qmstr does not wrap the mvn package command, as the pom.xml file is being previously patched by an initContainer in order for Maven to be able to send build information to Quartermaster.

Other useful information

• Docker images are being built, pushed and deployed through this GitHub Action.