Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug 1597598 - use taskgraph #226

Merged
merged 37 commits into from
Jun 24, 2020
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
cc6790f
bug 1597598 - use taskgraph
escapewindow Jun 2, 2020
181f03b
add pr kind
escapewindow Jun 5, 2020
a2e6876
Merge branch 'master' into taskgraph
escapewindow Jun 9, 2020
6a8f6fb
Merge branch 'master' into taskgraph
escapewindow Jun 10, 2020
d87c28c
address review comments
escapewindow Jun 15, 2020
cd9b912
Merge branch 'master' into taskgraph
escapewindow Jun 15, 2020
ca84d5d
add tox to pr kind-dependencies
escapewindow Jun 15, 2020
8b762c8
dict comprehension
escapewindow Jun 15, 2020
fef184b
don't pre-hash digest-extra; don't set empty `cached_task` attribute
escapewindow Jun 15, 2020
043b93e
digest-paths, using hash_paths
escapewindow Jun 15, 2020
32e55a0
digest-paths -> resources
escapewindow Jun 15, 2020
846eab3
script_name transform
escapewindow Jun 15, 2020
bf2a458
pushsnapscript - shared setup script
escapewindow Jun 15, 2020
e5fbab1
env.update
escapewindow Jun 15, 2020
fad9aed
pushflatpakscript - shared setup script
escapewindow Jun 16, 2020
8a1d21e
pushapkscript - shared setup script
escapewindow Jun 16, 2020
379a31e
_resolve_replace_string
escapewindow Jun 16, 2020
4a2d89e
add parameters. still need to add target tasks method
escapewindow Jun 16, 2020
4c67bec
cache type
escapewindow Jun 16, 2020
dfdd56b
label
escapewindow Jun 16, 2020
71ae5e9
remove tox artifacts dir for now
escapewindow Jun 16, 2020
8b0f04c
explicit docker-image dep for tox
escapewindow Jun 16, 2020
0d85a84
target tasks
escapewindow Jun 16, 2020
a69a77c
parameters['docker_tag']
escapewindow Jun 16, 2020
21d07e2
docker-image py3{7,8} -> python3{7,8}
escapewindow Jun 16, 2020
fae0314
push_docker_image param
escapewindow Jun 17, 2020
fcfbafe
Update taskcluster/docker/k8s-image/Dockerfile
escapewindow Jun 18, 2020
da91408
add comment to pushsnapscript/Dockerfile
escapewindow Jun 18, 2020
e9eaf62
rm chmod from pushsnapscript tox Dockerfile
escapewindow Jun 18, 2020
82815d3
Update taskcluster/scriptworker_taskgraph/parameters.py
escapewindow Jun 18, 2020
e094802
remove unused repo_name
escapewindow Jun 18, 2020
aaa9d7e
address resources review comments
escapewindow Jun 18, 2020
bb29494
fix
escapewindow Jun 18, 2020
f5dbbe7
adjust checks
escapewindow Jun 18, 2020
28c093e
Merge branch 'master' into taskgraph
escapewindow Jun 18, 2020
babf19b
remove skip_on_project_specific_branches
escapewindow Jun 18, 2020
3a9c6eb
Merge branch 'master' into taskgraph
escapewindow Jun 22, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
446 changes: 245 additions & 201 deletions .taskcluster.yml

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docker.d/generate_version_json.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#/bin/sh
set -e

test $GIT_HEAD_REV
test $HEAD_REV
test $TASK_ID
test $TASKCLUSTER_ROOT_URL
test $REPO_URL
Expand Down
9 changes: 5 additions & 4 deletions docker.d/push_image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ test $DOCKERHUB_EMAIL
test $DOCKERHUB_USER
test $DOCKER_REPO
test $DOCKER_TAG
test $GIT_HEAD_REV
test $HEAD_REV
test $HOME
test $PUSH_DOCKER_IMAGE
test $SECRET_URL
test $PROJECT_NAME
Expand All @@ -18,12 +19,12 @@ docker tag $DOCKER_REPO:$DOCKER_TAG $DOCKER_REPO:$DOCKER_ARCHIVE_TAG

echo "=== Generating dockercfg ==="
# docker login stopped working in Taskcluster for some reason
wget -qO- $SECRET_URL | jq '.secret.docker.dockercfg' > /root/.dockercfg
chmod 600 /root/.dockercfg
wget -qO- $SECRET_URL | jq '.secret.docker.dockercfg' > $HOME/.dockercfg
chmod 600 $HOME/.dockercfg

echo "=== Pushing to docker hub ==="
docker push $DOCKER_REPO:$DOCKER_TAG
docker push $DOCKER_REPO:$DOCKER_ARCHIVE_TAG

echo "=== Clean up ==="
rm -f /root/.dockercfg
rm -f $HOME/.dockercfg
46 changes: 46 additions & 0 deletions taskcluster/ci/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
trust-domain: scriptworker

task-priority: high

taskgraph:
register: scriptworker_taskgraph:register
repositories:
scriptworker:
name: "Scriptworker-scripts"
project-regex: scriptworker-scripts$
escapewindow marked this conversation as resolved.
Show resolved Hide resolved
default-repository: https://github.com/mozilla-releng/scriptworker-scripts
default-ref: master
type: git
escapewindow marked this conversation as resolved.
Show resolved Hide resolved

workers:
aliases:
b-linux:
provisioner: scriptworker-{level}
escapewindow marked this conversation as resolved.
Show resolved Hide resolved
implementation: docker-worker
os: linux
worker-type: 'b-linux'
images:
provisioner: scriptworker-{level}
implementation: docker-worker
os: linux
worker-type: 'images'

docker:
email: "[email protected]"
user: mozillarelengservices

notify:
email:
- [email protected]
prefix: "[scriptworker-scripts]"

release-promotion:
flavors:
promote:
target-tasks-method: promote_scriptworker

private-artifact-prefix: releng/scriptworker

scriptworker:
scope-prefix: project:scriptworker:releng
41 changes: 41 additions & 0 deletions taskcluster/ci/docker-image/kind.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
---

loader: taskgraph.loader.transform:loader

transforms:
- taskgraph.transforms.docker_image:transforms
- taskgraph.transforms.cached_tasks:transforms
- taskgraph.transforms.task:transforms

jobs:
k8s-image:
definition: k8s-image
python38:
definition: python
args:
PYTHON_VERSION: "3.8"
python37:
definition: python
args:
PYTHON_VERSION: "3.7"
pushapkscript-py38:
escapewindow marked this conversation as resolved.
Show resolved Hide resolved
definition: pushapkscript
parent: python38
pushapkscript-py37:
definition: pushapkscript
parent: python37
pushsnapscript-py38:
definition: pushsnapscript
parent: python38
pushsnapscript-py37:
definition: pushsnapscript
parent: python37
pushflatpakscript-py38:
definition: pushflatpakscript
parent: python38
pushflatpakscript-py37:
definition: pushflatpakscript
parent: python37
55 changes: 55 additions & 0 deletions taskcluster/ci/k8s-image/kind.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
---
loader: taskgraph.loader.transform:loader

kind-dependencies:
- tox

transforms:
- scriptworker_taskgraph.transforms.python_version:transforms
- scriptworker_taskgraph.transforms.k8s_image:transforms
- scriptworker_taskgraph.transforms.cached:transforms
- taskgraph.transforms.cached_tasks:transforms
- taskgraph.transforms.task:transforms

job-defaults:
description: "{name} k8s image builder py{python_version}"
label: "{name}-k8s-image-builder-python{python_version}"
run-on-tasks-for: ["action", "github-pull-request", "github-push"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: I've never used action in run-on-tasks-for before. What's the expected behavior, in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're probably going to use a different filter for actions, so it's possible this is extraneous. I originally used it as a way to filter for relpro, but we can also specifically filter for relpro elsewhere.

attributes:
code-review: true
worker-type: images
worker:
docker-image: {in-tree: 'k8s-image'}
max-run-time: 3600
docker-in-docker: true
taskcluster-proxy: true
chain-of-trust: true
artifacts:
- type: directory
name: public/
path: /builds/worker/artifacts/
command:
- sh
- -lxce
- >-
sh /usr/local/bin/build_and_push.sh
python-versions:
- 38
deploy-secret-url: http://taskcluster/secrets/v1/secret/project/releng/scriptworker-scripts/deploy
docker-repo: mozilla/releng-{name}
# TODO copy image to artifacts

jobs:
addonscript: {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Followup: I wonder if we should create a dedicated loader to avoid repeating each subproject here. For instance, there are more than a hundred subprojects all defined in this file. This loader reads that file, generates the right jobs and applies the default configuration before passing jobs to transforms.

I see we repeat the list of project here and in another kind so maybe we want a dedicated loader too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, possibly. We could add configuration as to whether we want tox, which versions of python, and whether we want a k8s image (and which versions of python). I think the current breakdown works too.

balrogscript: {}
beetmoverscript: {}
bouncerscript: {}
pushapkscript: {}
pushflatpakscript: {}
pushsnapscript: {}
shipitscript: {}
signingscript: {}
treescript: {}
18 changes: 18 additions & 0 deletions taskcluster/ci/pr/kind.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
---
loader: taskgraph.loader.transform:loader

kind-dependencies:
- k8s-image
escapewindow marked this conversation as resolved.
Show resolved Hide resolved

transforms:
- taskgraph.transforms.code_review:transforms
- taskgraph.transforms.task:transforms

jobs:
complete:
description: PR Summary Task
run-on-tasks-for: [github-pull-request]
worker-type: succeed
102 changes: 102 additions & 0 deletions taskcluster/ci/tox/kind.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
---
loader: taskgraph.loader.transform:loader

kind-dependencies:
- docker-image
escapewindow marked this conversation as resolved.
Show resolved Hide resolved

transforms:
- scriptworker_taskgraph.transforms.python_version:transforms
- scriptworker_taskgraph.transforms.cached:transforms
- taskgraph.transforms.cached_tasks:transforms
- taskgraph.transforms.job:transforms
- taskgraph.transforms.task:transforms

job-defaults:
description: "{name} tox-py{python_version}"
label: "tox-{name}-py{python_version}"
run-on-tasks-for: ["action", "github-pull-request", "github-push"]
attributes:
code-review: true
worker-type: b-linux
worker:
docker-image: {in-tree: 'python{python_version}'}
max-run-time: 1800
artifacts:
- type: directory
name: public/
path: /builds/worker/artifacts/
escapewindow marked this conversation as resolved.
Show resolved Hide resolved
python-versions:
- 38
- 37
run:
using: run-task
cache-dotcache: false
checkout:
scriptworker: {}
# sparse-profile: basestring/none
# workdir:
cwd: '{checkout}'
command:
- sh
- -lxce
- >-
tox -e {name}-py{python_version}

jobs:
addonscript:
digest-directories:
escapewindow marked this conversation as resolved.
Show resolved Hide resolved
- addonscript
balrogscript:
digest-directories:
- balrogscript
- scriptworker_client
beetmoverscript:
digest-directories:
- beetmoverscript
bouncerscript:
digest-directories:
- bouncerscript
configloader:
digest-directories:
- configloader
iscript:
digest-directories:
- iscript
- scriptworker_client
- vendored/mozbuild
notarization_poller:
digest-directories:
- notarization_poller
- scriptworker_client
pushapkscript:
worker:
docker-image: {in-tree: 'pushapkscript-py{python_version}'}
digest-directories:
- pushapkscript
pushflatpakscript:
worker:
docker-image: {in-tree: 'pushflatpakscript-py{python_version}'}
digest-directories:
- pushflatpakscript
pushsnapscript:
worker:
docker-image: {in-tree: 'pushsnapscript-py{python_version}'}
digest-directories:
- pushsnapscript
scriptworker_client:
digest-directories:
- scriptworker_client
shipitscript:
digest-directories:
- shipitscript
signingscript:
digest-directories:
- signingscript
- vendored/mozbuild
treescript:
digest-directories:
- treescript
- scriptworker_client
1 change: 1 addition & 0 deletions taskcluster/docker/REGISTRY
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
mozilla-releng
27 changes: 27 additions & 0 deletions taskcluster/docker/k8s-image/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

# We need to run a really old docker version because taskcluster is using a
# really old version in their setup
FROM docker:1.6.2
escapewindow marked this conversation as resolved.
Show resolved Hide resolved

RUN apk add --update xz
RUN mkdir -p /usr/local/bin
COPY build_and_push.sh /usr/local/bin/build_and_push.sh

# Add worker user
RUN mkdir /builds && \
adduser -D -h /builds/worker -g worker -u 1000 worker && \
mkdir /builds/worker/artifacts && \
chown worker:worker /builds/worker/artifacts

ENV SHELL=/bin/sh \
HOME=/builds/worker \
PATH=/builds/worker/.local/bin:$PATH

VOLUME /builds/worker/checkouts
VOLUME /builds/worker/.cache

# Set a default command useful for debugging
CMD ["/bin/sh"]
21 changes: 21 additions & 0 deletions taskcluster/docker/k8s-image/build_and_push.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/sh
# TODO: move docker.d/ files into the image?
set -e
test REPO_URL
test HEAD_REV
test PROJECT_NAME
test PUSH_DOCKER_IMAGE

mkdir -p /builds/worker/checkouts
cd /builds/worker/checkouts
wget ${REPO_URL}/archive/${HEAD_REV}.tar.gz
tar zxf ${HEAD_REV}.tar.gz
mv *-${HEAD_REV} src
cd src
cp ${PROJECT_NAME}/docker.d/* docker.d/
cp ${PROJECT_NAME}/Dockerfile .
sh ./docker.d/generate_version_json.sh
sh ./docker.d/build_image.sh
if [ "${PUSH_DOCKER_IMAGE}" == "1" ]; then
./docker.d/push_image.sh
fi
7 changes: 7 additions & 0 deletions taskcluster/docker/pushapkscript/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# %ARG DOCKER_IMAGE_PARENT
FROM $DOCKER_IMAGE_PARENT

VOLUME /builds/worker/checkouts
VOLUME /builds/worker/.cache
escapewindow marked this conversation as resolved.
Show resolved Hide resolved

RUN apt-get update && apt-get install -y default-jdk
escapewindow marked this conversation as resolved.
Show resolved Hide resolved
8 changes: 8 additions & 0 deletions taskcluster/docker/pushflatpakscript/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# %ARG DOCKER_IMAGE_PARENT
FROM $DOCKER_IMAGE_PARENT

VOLUME /builds/worker/checkouts
VOLUME /builds/worker/.cache

RUN apt-get update \
&& apt-get install -y gir1.2-ostree-1.0 libgirepository1.0-dev
9 changes: 9 additions & 0 deletions taskcluster/docker/pushsnapscript/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# %ARG DOCKER_IMAGE_PARENT
FROM $DOCKER_IMAGE_PARENT

VOLUME /builds/worker/checkouts
VOLUME /builds/worker/.cache

RUN apt-get update \
&& apt-get install -y libsodium-dev
RUN truncate -s 0 /etc/os-release
escapewindow marked this conversation as resolved.
Show resolved Hide resolved
Loading