Skip to content

Commit

Permalink
dist: build debian packages for multiple platforms (scionproto#4448)
Browse files Browse the repository at this point in the history
Build debian packages for amd64, arm64, i386 and armel.
- There are separate packages for router, control, daemon, dispatcher,
gateway and tools (scion and scion-pki).
- The packages include systemd unit files to run the services. For the
daemon, dispatcher and gateway one instance per host is supported by the
systemd service, and a default configuration file is included. For
router and control, multiple instances per host are supported, and as a
consequence of this, no default configuration file is provided.
- Currently, there is no man page contained in the packages. We should
be able to build these from our existing manuals, but it seems to
require a significant amount of fiddling to get something useful.

Building the .deb packages uses bazel with `rules_pkg`.
The target `//dist:deb_all` cross-builds packages for the default set of
target platforms. Alternatively, the target `//dist:deb` allows to build
(all) packages for the current target platform. This current platform
can be set with the `--platforms` bazel option (see
https://github.com/bazelbuild/rules_go#how-do-i-cross-compile for more
details).
- To increase reuse of build results while cross-building, some internal
targets related to openapi forcibly ignore the target platform
- The package version is based on the current git tag. `rules_pkg` can
include this version _in_ the package metadata, but bazel _cannot_ spit
out appropriately named package files. This is addressed by copying and
renaming the package files after build in a make target `make dist-deb`.

Add installation documentation for the packages and the systemd units.
As a side effect, slightly reorganize the build documentation also, trying to 
make give a simpler path to just build the binaries without installing the
entire development setup.
  • Loading branch information
matzf authored and juagargi committed Mar 8, 2024
1 parent 8ed74fa commit 30ed911
Show file tree
Hide file tree
Showing 29 changed files with 1,024 additions and 74 deletions.
7 changes: 4 additions & 3 deletions .buildkite/hooks/pre-command
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,10 @@ rm -f $HOME/.bazelrc
# --nostamp is required for better caching (only on non-release jobs).
if [ "$BUILDKITE_PIPELINE_SLUG" == "scion" ]; then
echo "build --nostamp" > $HOME/.bazelrc
# Also set a fixed GIT_VERSION so that the workspace_status_command always
# returns the same value on CI to improve cache reuse.
export GIT_VERSION="ci-fixed"
# Shorten the git version to omit commit information, improving cache reuse.
# The format of git-version is "<tag>-<number-of-commits-since-the-tag>-<commit-short-hash>"
# This will be shortened to "<tag>-modified-ci"
export GIT_VERSION=$(tools/git-version | sed 's/-.*/-modified-ci/')
else
echo "build --stamp" > $HOME/.bazelrc
fi
Expand Down
25 changes: 24 additions & 1 deletion .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ steps:
plugins:
- scionproto/metahook#v0.3.0:
post-artifact: |
cat << EOF | buildkite-agent annotate --style "info"
cat << EOF | buildkite-agent annotate --style "info" --context "binaries"
#### Build outputs
- <a href="artifact://bazel-bin/scion.tar">SCION binaries</a>
- <a href="artifact://bazel-bin/scion-ci.tar">SCION test tools and utilities</a>
Expand All @@ -23,6 +23,28 @@ steps:
- exit_status: 255 # Forced agent shutdown
timeout_in_minutes: 10
- wait
- label: "Package :debian:"
command:
- make dist-deb
- cd deb;
- tar -chaf scion-deb-amd64.tar.gz *_amd64.deb
- tar -chaf scion-deb-arm64.tar.gz *_arm64.deb
- tar -chaf scion-deb-i386.tar.gz *_i386.deb
- tar -chaf scion-deb-armel.tar.gz *_armel.deb
artifact_paths:
- "deb/*.tar.gz"
plugins:
- scionproto/metahook#v0.3.0:
post-artifact: |
cat << EOF | buildkite-agent annotate --style "info" --context "packages"
#### Packages :debian:
- <a href="artifact://deb/scion-deb-amd64.tar.gz">amd64</a>
- <a href="artifact://deb/scion-deb-arm64.tar.gz">arm64</a>
- <a href="artifact://deb/scion-deb-i386.tar.gz">i386</a>
- <a href="artifact://deb/scion-deb-armel.tar.gz">armel</a>
EOF
key: dist-deb
retry: *automatic-retry
- label: "Unit Tests :bazel:"
command:
- bazel test --config=race --config=unit_all
Expand Down Expand Up @@ -70,6 +92,7 @@ steps:
timeout_in_minutes: 20
key: check_generated
retry: *automatic-retry
- wait
- group: "End to End"
key: e2e
steps:
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ doc/venv/
/bin/*
!/bin/.keepme

# Generated package files
##########################
/deb/

# CTags
##########################
tags
Expand Down
17 changes: 16 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: all build build-dev antlr clean docker-images gazelle go.mod licenses mocks protobuf scion-topo test test-integration write_all_source_files
.PHONY: all build build-dev dist-deb antlr clean docker-images gazelle go.mod licenses mocks protobuf scion-topo test test-integration write_all_source_files

build-dev:
rm -f bin/*
Expand All @@ -11,6 +11,21 @@ build:
bazel build //:scion
tar -kxf bazel-bin/scion.tar -C bin

dist-deb:
bazel build //dist:deb_all
mkdir -p deb; rm -f deb/*;
@ # Bazel cannot include the version in the filename, if we want to set it automatically from the git tag.
@ # Extract the version from the .deb "control" manifest and expand the "__" in the filename to "_<version>_".
@ # See e.g. https://en.wikipedia.org/wiki/Deb_(file_format)#Control_archive
@for f in `bazel cquery //dist:deb_all --output=files 2>/dev/null`; do \
if [ -f "$$f" ]; then \
bf=`basename $$f`; \
v="$$(ar p $$f control.tar.gz | tar -xz --to-stdout ./control | sed -n 's/Version: //p')"; \
bfv=$${bf%%__*}_$${v}_$${bf#*__}; \
cp -v "$$f" deb/$$bfv; \
fi \
done

# all: performs the code-generation steps and then builds; the generated code
# is git controlled, and therefore this is only necessary when changing the
# sources for the code generation.
Expand Down
52 changes: 34 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,53 @@
# SCION

[![Slack chat](https://img.shields.io/badge/chat%20on-slack-blue?logo=slack)](https://scionproto.slack.com)
[![Matrix chat](https://img.shields.io/badge/chat%20on-matrix-blue?logo=matrix)](https://matrix.to/#/#dev:matrix.scion.org)
[![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/scionproto/awesome-scion)
[![ReadTheDocs](https://img.shields.io/badge/doc-reference-blue?version=latest&style=flat&label=docs&logo=read-the-docs&logoColor=white)](https://docs.scion.org/en/latest)
[![Documentation](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white)](https://pkg.go.dev/github.com/scionproto/scion)
[![Go Docs](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go&logoColor=white)](https://pkg.go.dev/github.com/scionproto/scion)
[![Nightly Build](https://badge.buildkite.com/b70b65b38a75eb8724f41a6f1203c9327cfb767f07a0c1934e.svg)](https://buildkite.com/scionproto/scion-nightly/builds/latest)
[![Go Report Card](https://goreportcard.com/badge/github.com/scionproto/scion)](https://goreportcard.com/report/github.com/scionproto/scion)
[![GitHub issues](https://img.shields.io/github/issues/scionproto/scion/help%20wanted.svg?label=help%20wanted&color=purple)](https://github.com/scionproto/scion/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22)
[![GitHub issues](https://img.shields.io/github/issues/scionproto/scion/good%20first%20issue.svg?label=good%20first%20issue&color=purple)](https://github.com/scionproto/scion/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)
[![Release](https://img.shields.io/github/release-pre/scionproto/scion.svg)](https://github.com/scionproto/scion/releases)
[![License](https://img.shields.io/github/license/scionproto/scion.svg?maxAge=2592000)](https://github.com/scionproto/scion/blob/master/LICENSE)

Welcome to the open-source implementation of
[SCION](http://www.scion-architecture.net) (Scalability, Control and Isolation
On next-generation Networks), a future Internet architecture. SCION is the first
clean-slate Internet architecture designed to provide route control, failure
isolation, and explicit trust information for end-to-end communication. To find
out more about the project, please visit our [documentation
site](https://docs.scion.org/en/latest/).
Welcome to the open-source implementation of [SCION](http://www.scion-architecture.net)
(Scalability, Control and Isolation On next-generation Networks), a future Internet architecture.
SCION provides route control, failure isolation, and explicit trust information for end-to-end communication.
To find out more about the project, please visit our [documentation site](https://docs.scion.org/en/latest/).

## Connecting to the SCION Test Network
## Installation

Join [SCIONLab](https://www.scionlab.org) if you're interested in playing with
SCION in an operational global test deployment of SCION. As part of the SCIONLab
project, we support [pre-built binaries as Debian
packages](https://docs.scionlab.org/content/install/).
Installation packages for Debian and derivatives are available for x86-64, arm64, x86-32 and arm.
These packages can be found in the [latest release](https://github.com/scionproto/scion/releases/latest).
Packages for in-development versions can be found from the [latest nightly build](https://buildkite.com/scionproto/scion-nightly/builds/latest).

## Building
Alternatively, "naked" pre-built binaries are available for Linux x86-64 and
can be downloaded from the [latest release](https://github.com/scionproto/scion/releases/latest) or the
[latest nightly build](https://buildkite.com/scionproto/scion-nightly/builds/latest).

To find out how to work with SCION, please visit our [documentation
site](https://docs.scion.org/en/latest/dev/setup.html)
for instructions on how to install build dependencies, build and run SCION.
### Build from sources

Pre-built binaries for x86-64 Linux are available from the [latest nightly build](https://buildkite.com/scionproto/scion-nightly/builds/latest).
SCION can be built with `go build`. To build all binaries used in a SCION deployment (i.e.
excluding the testing and development tools), run

```sh
CGO_ENABLED=0 go build -o bin ./router/... ./control/... ./dispatcher/... ./daemon/... ./scion/... ./scion-pki/... ./gateway/...
```

The default way to build SCION, however, uses Bazel.
In particular, this allows to run all the tests, linters etc.
Please visit our [documentation site](https://docs.scion.org/en/latest/dev/setup.html) for
instructions on how to set up Bazel and the full development environment.

### Connecting to the SCION Network

Join [SCIONLab](https://www.scionlab.org) if you're interested in playing with SCION in an
operational global test deployment of SCION.

The [awesome-scion](https://github.com/scionproto/awesome-scion#deployments) list contains
pointers to production deployments of SCION.

## Contributing

Expand Down
121 changes: 121 additions & 0 deletions dist/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
load(":package.bzl", "scion_pkg_deb")
load(":platform.bzl", "multiplatform_filegroup")
load(":git_version.bzl", "git_version")

git_version(
name = "git_version",
tags = ["manual"],
visibility = ["//visibility:private"],
)

scion_pkg_deb(
name = "router_deb",
depends = [
"adduser",
],
description = "SCION inter-domain network architecture border router",
executables = {
"//router/cmd/router:router": "scion-router",
},
package = "scion-router",
postinst = "debian/scion.postinst",
systemds = ["systemd/[email protected]"],
version_file = ":git_version",
)

scion_pkg_deb(
name = "control_deb",
configs = [],
depends = [
"adduser",
"scion-dispatcher",
],
description = "SCION inter-domain network architecture control service",
executables = {
"//control/cmd/control:control": "scion-control",
},
package = "scion-control",
systemds = ["systemd/[email protected]"],
version_file = ":git_version",
)

scion_pkg_deb(
name = "dispatcher_deb",
configs = ["conffiles/dispatcher.toml"],
depends = [
"adduser",
],
description = "SCION dispatcher",
executables = {
"//dispatcher/cmd/dispatcher:dispatcher": "scion-dispatcher",
},
package = "scion-dispatcher",
postinst = "debian/scion.postinst",
systemds = ["systemd/scion-dispatcher.service"],
version_file = ":git_version",
)

scion_pkg_deb(
name = "daemon_deb",
configs = ["conffiles/sciond.toml"],
depends = [
"adduser",
],
description = "SCION daemon",
executables = {
"//daemon/cmd/daemon:daemon": "scion-daemon",
},
package = "scion-daemon",
postinst = "debian/scion.postinst",
systemds = ["systemd/scion-daemon.service"],
version_file = ":git_version",
)

scion_pkg_deb(
name = "gateway_deb",
configs = [
"conffiles/sig.json",
"conffiles/sig.toml",
],
depends = [
"adduser",
"scion-dispatcher",
"scion-daemon",
],
description = "SCION-IP Gateway",
executables = {
"//gateway/cmd/gateway:gateway": "scion-ip-gateway",
},
package = "scion-ip-gateway",
systemds = ["systemd/scion-ip-gateway.service"],
version_file = ":git_version",
)

scion_pkg_deb(
name = "tools_deb",
depends = [
"adduser",
"scion-dispatcher",
"scion-daemon",
],
description = "SCION tools",
executables = {
"//scion/cmd/scion:scion": "scion",
"//scion-pki/cmd/scion-pki:scion-pki": "scion-pki",
},
package = "scion-tools",
version_file = ":git_version",
)

multiplatform_filegroup(
name = "deb",
srcs = [
"control_deb",
"daemon_deb",
"dispatcher_deb",
"gateway_deb",
"router_deb",
"tools_deb",
],
visibility = ["//dist:__subpackages__"],
)
10 changes: 10 additions & 0 deletions dist/conffiles/dispatcher.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[dispatcher]
id = "dispatcher"
socket_file_mode = "0777"

[log.console]
level = "info"

# Optionally expose metrics and other local inspection endpoints.
# [metrics]
# prometheus = "[127.0.0.1]:30441"
21 changes: 21 additions & 0 deletions dist/conffiles/sciond.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
[general]
id = "sd"
config_dir = "/etc/scion"
reconnect_to_dispatcher = true

[path_db]
connection = "/var/lib/scion/sd.path.db"

[trust_db]
connection = "/var/lib/scion/sd.trust.db"

# Optionally enable DRKey
# [drkey_level2_db]
# connection = "/var/lib/scion/sd.drkey_level2.db"

[log.console]
level = "info"

# Optionally expose metrics and other local inspection endpoints.
# [metrics]
# prometheus = "127.0.0.1:30455"
11 changes: 11 additions & 0 deletions dist/conffiles/sig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"ASes": {
"<remote_sig_AS>": {
"Nets": [
"<remote_sig_IPnet>"
]
}
},
"ConfigVersion": 9001
}

12 changes: 12 additions & 0 deletions dist/conffiles/sig.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[gateway]
traffic_policy_file = "/etc/scion/sig.json"

[tunnel]
name = "sig"

[log.console]
level = "info"

# Optionally expose metrics and other local inspection endpoints.
# [metrics]
# prometheus = "127.0.0.1:30456"
26 changes: 26 additions & 0 deletions dist/debian/scion.postinst
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/bin/bash
set -e

# summary of how this script can be called:
# * <postinst> `configure' <most-recently-configured-version>
# * <old-postinst> `abort-upgrade' <new version>
# * <conflictor's-postinst> `abort-remove' `in-favour' <package>
# <new-version>
# * <postinst> `abort-remove'
# * <deconfigured's-postinst> `abort-deconfigure' `in-favour'
# <failed-install-package> <version> `removing'
# <conflicting-package> <version>
# for details, see http://www.debian.org/doc/debian-policy/ or
# the debian-policy package

case "$1" in
configure)
# Create system user
adduser --system --home /var/lib/scion --group scion
# Create configuration directory
mkdir /etc/scion/ >& /dev/null || true
chown scion:scion /etc/scion/
;;
*)
;;
esac
20 changes: 20 additions & 0 deletions dist/git_version.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
def _git_version_impl(ctx):
ctx.actions.run_shell(
outputs = [ctx.outputs.outfile],
inputs = [ctx.info_file],
command = r"sed -n 's/STABLE_GIT_VERSION\s*v\?//p' " + ctx.info_file.path + " > " + ctx.outputs.outfile.path,
)

git_version = rule(
doc = """
Extracts the STABLE_GIT_VERSION from the workspace_status_command output.
See also .bazelrc and tools/bazel-build-env.
The output of this rule is a file containing the version only.
The leading "v" from the git tag is removed.
""",
implementation = _git_version_impl,
outputs = {
"outfile": "git-version",
},
)
Loading

0 comments on commit 30ed911

Please sign in to comment.