Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Builds a simple debian package on tag #386

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 0 additions & 42 deletions .github/install_dependencies

This file was deleted.

130 changes: 127 additions & 3 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Test build

on:
push:
branches: [main, unstable]
branches: [main, unstable, feature/debian-package]
pull_request:
schedule:
- cron: '39 13 * * *' # run daily
Expand All @@ -11,15 +11,16 @@ jobs:
build:
strategy:
matrix:
os: [ ubuntu-22.04, macos-latest, rpi3b, macos-11, macos-12, ubuntu-latest, ubuntu-20.04 ]
os: [ ubuntu-22.04, macos-latest, macos-11, macos-12, macos-13, ubuntu-latest, ubuntu-20.04 ]
# TODO: re-enable before creating PR: rpi3b,
runs-on: ${{ matrix.os }}
timeout-minutes: 35 # runtime across all OSs, runs can get queued and mac runs take 15-20 min
steps:
- name: Checkout repository
uses: actions/checkout@main

- name: Install packaged dependencies
run: .github/install_dependencies
run: script/install_dependencies.sh

- name: Configure
run: |
Expand Down Expand Up @@ -47,3 +48,126 @@ jobs:

- name: Test run
run: /usr/local/bin/rtl_airband -v

- name: Set snapshot version
id: sha
run: echo "snapshot_version=$(script/snapshot_version.sh)" >> $GITHUB_OUTPUT

- name: Create debian package
if: runner.os == 'Linux'
run: |
pkg_dir="$(mktemp --directory)"
export PACKAGE="rtl-airband"
export VERSION="${{ steps.sha.outputs.snapshot_version }}"
export ARCH="$(dpkg --print-architecture)"
export MAINTAINER="charlie-foxtrot"
export DEPENDS="libc6 (>= 2.34), libconfig++9v5 (>= 1.5-0.4), libmp3lame0 (>= 3.100), libshout3 (>= 2.4.5), librtlsdr0 (>= 0.6.0), libsoapysdr0.8 (>= 0.8.1), libfftw3-single3 (>= 3.3.8), libpulse0 (>= 14.2)"
export DESCRIPTION="RTLSDR Airband - A multiband decoder for AM and NFM signals"
script/package.sh "${pkg_dir}"

# TODO use brew for macOS? https://brew.sh/
- name: Create macOS package
if: runner.os == 'macOS'
run: |
tar -zcvf rtl-airband_${{ steps.sha.outputs.snapshot_version }}_${{ matrix.os }}.tar.gz build_Release_NFM/src/rtl_airband

- name: "Upload package as artifact"
uses: actions/upload-artifact@v3
with:
name: "rtl-airband_${{ steps.sha.outputs.snapshot_version }}_${{ matrix.os }}"
path: |
*.deb
*.tar.gz
retention-days: 7
svavassori marked this conversation as resolved.
Show resolved Hide resolved

build-foreign-arch:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this doing? its running on a linux host and using docker to build amd64, arm64, arm/v7, and arm/v6? how does it differ from a matrix build? if it is independent can it be its own .yml?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can be put in its own yaml. As first version I put together because was easier to link this job as a dependency of the previous one, I'll split in a separate one.

I used this way to get the build for foreign architecture in an easy and reliable manner. Another one would be to use gcc cross-compile capability (and it may be also faster), but with cross-compilation there is a small chance that linking with dynamic library could have some issues.
Same thing could be done on macOS (amd64 and arm64) if needed.

The main advantages of building by using this method are:

  • the build environment is under our control and it can be replicated in the local machine, so no dependency on which configuration/package version is used by the workers or side effects, so it makes debugging the build much easier (as a matter of fact, the version of the libraries we are compiling against is based on the one available in that OS version, not as a requirement of the project).
  • we are not limited to the architecture available on GitHub (e.g. RISC-V 32/64 bit in a future maybe?)
  • it doesn't need to modify the worker, so better isolation and pre-configured images could also be used.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a look at dependencies between workflows and it is possible to link them, but it will run only on default branch see notes.

What it can be done is to create a parent workflow that calls in order the child workflows. I'm not sure if it is really helpful for this size, thought.

needs: build
runs-on: ubuntu-latest
permissions:
contents: write
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@main

- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2

- name: Available build platforms
run: echo ${{ steps.buildx.outputs.platforms }}

- name: Login to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker metadata
id: meta_worker
uses: docker/metadata-action@v4
with:
# ghcr.io/charlie-foxtrot/rtl-airband
images: |
ghcr.io/svavassori/rtl-airband
# generate Docker tags based on the following events/attributes
tags: |
type=schedule
type=ref,event=branch
type=ref,event=pr
type=ref,event=tag
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha,prefix={{branch}}-

- name: Build and push Docker images
uses: docker/build-push-action@v4
with:
cache-to: type=local,dest=/tmp/buildx-cache
context: .
file: Dockerfile
platforms: linux/amd64,linux/arm64,linux/arm/v7,linux/arm/v6
outputs: type=image
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.meta_worker.outputs.tags }}
labels: ${{ steps.meta_worker.outputs.labels }}

- name: "Extract debian packages from docker cache"
run: |
mkdir packages
find /tmp/buildx-cache/ -type f -size +50k -a -size -512k | xargs -I {} tar -xvf {} --strip-components=1 --directory=packages/

- name: Set snapshot version
id: sha
run: echo "snapshot_version=$(script/snapshot_version.sh)" >> $GITHUB_OUTPUT

- name: "Upload packages as artifacts"
uses: actions/upload-artifact@v3
with:
name: "rtl-airband_${{ steps.sha.outputs.snapshot_version }}_multiarch-deb"
path: "packages/*.deb"
retention-days: 7

- name: "Download all OS packages"
run: mkdir /tmp/other-artifacts

- uses: actions/download-artifact@v3
with:
path: /tmp/other-artifacts

- name: "Copy artifacts into packages directory"
run: find /tmp/other-artifacts -type f -exec mv {} packages/ \;

- name: Release packages
uses: softprops/action-gh-release@v1
with:
files: |
packages/*.deb
packages/*.tar.gz
prerelease: ${{ github.ref_type != 'tag' }}
name: ${{ github.ref_name }}-${{ steps.sha.outputs.snapshot_version }}
tag_name: ${{ github.ref_name }}-${{ steps.sha.outputs.snapshot_version }}
fail_on_unmatched_files: true
57 changes: 57 additions & 0 deletions .github/workflows/package.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Debian packaging

on:
push:
tags: [ 'v[0-9]+.[0-9]+.[0-9]+' ]

jobs:
build:
permissions:
contents: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: Install packaged dependencies
run: .github/install_dependencies

- name: Configure
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's a lot of overlap between these steps and Dockerfile / script/package.sh, can this all be done in a single place / script that is leveraged by the two flows?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this package.yml is the old one that I dind't deleted, but it should be merge/substituted with the "foreign-arch" one.

So the final flow should be:
Build OS/arch matrix > package > join all build artifact and pre-release if on PR or release if on tag

run: |
cmake -B ${{github.workspace}}/build_Release_NFM -DCMAKE_BUILD_TYPE=Release -DNFM=TRUE

- name: Build
run: |
VERBOSE=1 cmake --build ${{github.workspace}}/build_Release_NFM -j

- name: create .deb content structure
run: |
mkdir --parents .debpkg/lib/systemd/system/
sed 's|/usr/local/bin|/usr/bin|g' init.d/rtl_airband.service > .debpkg/lib/systemd/system/rtl_airband.service

mkdir --parents .debpkg/usr/bin
cp build_Release_NFM/src/rtl_airband .debpkg/usr/bin/

mkdir --parents .debpkg/usr/share/rtl_airband/config
cp config/*.conf .debpkg/usr/share/rtl_airband/config/

mkdir --parents .debpkg/usr/share/doc/rtl_airband
cp README.md LICENSE .debpkg/usr/share/doc/rtl_airband/
gzip --keep --stdout NEWS.md > .debpkg/usr/share/doc/rtl_airband/NEWS.md.gz

- uses: jiro4989/build-deb-action@v3
with:
package: rtl-airband
package_root: .debpkg
maintainer: charlie-foxtrot
version: ${{ github.ref }} # refs/tags/v*.*.*
arch: 'amd64'
depends: 'libc6 (>= 2.34), libconfig++9v5 (>= 1.5-0.4), libmp3lame0 (>= 3.100), libshout3 (>= 2.4.5), librtlsdr0 (>= 0.6.0), libsoapysdr0.8 (>= 0.8.1), libfftw3-single3 (>= 3.3.8), libpulse0 (>= 14.2)'
desc: 'RTLSDR Airband - A multiband decoder for AM and NFM signals'
compress_type: xz

- name: Release
uses: softprops/action-gh-release@v1
with:
files: rtl-airband_*_amd64.deb
fail_on_unmatched_files: true

48 changes: 48 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
FROM debian:12.0-slim AS build

# copy script files
WORKDIR /build
COPY script /build/script

# install development dependencies
RUN export DEBIAN_FRONTEND="noninteractive" \
&& apt-get update \
&& apt-get upgrade --quiet --yes \
&& apt-get install --quiet --yes --no-install-recommends \
gettext-base `# provides envsubst` \
git `# to extract commit info` \
&& script/install_debian_dependencies.sh \
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calling the generic script/install_dependencies.sh would mean fewer changes if this was re-used for a different platform

&& rm -rf /var/lib/apt/lists/*

# copy the rest of the project
COPY . /build

# configure and build
RUN cmake --compile-no-warning-as-error -B build_Release_NFM -DCMAKE_BUILD_TYPE=Release -DNFM=TRUE \
&& VERBOSE=1 cmake --build build_Release_NFM -j$(nproc)

# create .deb content structure and package it
RUN pkg_dir="$(mktemp --directory)" \
&& PACKAGE="rtl-airband" \
VERSION="${VERSION:-$(script/snapshot_version.sh)}" \
ARCH="$(dpkg --print-architecture)" \
MAINTAINER="charlie-foxtrot" \
DEPENDS="libc6 (>= 2.34), libconfig++9v5 (>= 1.5-0.4), libmp3lame0 (>= 3.100), libshout3 (>= 2.4.5), librtlsdr0 (>= 0.6.0), libsoapysdr0.8 (>= 0.8.1), libfftw3-single3 (>= 3.3.8), libpulse0 (>= 14.2)" \
DESCRIPTION="RTLSDR Airband - A multiband decoder for AM and NFM signals" \
script/package.sh "${pkg_dir}"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this info ends up being hard coded 4 times (builds.yml, package.yml, package.sh, and Dockerfile). Can this be stored in a single file that is sourced in the other locations?


FROM debian:12.0-slim as install
COPY --from=build /build/rtl-airband*.deb /tmp
RUN export DEBIAN_FRONTEND="noninteractive" \
&& apt-get update \
&& apt-get upgrade --quiet --yes \
&& apt-get install --quiet --yes --no-install-recommends \
/tmp/rtl-airband*.deb \
# && rm /tmp/rtl-airband*.deb \
&& rm -rf /var/lib/apt/lists/*

# trick to get all the content without the additional "COPY" layer
#FROM scratch
#COPY --from=install / /
ENTRYPOINT ["/usr/bin/rtl_airband"]
CMD [ "-e", "-f", "-c", "/usr/share/rtl_airband/config/default.conf" ]
30 changes: 30 additions & 0 deletions config/basic_emergency.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# This is a minimalistic configuration file for RTLSDR-Airband.
# Just a single RTL dongle with one emergency AM channel.
#
# Refer to https://github.com/charlie-foxtrot/RTLSDR-Airband/wiki
# for description of keywords and config syntax.

devices:
({
type = "rtlsdr";
index = 0;
gain = 25;
centerfreq = 121;
correction = 48;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change correction to 0 or remove

channels:
(
{
freq = 121.5;
outputs: (
{
type = "file";
directory = "/tmp";
filename_template = "E1_";
name = "Tower Emergency";
genre = "ATC";
}
);
}
);
}
);
17 changes: 17 additions & 0 deletions script/install_debian_dependencies.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#! /bin/bash

set -o errexit
set -o nounset
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should these be set in install_dependencies.sh and install_macos_dependencies.sh as well?


export DEBIAN_FRONTEND="noninteractive"
apt-get update
apt-get install --quiet --yes --no-install-recommends \
build-essential \
cmake \
libconfig++-dev \
libfftw3-dev \
libmp3lame-dev \
libpulse-dev \
librtlsdr-dev \
libshout3-dev \
libsoapysdr-dev
21 changes: 21 additions & 0 deletions script/install_dependencies.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash

unameOut="$(uname -s)"

echo "Running on ${unameOut} as ${USER}"

case "${unameOut}" in
Linux*)
echo "Installing Linux dependencies"
sudo script/install_debian_dependencies.sh
;;

Darwin*)
echo "Installing MacOS dependencies"
script/install_macos_dependencies.sh
;;

*)
echo "Error: Machine not supported"
exit -1
esac
14 changes: 14 additions & 0 deletions script/install_macos_dependencies.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#! /bin/bash

brew update
brew upgrade
brew install \
lame \
libshout \
libconfig \
fftw \
librtlsdr \
soapysdr \
pulseaudio \
pkg-config
brew link openssl
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is openssl a new dependency? should it be included in the install list as well?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't a new dependency and it is installed by default; it is used by libshout3 if I remember correctly, so probably a hidden one.

On macOS 13 it doesn't have the link created by default, so the build fails because cannot find it (even if it's installed).
I don't know if it's a new default from macOS-13 or if it's a bug in the setup configuration of the worker VM

Loading