Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(cpan-libs): Add cpan libs tests #5324

Draft
wants to merge 24 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f963d23
Adding cpan libs tests
sdepassio Dec 7, 2024
64f46c1
Removing Digest::SHA1 (Digest::SHA used in plugins, and on official d…
sdepassio Dec 10, 2024
c2cde6b
log error message only for true error
sdepassio Dec 10, 2024
ed360fe
check all cpan libs
sdepassio Dec 10, 2024
4621e25
Check only deb packages with error
sdepassio Dec 11, 2024
bb8655e
Fix to skip packages tested on the wrong architecture
sdepassio Dec 11, 2024
04af2bc
Keep only packages with error, and update error messages
sdepassio Dec 11, 2024
9530c9b
fix action-lint
sdepassio Dec 11, 2024
db8c1b7
update workflow + fix dependencies installation
sdepassio Dec 12, 2024
cf1ae02
test all cpan libs
sdepassio Dec 12, 2024
b118992
Add installation informations in the CI
sdepassio Dec 13, 2024
c36fa82
filter dependencies
sdepassio Dec 13, 2024
accea26
fix for rpm dependencies
sdepassio Dec 13, 2024
5e99225
update
sdepassio Dec 13, 2024
d843ea3
fix dependencies installation
sdepassio Dec 13, 2024
90e213c
fix deb dependencies
sdepassio Dec 16, 2024
b3f4fa1
Add missing lib for jammy
sdepassio Dec 17, 2024
580f190
test
sdepassio Dec 18, 2024
2b6a3a9
- Package only the deb packages that doesn't exist in the debian repo
sdepassio Dec 19, 2024
1a764cf
remove df and free commands + install local dependency only if it doe…
sdepassio Dec 19, 2024
0094f38
use dpkg to install a package, and install all his dependencies before
sdepassio Dec 20, 2024
6ef150d
test building all debs with fpm
sdepassio Dec 20, 2024
62f5134
Cleaning installation of tools/libs already in the docker image
sdepassio Dec 26, 2024
b402ea3
update packages existence checking + dependencies after cpan build
sdepassio Dec 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ apt-get install -y \
libapp-fatpacker-perl \
libcurl4-openssl-dev \
libczmq-dev \
libczmq-dev\
libfile-copy-recursive-perl \
libjson-perl \
libmodule-build-tiny-perl \
Expand All @@ -58,7 +57,6 @@ cpanm Crypt::OpenSSL::Guess

gem install fpm


echo 'deb [trusted=yes] https://repo.goreleaser.com/apt/ /' | tee /etc/apt/sources.list.d/goreleaser.list
apt-get update
apt-get install -y nfpm=2.41.0
Expand Down
225 changes: 193 additions & 32 deletions .github/workflows/perl-cpan-libraries.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,17 @@ concurrency:

on:
workflow_dispatch:
pull_request:
paths:
- ".github/workflows/perl-cpan-libraries.yml"
push:
branches:
- develop
- dev-[2-9][0-9].[0-9][0-9].x
- master
- "[2-9][0-9].[0-9][0-9].x"
paths:
- ".github/workflows/perl-cpan-libraries.yml"
# pull_request:
# paths:
# - ".github/workflows/perl-cpan-libraries.yml"
# push:
# branches:
# - develop
# - dev-[2-9][0-9].[0-9][0-9].x
# - master
# - "[2-9][0-9].[0-9][0-9].x"
# paths:
# - ".github/workflows/perl-cpan-libraries.yml"

jobs:
get-environment:
Expand Down Expand Up @@ -92,6 +92,7 @@ jobs:
- rpm_provides: ""
- version: ""
- spec_file: ""
- no-auto-depends: false
- distrib: el8
package_extension: rpm
image: packaging-plugins-alma8
Expand All @@ -112,6 +113,10 @@ jobs:
rpm_provides: "perl(FFI::Platypus::Buffer) perl(FFI::Platypus::Memory)"
rpm_dependencies: "perl(Capture::Tiny) perl(FFI::CheckLib) perl(File::Spec::Functions) perl(IPC::Cmd) perl(JSON::PP) perl(List::Util) perl(autodie) perl(constant) perl(parent)"
no-auto-depends: true
- name: "Mojo::IOLoop::Signal"
rpm_dependencies: "perl-Mojolicious"
rpm_provides: "perl(Mojo::IOLoop::Signal)"
no-auto-depends: true
- name: "Net::DHCP"
rpm_provides: "perl(Net::DHCP::Constants) perl(Net::DHCP::Packet)"
- name: "Net::SMTPS"
Expand All @@ -131,11 +136,6 @@ jobs:
- name: "ZMQ::LibZMQ4"
version: "0.01"
rpm_dependencies: "zeromq"
- name: "Mojo::IOLoop::Signal"
rpm_dependencies: "perl-Mojolicious"
rpm_provides: "perl(Mojo::IOLoop::Signal)"
no-auto-depends: true


name: package ${{ matrix.distrib }} ${{ matrix.name }}
container:
Expand Down Expand Up @@ -372,6 +372,8 @@ jobs:
build_distribs: "bookworm"
- name: "Device::Modbus::TCP::Client"
build_distribs: "bookworm"
- name: "Digest::SHA1"
build_distribs: "jammy"
- name: "Net::Amazon::Signature::V4"
build_distribs: ["bullseye", "jammy"]
- name: "Net::MQTT::Simple"
Expand Down Expand Up @@ -409,9 +411,6 @@ jobs:
name: Get package version
id: package-version
run: |
apt-get update
apt-get install -y cpanminus

if [ -z "${{ matrix.version }}" ]; then
CPAN_PACKAGE_VERSION=$(cpanm --info ${{ matrix.name }} | sed 's/\.tar\.gz$//' | sed 's/.*\-//' | sed 's/v//')

Expand All @@ -428,10 +427,23 @@ jobs:
echo "package_version=$(echo $PACKAGE_VERSION)" >> $GITHUB_OUTPUT
shell: bash

- if: ${{ contains(matrix.build_distribs, matrix.distrib) && matrix.use_dh_make_perl == 'false' }}
- if: ${{ contains(matrix.build_distribs, matrix.distrib) }}
name: Check if package already exists
run: |
apt-get install -y ruby libcurl4-openssl-dev libssh-dev uuid-dev libczmq-dev
package_name=$(echo "lib${{ matrix.name }}-perl" | sed 's/::/-/g' | tr '[:upper:]' '[:lower:]')
package_info=$(apt-cache policy $package_name)
if [[ -n $package_info ]]; then
echo "Package already exists in the repository."
candidate_version=$(echo "$package_info" | grep 'Candidate:' | awk '{print $2}')
if [[ "$candidate_version" == "${{ steps.package-version.outputs.package_version }}"* ]]; then
echo "Package already exists in the repository with the same version."
exit 0
fi
fi
shell: bash

- if: ${{ contains(matrix.build_distribs, matrix.distrib) && matrix.use_dh_make_perl == 'false' }}
run: |
if [ -z "${{ matrix.deb_dependencies }}" ]; then
PACKAGE_DEPENDENCIES=""
else
Expand All @@ -444,21 +456,11 @@ jobs:
PACKAGE_DEPENDENCIES="$PACKAGE_DEPENDENCIES --no-auto-depends"
fi

cpanm Module::Build::Tiny
cpanm Module::Install

gem install fpm
# Patch to apply fpm fix for debian package generation while waiting for the official fix to be released.
patch -i .github/patch/fpm-deb.rb.diff $(find / -type f -name "deb.rb")

fpm -a native -s cpan -t ${{ matrix.package_extension }} --deb-dist ${{ matrix.distrib }} --iteration ${{ steps.parse-distrib.outputs.package_distrib_name }} --verbose --cpan-verbose --no-cpan-test$PACKAGE_DEPENDENCIES -v ${{ steps.package-version.outputs.package_version }} ${{ matrix.name }}
shell: bash

- if: ${{ contains(matrix.build_distribs, matrix.distrib) && matrix.use_dh_make_perl == 'true' }}
run: |
apt-get install -y libcurl4-openssl-dev dh-make-perl libssh-dev uuid-dev libczmq-dev libmodule-install-perl libmodule-build-tiny-perl
# module-build-tiny is required for Mojo::IOLoop::Signal build.

DEB_BUILD_OPTIONS="nocheck nodocs notest" dh-make-perl make --dist ${{ matrix.distrib }} --build --version ${{ steps.package-version.outputs.package_version }}${{ steps.parse-distrib.outputs.package_distrib_separator }}${{ steps.parse-distrib.outputs.package_distrib_name }} --cpan ${{ matrix.name }}
shell: bash

Expand Down Expand Up @@ -524,8 +526,167 @@ jobs:
path: ./*.deb
key: ${{ github.sha }}-${{ github.run_id }}-deb-${{ matrix.distrib }}

deliver-packages:
test-packages:
needs: [get-environment, sign-rpm, download-and-cache-deb]
strategy:
fail-fast: false
matrix:
include:
- package_extension: rpm
image: almalinux:8
distrib: el8
arch: amd64
runner_name: ubuntu-24.04
- package_extension: rpm
image: almalinux:9
distrib: el9
arch: amd64
runner_name: ubuntu-24.04
- package_extension: deb
image: debian:bullseye
distrib: bullseye
arch: amd64
runner_name: ubuntu-24.04
- package_extension: deb
image: debian:bookworm
distrib: bookworm
arch: amd64
runner_name: ubuntu-24.04
- package_extension: deb
image: ubuntu:jammy
distrib: jammy
arch: amd64
runner_name: ubuntu-24.04
- package_extension: deb
image: arm64v8/debian:bullseye
distrib: bullseye
arch: arm64
runner_name: ["self-hosted", "collect-arm64"]

runs-on: ${{ matrix.runner_name }}
container:
image: ${{ matrix.image }}

name: Test perl CPAN libs packages on ${{ matrix.package_extension }} ${{ matrix.distrib }} ${{ matrix.arch }}

steps:
- if: ${{ matrix.package_extension == 'rpm' }}
name: Install zstd, perl and Centreon repositories
run: |
dnf install -y zstd perl epel-release 'dnf-command(config-manager)'
dnf config-manager --set-enabled powertools || true # alma 8
dnf config-manager --set-enabled crb || true # alma 9
# Import Centreon GPG key
GPG_KEY_URL="https://yum-gpg.centreon.com/RPM-GPG-KEY-CES"
curl -sSL $GPG_KEY_URL -o RPM-GPG-KEY-CES
rpm --import RPM-GPG-KEY-CES
shell: bash

- if: ${{ matrix.package_extension == 'deb' }}
name: Install zstd, perl and Centreon repositories
run: |
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y zstd perl wget gpg apt-utils procps
wget -O- https://apt-key.centreon.com | gpg --dearmor | tee /etc/apt/trusted.gpg.d/centreon.gpg > /dev/null 2>&1
apt-get update
shell: bash

- name: Restore packages from cache
uses: actions/cache/restore@6849a6489940f00c2f30c0fb92c6274307ccb58a # v4.1.2
with:
path: ./*.${{ matrix.package_extension }}
key: ${{ github.sha }}-${{ github.run_id }}-${{ matrix.package_extension }}-${{ matrix.distrib }}
fail-on-cache-miss: true

- if: ${{ matrix.package_extension == 'rpm' }}
name: Install packages
run: |
error_log="install_error_${{ matrix.distrib }}_${{ matrix.arch }}.log"
for package in ./*.rpm; do
echo "Installing package: $package"
# List dependencies, and remove version and comparison operators
dependencies=$(rpm -qpR $package | sed 's/ [0-9.-]*\(\s\|$\)/ /g' | sed 's/ [<>!=]*\(\s\|$\)/ /g')
for dependency in $dependencies; do
# Skip non-perl dependencies
if [[ $dependency != perl* ]]; then
continue
else
echo "Check dependency: $dependency"
# Update the dependency name to match the package name
dependency=$(echo $dependency | sed 's/(/-/g' | sed 's/)//g' | sed 's/::/-/g')
fi
# If the dependency has been built in the same workflow, install it
if [[ -n $(find . -maxdepth 1 -regex "\.\/$dependency-[0-9v].*\.rpm") ]]; then
echo "Installing dependency: $dependency"
error_output=$(dnf install -y ./$dependency*.rpm 2>&1) || { echo "$error_output" >> $error_log; echo "Error during installation of the dependency $dependency" >> $error_log; true; }
fi
done
# Install package, then uninstall it with all his dependencies
echo "Package installation..."
error_output=$(dnf install -y $package 2>&1) || { echo "$error_output" >> $error_log; echo "Error during installation of the package $package" >> $error_log; true; }
echo "Package installation done."
echo "Package uninstallation..."
error_output=$(dnf autoremove -y $(echo $package | sed 's/_[0-9].*\.rpm//' | sed 's/.\///') 2>&1) || { echo "$error_output" >> $error_log; echo "Error during autoremove of the package $package" >> $error_log; true; }
echo "Package uninstallation done."
done
# If the file error_log exists and is not empty, the workflow is in error
if [[ -s $error_log ]]; then
cat $error_log
exit 1
fi
shell: bash

- if: ${{ matrix.package_extension == 'deb' }}
name: Install packages
run: |
error_log="install_error_${{ matrix.distrib }}_${{ matrix.arch }}.log"
for package in ./*.deb; do
# If the debian package name ends with amd64 or arm64, we only install it if the tested architecture is the same, otherwise we skip it
if [[ $package == *amd64.deb && ${{ matrix.arch }} != "amd64" || $package == *arm64.deb && ${{ matrix.arch }} != "arm64" ]]; then
continue
fi
echo "Installing package: $package"
# List dependencies
dependencies=$(dpkg-deb -I $package | grep Depends | sed 's/Depends: //' | sed 's/,//g' | sed 's/(\(.*\)//g') || { echo "$error_output" >> $error_log; echo "Error while listing dependencies of the package $package" >> $error_log; true; }
for dependency in $dependencies; do
# If the dependency exists in the Debian repository, don't check the local dependencies
dependency_info=$(apt-cache policy $dependency)
if [[ -n $dependency_info ]]; then
echo "Dependency $dependency exists in debian repository."
apt-get install -y $dependency
else
# If the dependency has been built in the same workflow, install it
for dependency_package in $(find . -maxdepth 1 -regex "\.\/${dependency}_[0-9].*all\.deb" -o -regex "\.\/${dependency}_[0-9].*${{ matrix.arch }}\.deb"); do
echo "Installing dependency: $dependency_package"
error_output=$(apt-get install -y ./$dependency_package 2>&1) || { echo "$error_output" >> $error_log; echo "Error during installation of the dependency $dependency" >> $error_log; true; }
done
fi
done
# Install package, then uninstall it with all his dependencies
echo "Package installation..."
error_output=$(dpkg -i $package 2>&1) || { echo "$error_output" >> $error_log; echo "Error during installation of the package $package" >> $error_log; true; }
echo "Package installation done."
echo "Package uninstallation..."
error_output=$(apt autoremove --purge $(echo $package | sed 's/_[0-9].*\.deb//' | sed 's/.\///') 2>&1) || { echo "$error_output" >> $error_log; echo "Error during autoremove of the package $package" >> $error_log; true; }
echo "Package uninstallation done."
done
# If the file error_log exists and is not empty, the workflow is in error
if [[ -s $error_log ]]; then
cat $error_log
exit 1
fi
shell: bash

- name: Upload error log
if: failure()
uses: actions/upload-artifact@v4
with:
name: install_error_log_${{ matrix.distrib }}-${{ matrix.arch }}
path: install_error_${{ matrix.distrib }}_${{ matrix.arch }}.log

deliver-packages:
needs: [get-environment, sign-rpm, download-and-cache-deb, test-packages]
if: |
needs.get-environment.outputs.skip_workflow == 'false' &&
(contains(fromJson('["testing", "unstable"]'), needs.get-environment.outputs.stability) || ( needs.get-environment.outputs.stability == 'stable' && github.event_name != 'workflow_dispatch')) &&
Expand Down
Loading
Loading