Import RPM build dependencies from CentOS and EPEL

This page is now obsolete, but I'm keeping it in case it could be useful to someone

Build dependencies from CentOS and EPEL

The source RPMs ISO provided by the Xenserver project is missing many needed build dependencies from CentOS and EPEL.

One can guess that Citrix has internal frozen repositories for those packages and does not communicate about their contents (we can mostly infer it from RPM build dependencies), nor says what exact versions of the packages are used (we have to guess). It looks like they stick to a given version of CentOS (e.g. 7.2) for some time, maybe with a few exceptions.

Xenserver 7.6 still relied mostly on the old CentOS 7.2 so we can use the frozen CentOS 7.2 repositories from vault.centos.org (version 8.0 should bring fresh air with more recent packages from CentOS). However EPEL packages (they are only a few, thankfully) are not frozen so there's a risk we'd use a version that is "too" recent and brings incompatibilities.

Add the missing dependencies to Koji

Hunting the missing dependencies one by one based on build errors would be very time-consuming and frustrating, so of course we scripted most of it. It's in a rough stage, though.

Requirements:

• scripts from https://github.com/xcp-ng/xcp/tree/master/scripts
• source RPMs from Xenserver's source ISO
• a local copy of all RPMs and source RPMs from all CentOS 7 versions (and EPEL ideally, those for the latter we use much less packages so picking them manually is feasible). I download them all to the same directory to avoid duplication. If you know exactly what version of CentOS you need, you can simplify this a lot.

# *** CentOS ***
cd /path/to/local/centos/
mkdir -p SRPMS/
# built RPMs for old relases
rsync -avP rsync://mirror.nsc.liu.se/centos-store/{7.2.1511,7.3.1611,7.4.1708,7.5.1804}/{os,updates}/x86_64/Packages/*.rpm . --exclude "*i686*"
# built RPMs for current release
rsync -avP rsync://mirror.nsc.liu.se/CentOS/7/{os,updates}/x86_64/Packages/*.rpm . --exclude "*i686*"
# source RPMs for old releases
rsync -avP rsync://mirror.nsc.liu.se/centos-store/{7.2.1511,7.3.1611,7.4.1708,7.5.1804}/{os,updates}/Source/SPackages/*.rpm ./SRPMS
# source RPMs for current release
rsync -avP rsync://mirror.nsc.liu.se/CentOS/7/{os,updates}/Source/SPackages/*.rpm ./SRPMS
# store relationship between SRPMs and RPMs. Adapt path of output file.
for f in $(ls *.rpm|grep -v "\.i686\."); do echo "$f,$(rpm -qp $f --qf '%{sourcerpm}')"; done > ~/Vates/git/xcp/data/centos_rpm_sourcerpm-$(date --iso-8601).txt

# *** EPEL ***
cd /path/to/local/epel/
mkdir -p SRPMS/
rsync -avP rsync://mirror.in2p3.fr/pub/epel/7/x86_64/Packages/*/*.rpm . --exclude "*i686*"
rsync -avP rsync://mirror.in2p3.fr/pub/epel/7/SRPMS/Packages/*/*.rpm SRPMS/
# store relationship between SRPMs and RPMs. Adapt path of output file.
for f in $(ls *.rpm|grep -v "\.i686\."); do echo "$f,$(rpm -qp $f --qf '%{sourcerpm}')"; done > ~/Vates/git/xcp/data/epel_rpm_sourcerpm-$(date --iso-8601).txt

(change mirrors depending on your location).

Download build dependencies for all the XS source RPMs we have...

... using yum-builddep in a computer or container running the appropriate version of CentOS, with the appropriate media activated (if some are missing, mostly packages that we will obtain by rebuilding some of the source RPMs or by building RPMs from our own RPM git repo, ignore them for now)

#!/bin/bash

for f in *.src.rpm
do
    if [ "$(rpm -qp $f --qf '%{vendor}' 2>/dev/null)" == "(none)" ];
    then
        yum-builddep --downloadonly --downloaddir=test/ $f
    fi
done

This is not perfect, because it will omit packages that are already installed on the system the command runs on.

Sort the RPMs based on their Vendor tag...

... in order to identify packages from CentOS and packages from EPEL. At the same time, produce a list of source RPMs extracted from those RPMs.

#!/bin/bash

mkdir -p XS epel centos other

for f in *.rpm; do
    VENDOR=$(rpm -qp $f --qf "%{vendor}" 2>/dev/null)
    if [ "$VENDOR" == "(none)" -o "$VENDOR" == "Citrix Systems, Inc." -o "$VENDOR" == "Citrix" ]; then
      mv $f XS/
    elif [ "$VENDOR" == "CentOS" ]; then
      mv $f centos/
    elif [ "$VENDOR" == "Fedora Project" ]; then
      mv $f epel/
    else
      mv $f other/
    fi
done

for d in XS epel centos other; do
  cd $d
  rm -f srpms.txt
  for f in *.rpm; do
      SRPM=$(rpm -qp $f --qf "%{sourcerpm}" 2>/dev/null)
      echo $SRPM >> srpms.txt
  done
  sort -u srpms.txt -o srpms.txt
  cd ..
done

TODO: rewrite as python script and submit to the repository with the other scripts.

Get all the SRPMs listed in centos/srpms.txt and epel/srpms.txt

Loop through the lists and copy the matching files from the local copy of CentOS / EPEL source RPMs (or download them manually for the latter).

Files copied to centos/SRPMS and epel/SRPMS

Download all RPMs that match those source RPMs

"WHAT? We first downloaded the RPMs, then their matching SRPMs and now we must go back to the RPMs from those SRPMs? Aren't we caught in a loop?"

It might look so, but one source RPM can give birth to several built RPMs. Hundreds of them even in some cases (yeah, I'm talking about you, texlive!). We want our imported "builds" in koji to be complete (a build is a source RPM + all its RPMs), meaning that each source RPM is accompanied by all its x86_64 and noarch built RPMs.

cd centos
/path/to/xcp/scripts/download_centos.py . SRPMS/ ~/Vates/git/xcp/data/centos_rpm_sourcerpm-2019-02-11.txt .
cd ..
cd epel
/path/to/xcp/scripts/download_centos.py . SRPMS/ ~/Vates/git/xcp/data/epel_rpm_sourcerpm-2019-02-11.txt .

Import the builds into koji

This stage supposes that you can import packages to koji.

At this stage the important thing is to get the tags right in Koji.

In this example, we're importing build dependencies for XCP-ng "DEV", which means the next release of XCP-ng.

[samuel2@localhost centos]$ ~/Vates/git/xcp/scripts/koji/import_rpms.py
usage: import_rpms.py [-h] [--owner OWNER]
                      srpm_directory rpm_directory package_tags build_tags

We'll run this, shortly, but first I'll explain what it means:

• srpm_directory is where we put the SRPMs to be imported
• rpm_directory is where we put the matching RPMs
• package_tags are tags at the package level (smartmontools is a package, smartmontools-6.2-4.el7 is a build). As a convention we write package tags in upper case and build tags in lower case. We'll tag all CentOS 7 packages CENTOS-7 and EPEL packages EPEL-7. This is just a way for us to know what packages come from where and there's also a technical reason to allow us tagging the builds afterwards (tags of a build need to inherit a tag of the matching package). We also tag the packages with the tag that defines the release of XCP-ng. Here's it's DEV.
• build_tags are the most interesting tags because they are associated to a specific build. We'll tag builds from CentOS centos-7.6 if we've identified that all those builds come from CentOS 7.6. If they come from mixed versions (I hope that will not happen), we need first to identify which version they come from. We'll tag builds from EPEL epel-7-date-of-last-sync, for example epel-7-2019-02-11. This is because EPEL always evolves so we use the date of last sync as the version. This may change in the future. We also tag all our builds dev-base which is the base tag for all our builds in the DEV release. This automatically makes the imported packages available for building packages that depend on them.

Don't forget to create the appropriate tags first if needed.

# let's start with EPEL: fewer packages
cd epel
/path/to/xcp/scripts/koji/import_rpms.py SRPMS/ . DEV,EPEL-7 dev-base,epel-7-2019-02-11
cd ..

cd centos
/path/to/xcp/scripts/koji/import_rpms.py SRPMS/ . DEV,CENTOS-7 dev-base,centos-7.6

Rebuild packages

Now you can start rebuilding XCP-ng packages. You need to do it in the proper order so that build dependencies, such as -devel packages that result from the build of one of our packages are made available to next builds that need them (you didn't think we were done yet with build dependencies, did you?).

This will be another story, in another page of this wiki.