Import RPM build dependencies from CentOS and EPEL

Build dependencies from CentOS and EPEL

The source RPMs provided by the Xenserver project do not include many needed build dependencies from CentOS and EPEL.

One can guess that Citrix has internal frozen repositories for those packages and does not communicate about their contents (we can mostly infer it from RPM build dependencies), nor says what exact versions of the packages are used (we have to guess). It looks like they stick to a given version of CentOS (e.g. 7.2) for some time, maybe with a few exceptions.

Xenserver 7.6 still relied mostly on the old CentOS 7.2 so we can use the frozen CentOS 7.2 repositories from vault.centos.org (version 8 should bring fresh air with more recent packages from CentOS). However EPEL packages (they are only a few, thankfully) are not frozen so there's a risk we'd use a version that is "too" recent and brings incompatibilities.

Add the missing dependencies to Koji

Hunting the missing dependencies one by one based on build errors would be very time-consuming and frustrating, so of course we scripted most of it. It's in a rough stage, though.

Requirements:

• scripts from https://github.com/xcp-ng/xcp/tree/master/scripts
• source RPMs from Xenserver's source ISO
• a local copy of all RPMs and source RPMs from all CentOS 7 versions (and EPEL ideally, those for the latter we use much less packages so picking them manually is feasible). I download them all to the same directory to avoid duplication. If you know exactly what version of CentOS you need, you can simplify this a lot.

# *** CentOS ***
cd /path/to/local/centos/
mkdir -p SRPMS/
# built RPMs for old relases
rsync -avP rsync://mirror.nsc.liu.se/centos-store/{7.2.1511,7.3.1611,7.4.1708,7.5.1804}/{os,updates}/x86_64/Packages/*.rpm . --exclude "*i686*"
# built RPMs for current release
rsync -avP rsync://mirror.nsc.liu.se/CentOS/7/{os,updates}/x86_64/Packages/*.rpm . --exclude "*i686*"
# source RPMs for old releases
rsync -avP rsync://mirror.nsc.liu.se/centos-store/{7.2.1511,7.3.1611,7.4.1708,7.5.1804}/{os,updates}/Source/SPackages/*.rpm ./SRPMS
# source RPMs for current release
rsync -avP rsync://mirror.nsc.liu.se/CentOS/7/{os,updates}/Source/SPackages/*.rpm ./SRPMS
# store relationship between SRPMs and RPMs. Adapt path of output file.
for f in $(ls *.rpm|grep -v "\.i686\."); do echo "$f,$(rpm -qp $f --qf '%{sourcerpm}')"; done > ~/Vates/git/xcp/data/centos_rpm_sourcerpm-$(date --iso-8601).txt

# *** EPEL ***
cd /path/to/local/epel/
mkdir -p SRPMS/
rsync -avP rsync://mirror.in2p3.fr/pub/epel/7/x86_64/Packages/*/*.rpm . --exclude "*i686*"
rsync -avP rsync://mirror.in2p3.fr/pub/epel/7/SRPMS/Packages/*/*.rpm SRPMS/
# store relationship between SRPMs and RPMs. Adapt path of output file.
for f in $(ls *.rpm|grep -v "\.i686\."); do echo "$f,$(rpm -qp $f --qf '%{sourcerpm}')"; done > ~/Vates/git/xcp/data/epel_rpm_sourcerpm-$(date --iso-8601).txt

(change mirror depending on your location).

Download build dependencies for all the source RPMs we have...

... using yum-builddep (if some are missing, mostly packages that we will obtain by rebuilding some of the source RPMs or by building RPMs from our own RPM git repo, ignore them for now)

#/bin/bash

for f in *.src.rpm
do
    if [ "$(rpm -qp $f --qf '%{vendor}' 2>/dev/null)" == "(none)" ];
    then
        yum-builddep --downloadonly --downloaddir=test/ $f
    fi
done

Sort the RPMs based on their Vendor tag...

... in order to identify packages from CentOS and packages from EPEL. At the same time, produce a list of source RPMs extracted from those RPMs.

/bin/bash

mkdir -p epel
mkdir -p centos
mkdir -p other

for f in *.rpm; do
    VENDOR=$(rpm -qp $f --qf "%{vendor}" 2>/dev/null)
    if [ "$VENDOR" == "CentOS" ]; then
      mv $f centos/
    elif [ "$VENDOR" == "Fedora Project" ]; then
      mv $f epel/
    else
      mv $f other/
    fi
done

cd epel
rm -f srpms.txt
for f in *.rpm; do
    SRPM=$(rpm -qp $f --qf "%{sourcerpm}" 2>/dev/null)
    echo $SRPM >> srpms.txt
done
sort -u srpms.txt -o srpms.txt
cd ..

cd centos
rm -f srpms.txt
for f in *.rpm; do
    SRPM=$(rpm -qp $f --qf "%{sourcerpm}" 2>/dev/null)
    echo $SRPM >> srpms.txt
done
sort -u srpms.txt -o srpms.txt
cd ..

• Get all the SRPMs listed in centos/srpms.txt and epel/srpms.txt

This stage is not well scripted at the moment. Looped through the lists and copy the matching files from the local copy of CentOS / EPEL source RPMs (or download them manually for the latter).

Files copied to centos/SRPMS and epel/SRPMS

• Download all RPMs that match those source RPMs

"WHAT? We first downloaded RPMs, then their matching SRPMs and now we must go back to the RPMs from those SRPMs? Aren't we caught in a loop?"

It might look so, but one source RPM can give birth to several built RPMs. Hundreds of them even in some cases (yeah, I'm talking about you, texlive!). We want our imported "builds" in koji to be complete (a build is a source RPM + all its RPMs).

cd centos
download_centos.py . SRPMS/ ~/Vates/git/xcp/data/epel_rpm_sourcerpm-2019-01-30.txt .