Add support for hashed mode to Poudriere #751

allanjude · 2020-04-13T20:07:18Z

Requires the related patches to pkg and the ports tree

src/share/poudriere/common.sh

bdrewery · 2020-04-13T20:27:36Z

Can you also give more context for what this is?

allanjude · 2020-04-13T20:49:45Z

Can you also give more context for what this is?

When combined with freebsd/pkg#1829

This creates a pkg repo that looks like this:

#ls -al /usr/local/poudriere/data/packages/121amd64-default/All/
total 12021
drwxr-xr-x  2 root    wheel       16 Apr 13 20:06 ./
drwxr-xr-x  4 root    wheel        9 Apr 13 20:06 ../
-rw-r--r--  1 nobody  wheel   161820 Apr 13 19:27 gettext-runtime-0.20.1+ec3887d3a2.txz
lrwxr-xr-x  1 nobody  wheel       37 Apr 13 19:27 gettext-runtime-0.20.1.txz@ -> gettext-runtime-0.20.1+ec3887d3a2.txz
-rw-r--r--  1 nobody  wheel  2527876 Apr 13 19:29 gettext-tools-0.20.1_1+7101c9ffe5.txz
lrwxr-xr-x  1 nobody  wheel       37 Apr 13 19:29 gettext-tools-0.20.1_1.txz@ -> gettext-tools-0.20.1_1+7101c9ffe5.txz
-rw-r--r--  1 nobody  wheel     5828 Apr 13 19:19 indexinfo-0.3.1+1cd9c1a735.txz
lrwxr-xr-x  1 nobody  wheel       30 Apr 13 19:19 indexinfo-0.3.1.txz@ -> indexinfo-0.3.1+1cd9c1a735.txz
-rw-r--r--  1 nobody  wheel   387088 Apr 13 19:27 libtextstyle-0.20.1+6c117ad74e.txz
lrwxr-xr-x  1 nobody  wheel       34 Apr 13 19:27 libtextstyle-0.20.1.txz@ -> libtextstyle-0.20.1+6c117ad74e.txz
-rw-r--r--  1 nobody  wheel   236640 Apr 13 20:04 nano-4.8+74c3c14712.txz
lrwxr-xr-x  1 nobody  wheel       23 Apr 13 20:04 nano-4.8.txz@ -> nano-4.8+74c3c14712.txz
-rw-r--r--  1 nobody  wheel  8787688 Apr 13 20:06 pkg-1.13.99.7.l+46fd66c8a7.txz
lrwxr-xr-x  1 nobody  wheel       30 Apr 13 20:06 pkg-1.13.99.7.l.txz@ -> pkg-1.13.99.7.l+46fd66c8a7.txz
-rw-r--r--  1 nobody  wheel    35452 Apr 13 19:54 zxfer-1.1.7+2e39bad872.txz
lrwxr-xr-x  1 nobody  wheel       26 Apr 13 19:54 zxfer-1.1.7.txz@ -> zxfer-1.1.7+2e39bad872.txz

So when you install a package, it fetches the file with the hash in the URL:

# pkg install zxfer
Updating Test repository catalogue...
Test repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        zxfer: 1.1.7

Number of packages to be installed: 1

35 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching zxfer-1.1.7+2e39bad872.txz: 100%   35 KiB  35.5kB/s    00:01
Checking integrity... done (0 conflicting)
[1/1] Installing zxfer-1.1.7...
[1/1] Extracting zxfer-1.1.7: 100%

This will allow package repositories to be served from CDNs and web caches, since the unique hash in the filename will avoid the need for cache invalidation on the actual package files, and a short lifetime on the pkg meta files is all that would be required.

src/share/poudriere/common.sh

allanjude · 2020-07-17T02:58:56Z

@bapt with freebsd/pkg@36dfb48 merged into pkg, I've refreshed this patch to add a -H flag to poudriere bulk, which builds a repo using the hashed mode.

It currently implies --symlink as well, because poudriere doesn't find the already build packages during an incremental without it yet.

allanjude · 2020-07-17T03:00:09Z

The change committed to pkg is different than the original proposal (create hashed filenames during pkg create). The version that was merged to pkg is for pkg repo which does all the work in one step at the end, and requires a lot fewer changes to poudriere that way.

darkfiberiru · 2020-09-16T19:25:30Z

@allanjude Can we also have a poudriere.conf knob. Below is from my attempt I will write and test an additional commit to work with yours to do that.

# Have pkg create hashed versions of the pkg filenames with symlinks to
# original pkg names. The packagesite.yaml file will point to the hashed version
 # of these files. By using hashed pkg filenames, this allows users to lazily
 # synchronise packages without conflicting with the current packages,
 # for example using rsync or CDNs.  Once the packages are synced the much> 
# smaller meta files can then be synced. Allowing a near atomic update of repo.
 # On caching cdn this means a need to purge 2-5 files instead of all pkgs that
 # have been updated.
 #PKG_HASH=no**

darkfiberiru · 2020-09-18T17:40:41Z

@allanjude As discussed oob I will try to get a patch version that includes -H flag or poudriere.conf options and generate a new pr/reopen #786

igalic · 2020-12-21T23:24:10Z

@darkfiberiru do you still have plans to pick this up again?
it seems Allan is perpetually busy with something else

allanjude · 2021-11-01T02:20:26Z

One thing I noticed, with the new default config, the pkgs get owned by 'nobody', but the symlink's to the hashed versions are owned by root.
Is this a case of pkg repo should be run as nobody, or that we just need to do a chown after pkg repo?

I notice packagesite etc are not owned by nobody.

Creates the repo with hash-based filenames to allow use of a CDN Setting `PKG_HASH="yes"` in poudriere.conf will build a repo where all of the packages are in All/Hashed/ and the repo manifest points there. It also creates a set of symlinks in the All/ directory, but these are purely for poudriere itself, to find dependencies. The symlinks should NOT be published, only the Hashed/ directory is required. This mechanism ensures that the package files themselves can be cached by a CDN as the filename will change if the contents ever differ. The repo metadata files (those outside of All/) should be set to have a very low cache expiration, so that when a new package set is published they are updated and reflect the new packages. Technically this feature also allows "previous" versions of packages to continue to be available via the CDN, but that is a side-effect not a purposeful feature. Sponsored-by: Klara, Inc. Sponsored-by: TitanHQ

allanjude · 2024-06-29T19:39:26Z

I have refreshed this and solved the issues that have been reported (Poudriere was trying to delete the Hashed directory as it thought it was an orphaned package)

bdrewery · 2024-07-09T17:40:12Z

Looking at this today.

bdrewery

Quick changes requested

bdrewery · 2024-07-09T17:41:31Z

src/share/poudriere/common.sh

+
+	# Remount rw
+	# mount_nullfs does not support mount -u
+	umount ${UMOUNT_NONBUSY} ${MASTERMNT}/packages || \


See 43cca93
s/${UMOUNT_NONBUSY}/-n/

Although I think you can just use remount_packages.

bdrewery · 2024-07-09T17:41:48Z

src/share/poudriere/common.sh

@@ -9542,6 +9560,11 @@ build_repo() {
 			sign_pkg pubkey "${PACKAGES:?}/Latest/pkg.${PKG_EXT}"
 		fi
 	fi
+
+	# Remount ro
+	umount ${UMOUNT_NONBUSY} ${MASTERMNT}/packages || \


UMOUNT_NONBUSY to -n here too. Or better remount_packages -o ro.

bdrewery · 2024-07-09T17:46:33Z

src/share/poudriere/include/pkg.sh

@@ -417,6 +421,8 @@ delete_pkg_xargs() {
 	# Delete the package and the depsfile since this package is being deleted,
 	# which will force it to be recreated
 	{
+		# If ${pkg} is a symlink, delete the target as well
+		[ -L "${pkg}" ] && echo $(realpath "${pkg}")


Please avoid && as if it ends up being the last in a statement the statement can return non-zero with set -e. So we avoid it as a pattern because it sneaks in eventually.

# sh -c 'set -o pipefail; set -e; dolink() { [ -L / ] && true; }; dolink; echo done'; echo $? 1

bdrewery · 2024-07-09T17:48:38Z

src/man/poudriere-bulk.8

@@ -28,7 +28,7 @@
 .\"
 .\" Note: The date here should be updated whenever a non-trivial
 .\" change is made to the manual page.
-.Dd July 5, 2022
+.Dd September 26, 2022


Update to more recent given the PR idle time.

bdrewery · 2024-07-09T17:50:18Z

src/share/poudriere/common.sh

 	mkdir -p ${MASTERMNT}/tmp/packages
 	if [ -n "${PKG_REPO_SIGNING_KEY}" ]; then
 		msg "Signing repository with key: ${PKG_REPO_SIGNING_KEY}"
 		install -m 0400 "${PKG_REPO_SIGNING_KEY}" \
 			"${MASTERMNT:?}/tmp/repo.key"
 		injail ${PKG_BIN:?} repo \
+			${PKG_REPO_FLAGS} \


A goal I have is for set -u to work. Please use ${PKG_REPO_FLAGS-} here and in the next places.

bdrewery · 2024-12-09T18:00:34Z

@allanjude I can make the suggested changes myself if you are too busy. Want me to?

allanjude · 2024-12-09T22:42:52Z

@allanjude I can make the suggested changes myself if you are too busy. Want me to?

If you have time that would be great, I'd really like to get this landed so FreeBSD can switch to using it, so we can have per-release pkg repos without having to grow the mirrors

bdrewery requested changes Apr 13, 2020

View reviewed changes

src/share/poudriere/common.sh Outdated Show resolved Hide resolved

bdrewery requested changes Apr 13, 2020

View reviewed changes

src/share/poudriere/common.sh Outdated Show resolved Hide resolved

allanjude force-pushed the hashed_pkgs branch 2 times, most recently from 4d1ad4f to 668a61f Compare April 14, 2020 01:23

allanjude force-pushed the hashed_pkgs branch from 668a61f to 8f5c4bc Compare July 17, 2020 02:55

darkfiberiru mentioned this pull request Sep 15, 2020

Add new pkg repo file hash ability to poudriere.conf Default to off #786

Closed

allanjude force-pushed the hashed_pkgs branch 2 times, most recently from a59ca49 to 30c4b23 Compare October 31, 2021 19:10

allanjude force-pushed the hashed_pkgs branch from 30c4b23 to 6cd9729 Compare June 29, 2024 19:38

bdrewery requested changes Jul 9, 2024

View reviewed changes

bdrewery self-assigned this Dec 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for hashed mode to Poudriere #751

Add support for hashed mode to Poudriere #751

allanjude commented Apr 13, 2020

bdrewery commented Apr 13, 2020

allanjude commented Apr 13, 2020

allanjude commented Jul 17, 2020

allanjude commented Jul 17, 2020

darkfiberiru commented Sep 16, 2020 •

edited

Loading

darkfiberiru commented Sep 18, 2020

igalic commented Dec 21, 2020

allanjude commented Nov 1, 2021

allanjude commented Jun 29, 2024

bdrewery commented Jul 9, 2024

bdrewery left a comment

bdrewery Jul 9, 2024

bdrewery Jul 9, 2024

bdrewery Jul 9, 2024

bdrewery Jul 9, 2024

bdrewery Jul 9, 2024

bdrewery Jul 9, 2024

bdrewery commented Dec 9, 2024

allanjude commented Dec 9, 2024

Add support for hashed mode to Poudriere #751

Are you sure you want to change the base?

Add support for hashed mode to Poudriere #751

Conversation

allanjude commented Apr 13, 2020

bdrewery commented Apr 13, 2020

allanjude commented Apr 13, 2020

allanjude commented Jul 17, 2020

allanjude commented Jul 17, 2020

darkfiberiru commented Sep 16, 2020 • edited Loading

darkfiberiru commented Sep 18, 2020

igalic commented Dec 21, 2020

allanjude commented Nov 1, 2021

allanjude commented Jun 29, 2024

bdrewery commented Jul 9, 2024

bdrewery left a comment

Choose a reason for hiding this comment

bdrewery Jul 9, 2024

Choose a reason for hiding this comment

bdrewery Jul 9, 2024

Choose a reason for hiding this comment

bdrewery Jul 9, 2024

Choose a reason for hiding this comment

bdrewery Jul 9, 2024

Choose a reason for hiding this comment

bdrewery Jul 9, 2024

Choose a reason for hiding this comment

bdrewery Jul 9, 2024

Choose a reason for hiding this comment

bdrewery commented Dec 9, 2024

allanjude commented Dec 9, 2024

darkfiberiru commented Sep 16, 2020 •

edited

Loading