Improve apt-transport stdout

[fepitre]

I fill this issue as a TODO. When fetching in-toto metadata of a unreproducible package, the APT output looks not very user friendly:

$ apt reinstall bash
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  bash-doc
The following packages will be upgraded:
  bash
1 upgraded, 0 newly installed, 0 to remove and 620 not upgraded.
Need to get 1,417 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Get:1 intoto://ftp.fr.debian.org/debian bullseye/main amd64 bash amd64 5.1-2+b1 [1,417 kB]
80% [1 bash 1,417 kB]Prepare in-toto verification for '/var/cache/apt/archives/partial/bash_5.1-2+b1_amd64.deb'
Create verification directory '/tmp/tmpcdvpjfuz'
Request in-toto metadata from 2 rebuilder(s) (apt config)
Request in-toto metadata from https://debian.notset.fr/rebuild/sources/bash/5.1-2+b1/metadata
Successfully downloaded in-toto metadata 'rebuild.8deb0bef.link' from rebuilder 'https://debian.notset.fr/rebuild/'
Request in-toto metadata from https://qubes.notset.fr/rebuild/deb/r4.1/vm/sources/bash/5.1-2+b1/metadata
Could not retrieve in-toto metadata from rebuilder 'https://qubes.notset.fr/rebuild/deb/r4.1/vm/', reason was: server response: 404
Copy final product to verification directory
Load in-toto layout '/var/lib/intoto/root.layout' (apt config)
Load in-toto layout key(s) '['9fa64b92f95e706bf28e2ca6484010b5cdc576e2']' (apt config)
Use gpg keyring '/var/lib/intoto/gnupg' (apt config)
Run in-toto verification
In-toto verification for '/var/cache/apt/archives/partial/bash_5.1-2+b1_amd64.deb' failed, reason was: 'DISALLOW *.deb' matched the following artifacts: ['bash_5.1-2+b1_amd64.deb']
Full trace for 'expected_materials' of item 'verify-reprobuilds':
Available materials (used for queue):
['bash_5.1-2+b1_amd64.deb']
Available products:
['bash_5.1-2+b1_amd64.deb']
Queue after 'MATCH *.deb WITH PRODUCTS FROM rebuild':
['bash_5.1-2+b1_amd64.deb']

Err:1 intoto://ftp.fr.debian.org/debian bullseye/main amd64 bash amd64 5.1-2+b1
  In-toto verification for '/var/cache/apt/archives/partial/bash_5.1-2+b1_amd64.deb' failed, reason was: 'DISALLOW *.deb' matched the following artifacts: ['bash_5.1-2+b1_amd64.deb']\nFull trace for 'expected_materials' of item 'verify-reprobuilds':\nAvailable materials (used for queue):\n['bash_5.1-2+b1_amd64.deb']\nAvailable products:\n['bash_5.1-2+b1_amd64.deb']\nQueue after 'MATCH *.deb WITH PRODUCTS FROM rebuild':\n['bash_5.1-2+b1_amd64.deb']\n
E: Failed to fetch intoto://ftp.fr.debian.org/debian/pool/main/b/bash/bash_5.1-2+b1_amd64.deb  In-toto verification for '/var/cache/apt/archives/partial/bash_5.1-2+b1_amd64.deb' failed, reason was: 'DISALLOW *.deb' matched the following artifacts: ['bash_5.1-2+b1_amd64.deb']\nFull trace for 'expected_materials' of item 'verify-reprobuilds':\nAvailable materials (used for queue):\n['bash_5.1-2+b1_amd64.deb']\nAvailable products:\n['bash_5.1-2+b1_amd64.deb']\nQueue after 'MATCH *.deb WITH PRODUCTS FROM rebuild':\n['bash_5.1-2+b1_amd64.deb']\n
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

The configuration and root.layout used can be found in https://github.com/fepitre/package-rebuilder#configure-apt-transport-in-toto. For this package bash the corresponding rebuild log shows that checksums comparison failed with the original. The produced metadata can be found in https://debian.notset.fr/rebuild/sources/bash/5.1-2%2Bb1/.

We would need to figure out the best way to present failing in-toto test with respect to a fail because of no metadata at all like e.g.:

In-toto verification for '/var/cache/apt/archives/partial/mc_3%3a4.8.26-1_amd64.deb' failed, reason was: Step 'rebuild' requires '1' link metadata file(s), found '0'.

Unrelated remark: we have made a policy choice to not fail on checksums verification because that would help user to identify a package being unreproducible/having possibly an issue.

[fepitre]

@lukpueh if you agree, I'm ok to be assigned to it of course :)

[lukpueh]

This is very much appreciated, @fepitre, especially if you volunteer to work on it! :P

[lukpueh]

IIRC I wanted send messages with 1xx (informational) status code from the intoto transport to apt, in order to have apt present the message nicely to the user, but I think it did not work as expected.

So for a live demo at MiniDebConf, I ended up formatting/highlighting the Python log messages (see 660f622), which is rather quick and dirty.