Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debian 5.9.1 / 5.9.2 package build candidate #18

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

zebity
Copy link

@zebity zebity commented Sep 23, 2019

Hi Debian Packagers,

I am submitting this pull to help get ball moving on 5.9.1 debian / ubuntu build.
I have tested this on Ubuntu 19.04 got to point of build full set of packages (core, utils, dev, dbg, source, libcollectdclient1, libcollectdclient-dev).

I have also installed core into machine and verified that runs ok.

Removed patches for liblvm2app deprecation & i2c dectectiom as these build via upstream.

I had issues with testing of configure result which then force build failure (debian/rules) so update to let build continue.

changeling is just brief summary. There are been quite a few upstream updates into 5.9.1.

Package build fails is you use tar ball.
To build package successfully I had to:

  1. clone collectd / collectd
  2. git checkout (branch) collectd-5.9
  3. build.sh , configure, make dist
  4. copy generated compressed tar
  5. git clone pkg-debian/collectd (my master with tag 5.9.1-1 added)
  6. extract tar into pkg-debian: bzcat collectd-5.9.1.6.g1f778e9+.tar.bz2 | tar xvf - -C pkg-debian --strip-components=1
  7. Run build: dpkg-buildpackage -us -uc

@mfournier & @tokkee , your review and feedback would be appreciated so we can make some progress on next collectd release.

Thanks

Zebity

@zebity zebity changed the title Debian 5.9.1 Package Build Debian 5.9.1 package build candidate Sep 23, 2019
@zebity zebity changed the title Debian 5.9.1 package build candidate Debian 5.9.2 package build candidate Sep 24, 2019
@zebity
Copy link
Author

zebity commented Sep 24, 2019

Responding here to @mrunge recommendation to move to 5.9.2 (see #17) due to defects in 5.9.1.

I have updated changelog to reflect 5.9.2 (and title of this pull request) build and added / push new 5.9.2-1 tag in my master and done git checkout to candidate collectd-5.9 branch.

All build oks (based on collectd-5.9 branch) if I do build when I have code in pkg-debian directory.

Fails when I try to build from ../collectd_5.9.2.orig.tar.gz generated from my collectd master (with updated version-gen.sh to reflect 5.9.2 target in collectd-5.9 branch), but this pkg-debian issue not a collectd one.

@bzed - lets see how we go towards getting new 5.9.2 package out. Any chance you could help with working out why package build fails when using ../collectd_5.9.2.orig.tar.gz based package build ?
I looked at various forked pkg-debian trees, but they are all well behind master so not much help with detective work...

@zebity zebity changed the title Debian 5.9.2 package build candidate Debian 5.9.1 / 5.9.2 package build candidate Sep 24, 2019
@zebity
Copy link
Author

zebity commented Sep 24, 2019

Hi Packagers,

I have done further testing and can confirm that collectd-5.9 branch is working on Ubuntu 19.04
I did debian pkg build from collect-5.9 branch using candidate 5.9.1/5.9.2 package build tree.

I have tested: cpu, df, disk, entropy, interface, irq, memory, process, retool, swap, virt (+ ExtraStats) and write_log.

[2019-09-24 23:17:11] plugin_load: plugin "cpu" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "df" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "disk" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "entropy" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "interface" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "irq" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "load" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "memory" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "processes" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "rrdtool" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "swap" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "users" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "write_log" successfully loaded.
[2019-09-24 23:17:11] plugin_load: plugin "virt" successfully loaded.
[2019-09-24 23:17:11] Systemd detected, trying to signal readiness.
[2019-09-24 23:17:11] virt plugin: reader virt-0 initialized
[2019-09-24 23:17:11] Initialization complete, entering read-loop.
[2019-09-24 23:17:11] write_log values:
blue.entropy.entropy 3780 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.interface-eno4.if_dropped.rx 0 1569331031^M
blue.interface-eno4.if_dropped.tx 0 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.interface-eno4.if_packets.rx 2303995 1569331031^M
blue.interface-eno4.if_packets.tx 765889 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.interface-vnet26.if_errors.rx 0 1569331031^M
...
...
blue.entropy.entropy 3780 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.interface-eno4.if_dropped.rx 0 1569331031^M
blue.interface-eno4.if_dropped.tx 0 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.interface-eno4.if_packets.rx 2303995 1569331031^M
blue.interface-eno4.if_packets.tx 765889 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.interface-vnet26.if_errors.rx 0 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue_work04-kube-ubuntu-18_04_d947f4b7-fc40-4a88-badd-f6deb25f11ff.virt.domain_state.state 5 1569331031^M
blue_work04-kube-ubuntu-18_04_d947f4b7-fc40-4a88-badd-f6deb25f11ff.virt.domain_state.reason 0 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.ps_cputime.user 6948230000000 1569331031^M
blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.ps_cputime.syst 4887520000000 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.virt_cpu_total 45467950000000 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.memory-total 17179869184 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.processes.ps_state-blocked 1 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue.processes.fork_rate 358357 1569331031^M

[2019-09-24 23:17:11] write_log values:
blue_master-od-macos-10_13_a42f4b94-bd9a-451a-a1a8-96d6aea1097a.virt.cpu_affinity-vcpu_0-cpu_0 1 1569331031^M

So clearly virt plug in work mostly by @rpv-tomsk has been pulled in ok and wrapper work has got ride of need to turn off W-all flag and rrdtool fix has been merged.

Not getting extra virt metadata, but need to look into my config for that I believe (problem is not with packaging).

@mrunge , now I know how to do do the packaging, just let me know when you would like to test 5.10 branch ;-)

Final comment: I am able to build with or without libstatgrab . I am sure I read somewhere that libstatgrab was only valuable for freebsd build, but cannot find where I read this...

@zebity
Copy link
Author

zebity commented Oct 25, 2019

@bzed @mrunge @rpv-tomsk

I have spent quite a bit of morning re-testing and checking debian build. In particular if problem were due to my failure to understand how to get package built or just the nature of the process.

After a bit more testing and reading I have concluded that the rather clunky process is just how it is at the moment unless we put some additional automation in place.

I have together summary on my blog page:

Here are my build steps:

mkdir build-collectd
cd build-collectd/
git clone https://github.com/collectd/collectd.git
cd collectd/
git checkout collectd-5.9.2
git tag -a collectd-5.9.2-1 -m "Add 5.9.2 tag"
./build.sh
./configure
make dist
cd ..
ln -s collectd/collectd-5.9.2.1.tar.bz2 collectd_5.9.2.orig.tar.bz2
git clone https://github.com/zebity/pkg-debian.git
bzcat collectd_5.9.2.orig.tar.bz2 | tar xvf - -C pkg-debian --strip-components=1
cd pkg-debian/
dpkg-buildpackage -us -uc

The fiddlely bit is the git describe (tag annotations) file and version naming approach, which can and likely will diverge if updates get out of sync as the collectd and pkg_debian are on different branches.

As part of collectd release process it would be helpful if some CI automation could be triggered to also do a debian build to we can see if it breaks... and hopefully automatically flowing into debian release through with main collectd release.

@mfournier & @tokkee , another gentle call to ask if you could please help with getting 5.9.2 flowing into debian / ubuntu stream as defects have been sorted and likely 5.10 will be following in the not to far future, so we need to get some process going to help with currency.

@zebity
Copy link
Author

zebity commented Nov 10, 2019

@mfournier @tokkee package rules update to add additional flags (CFLAGS += -Wno-error=address-of-packed-member -Wno-stringop-truncation) for GCC V9 (now part used in Ubuntu 19.10 distribution).

This addresses: collectd/collectd#3341 for Ubuntu / Debian and ensure packaging is ready for next GCC version.

@octo @mrunge @sunkuranganath if any of you have personal contact with @tokkee (as per collectd/collectd#3223 release thread discussion) could you see how we can get updates done to get Debian/Ubuntu updated beyond current ancient and now not working 5.8 package that keeps sitting unchanged while all around it is evolving. Any such efforts would be appreciated.

@bzed
Copy link
Contributor

bzed commented Nov 10, 2019

@zebity I'm preparing an upload at the moment. But I've taken all old versions and created a proper new repository here: https://salsa.debian.org/debian/pkg-collectd - the common place where packages in debian are being maintained.

I've merged your changes, added some fixes. The CI is building right now, as soon as it passes, I'll upload it.

@zebity
Copy link
Author

zebity commented Nov 10, 2019

@bzed that is great to hear 👂, if we manage to get this to flow through I will be breaking out the Champagne !

Thank you very much. BTW I am happy to help with this go avoid having all the work going to single set of hands.

If your build is based on 5.9.2 and you are including 19.10 Ubuntu target, then we will have to get pull requests (collectd/collectd#3342) I put in for for sysctl related fix.

Likely do:
5.9.2 up to Ubuntu 19.04 &
5.10.0 for 19.10

Again thanks for persisting.

@bzed
Copy link
Contributor

bzed commented Nov 10, 2019

@zebity I don't care much about Ubuntu, but @cpaelzer might be interested :)

@bzed
Copy link
Contributor

bzed commented Nov 11, 2019

@zebity and regarding help - that would be appreciated. You could get a salsa.debian.org guest account and send pull requests there, I'm happy to merge.

@zebity
Copy link
Author

zebity commented Nov 11, 2019

@bzed thanks for pointer (and registration underway) ;-)

@cpaelzer fyi, I have been beating my head against wall trying to get updated Ubuntu package for collectd since I raised this defect: collectd/collectd#3143 (in Apr 19) and then came across this: collectd/collectd#3153

Now that it seems there is some headway on getting new debian package going could we please get this flowing into Ubuntu as collectd package is well and truely out of date with current Ubuntu release cycle.

Cheers,

Zebity

Copy link
Member

@mrunge mrunge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you, looks good to me.

@zebity
Copy link
Author

zebity commented Nov 11, 2019

@bzed @cpaelzer not sure why but work on debian sides appears to have kicked off flow into Ubuntu stream: https://code.launchpad.net/ubuntu/+source/collectd
So even if you don't care @bzed , good things appear to be happening ;-)
@mrunge thanks for approval.
If we can get 5.9.2 , then I expect 5.10.X with further fixes should be able to get going shortly.

Thanks again to all for help.

@cpaelzer
Copy link

Thanks @bzed / @mrunge for the upload of 5.9.2!

@zebity all code flows into the link you found it essentially imports all Ubuntu/Debian uploads into git. But it doesn't mean it will appear in Focal.

The Delta Ubuntu currently carries is all about FTBFS with newer toolchains.

See:
https://git.launchpad.net/ubuntu/+source/collectd/diff/?id=007a913e936752e88691a1aa84c2c1a0a96be6c7&id2=6e490489627771808d18b0d2a6c5f5dea4c18fed
And all FTBFS fixes in:
https://git.launchpad.net/ubuntu/+source/collectd/tree/debian/patches?h=ubuntu/focal-devel

I'm giving this 5.9.2 a build try in Focal right now. If it builds fine we can make it a sync again asap. If it doesn't then I expect Debian to break as well sooner or later and we'd need either the fixes I mentioned or directly 5.10 in Debian to fix it for good.

@cpaelzer
Copy link

Build failed in focal, from the errors it seems at least https://git.launchpad.net/ubuntu/+source/collectd/diff/?id=007a913e936752e88691a1aa84c2c1a0a96be6c7&id2=6e490489627771808d18b0d2a6c5f5dea4c18fed will be needed.

That would be glibc 2.30 which Debian still is on 2.29 so you are ok for now.

@cpaelzer
Copy link

Heads up, going forward in glibc and compiler versions this is what you'll need (that works In Ubuntu/focal right now on top of this new upload here):

  • CFLAGS += -Wno-error=address-of-packed-member -Wno-stringop-truncation
  • CFLAGS += -Wno-error=address-of-packed-member -Wno-stringop-truncation -Wno-cpp -Wno-error=format-truncation

@cpaelzer
Copy link

Final FYI, then I'll stay silent :-)
I started a merge of this into Ubuntu here

@zebity
Copy link
Author

zebity commented Nov 17, 2019

@bzed @cpaelzer @mrunge

I have done updates to align this pull request with work done by @bzed over on debian repository: https://salsa.debian.org/debian/pkg-collectd
This includes: adding mqtt (mosquitto), moving to python3, removing my configure hack (need to disable gpu plugin to avoid configure failure), NEWS rather than NEWS.debian.

I have built on Ubuntu 19.10 with GCC 9.2.1. using collectd 5.9.2 with additional sysctl/sysctlbyname update: collectd/collectd#3342

At this point I do not see any further value in working on debian/ubuntu packaging from here and will do any updates via debian tree.

Many thanks to @bzed & @cpaelzer for getting updates flowing through to debian and ubuntu trees.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants