Avoid the use of python with metrics-per-process

[rockandska]

Is there any particular reason to use python instead of ruby for metrics-per-process ?

I try to create a minimal docker image for sensu-client and need to had python only for this metric and will be great if this metric could be write in Ruby.

Regards,

[majormoses]

I can't think of any reason why it can not be done in ruby. I would certainly be open to accepting a PR to write a ruby equivalent.

[rockandska]

Would you like the ruby version to be cross platform compatible due to the use of psutil inside python version ?
Unfortunately, after taking a look around, the only portage to ruby was abandoned 2 years ago

Edit: in the meantime, i compiled this check with pyinstaller under alpine (~3.5Mb) for a gain of ~40Mb on the image size due to the fact than python is not required anymore.

[majormoses]

Hmm I will have to do some digging if there is another lib that is being maintained that provides similar functionality. ping @eheydrick you aware of anything off the top of your head?

[majormoses]

The TL;DR version:

• Yes we should see if we can write a ruby equivalent if possible. It would not replace it but be an option for those wishing to avoid python for one of numerous reasons ranging from religious ideology to reducing the footprint of sensu setup.
• There is a reasonable path forward for trying to remove checks of a particular language without splitting out into multiple repos.
• I do not see us splitting the checks out purely on language at this point. The notable exception are python library which is used to allow people to write python plugins with the same ease as their ruby cousins. I would support there being similar libraries but there needs to be active enough members wanting to maintain those for it to work. The python sensu user base is seeing a resurgence but was left largely unmaintained for several years.
• I'd love to see a write up that we can point users to use pyinstaller to compile the check for deployment in a container without the python runtime.

The longer Version

I have been thinking about this not from the sense of how to implement this metrics script in ruby but from a broader perspective of how we can optionally remove any checks/dependencies of a particular language to reduce the footprint.

The best I can come up with is to use a ruby extension and based on the presence of a particular ENV var we could easily remove the unwanted language checks therefore reducing the size. The other option which I don't particularly like (at the moment but could be very attractive in future) is to split out the non ruby checks into their own repositories and use the language native package managers. This comes with a LOT of initial and maintenance cost that I do not see the few maintainers we have or sensu inc having the ability to pay that.

This might change with the next generation of sensu being written in golang as it has a very different asset management solution which is still yet to be realized. That being said unless there is a surge of new maintainers with the platform and language expertise it is far off enough that it's not really worth considering.

I realize that I am pretty partial to ruby and some of my viewpoints reflect as that is what I have been primarily using for the last few years professionally. I don't see the real draw to shaving a few kbs here and there. Even in the practical context of containerization the metric script is under 5 kb. Now requiring the actual runtime is a different story; depending on what you need it could be a few hundred mbs. There is value in that but it comes at a cost and unless someone is willing to pay it I just don't see that happening.

Sorry for sounding like a broken record but the bottom line is that we do not have enough maintainers to embark on such an undertaking to focus that much effort on a minority of use cases. I think in time this will naturally evolve and become more important.

[majormoses]

This is probably the closest: https://github.com/djberg96/sys-proctable I have not really looked through it enough to know really what things are missing.

[rockandska]

First, thanks for your long answer.

I'd love to see a write up that we can point users to use pyinstaller to compile the check for deployment in a container without the python runtime.

I personally proceeded like this inside my Dockerfile (the ldd fork for alpine and process came from here):

# Create an executable of metrics-per-process.py to avoid the installation of python in the final stage ( ~4Mo instead of ~40Mo )
# https://github.com/mrasband/pyinstaller-alpine
ADD tmp/ldd /tmp/ldd
RUN chmod a+x /tmp/ldd \
    && PATH="/tmp:$PATH" \
    && pip install psutil backport_collections enum \
    && curl -SL --progress-bar --fail -o - \
        "https://github.com/pyinstaller/pyinstaller/releases/download/v${PYINSTALLER_VERSION}/PyInstaller-${PYINSTALLER_VERSION}.tar.gz" \
        | tar -zx -C /tmp \
    && cd /tmp/PyInstaller-${PYINSTALLER_VERSION}/bootloader \
    && python ./waf configure --no-lsb all \
    && pip install .. \
    && cd /tmp \
    && cp /usr/local/bundle/gems/sensu-plugins-process-checks-*/bin/metrics-per-process.py . \
    && pyinstaller --exclude-module PyInstaller --exclude-module pycrypto --exclude-module httplib --exclude-module pyexpat --exclude-module email --exclude-module ftplib --exclude-module getpass --exclude-module urllib --exclude-module uu --exclude-module xml --noconfirm --onefile --clean metrics-per-process.py \
    && cp /tmp/dist/metrics-per-process /usr/local/bundle/gems/sensu-plugins-process-checks-*/bin/metrics-per-process.py \
    && apk del build-deps \
    && rm -rf /tmp/*

This might change with the next generation of sensu being written in golang as it has a very different asset management solution which is still yet to be realized. That being said unless there is a surge of new maintainers with the platform and language expertise it is far off enough that it's not really worth considering.

golang is certainly a great langage, but if you build all the plugins with their own library versions, i think it will add too many Kb unnecessary

This is probably the closest: https://github.com/djberg96/sys-proctable I have not really looked through it enough to know really what things are missing.

Forgot why i didn't retain it at my first candidate for my ruby transition , but if i'm correct, i think it was because many functions present inside psutil wasn't builtin inside sys-proctable ( number of tcp connections for exemple ) and it was necessary to rewrite many functions.

[rockandska]

Another thing disturbing with plugins than use "python", the need to install the python dependencies manually.
There is no requirements.txt or setup.py inside repositories who use python, and sensu-install will not use them even if they were present

[majormoses]

Another thing disturbing with plugins than use "python", the need to install the python dependencies manually.

Agreed, I have considered having a ruby extension that would be used to install python requirements during install time.