ICMP not working on Kubernetes even if sysctl -w net.ipv4.ping_group_range="0 2147483647"

[jerome-karabenli]

Describe the bug

Ping not working even if sysctl -w net.ipv4.ping_group_range="0 2147483647" is configured.
net.ipv4.ping_group_range is set in initContainer which have privileged role an run as root.

I tried to mount an alpine container in same pod where gatus is. I can successfully use ping in this alpine container which have same securitycontext as gatus container, is running a non root user, on uid and guid 65534 (nobody)

I tried to ping google.com

Used config in config.yaml
endpoints:

• name: TEST
  enabled: true
  url: "icmp://google.com"
  interval: 60s
  conditions:
  • "[CONNECTED] == true"
    client:
    timeout: 30s

What do you see?

Endpoint is returning not ok

What do you expect to see?

Ping success

List the steps that must be taken to reproduce this issue

Use icmp on endpoint running in Kubernetes using this endpoints config
endpoints:

• name: TEST
  enabled: true
  url: "icmp://google.com"
  interval: 60s
  conditions:
  • "[CONNECTED] == true"
    client:
    timeout: 30s

Use initContainer with root privlege using alpine image and execute this command:
sysctl -w net.ipv4.ping_group_range="0 2147483647"

Version

twinproduction/gatus:v5.7.0

Additional information

No response

[kevin7s-io]

I'm seeing the same behavior. I toyed around with passing sysctls to the securityContext and the method described by @jerome-karabenli -- both are unable to ping outside the pod.

I've search existing issues (#633, #182, #105) and I'm wondering if I'm missing something,

[h3mmy]

The issue is here https://github.com/TwiN/gatus/blob/master/client/client.go#L246

pinger.SetPrivileged(runtime.GOOS != "darwin")

This will set privileged to true on linux and need to use the privileged ping instead of the unprivileged one. See https://github.com/prometheus-community/pro-bing/blob/ac3b40f1f0a7438a429e9bf6f2bc2a94ba286e39/ping.go#L430

Linux and darwin both support NonPrivileged ping (https://pkg.go.dev/golang.org/x/net/icmp?utm_source=godoc#example-PacketConn-NonPrivilegedPing) so I would expect it to be safe to only filter for windows.

The change was made here: c423afb for issue #132 but darwin supports non-privileged pings so the windows only condition should be okay.

[TwiN]

Feel free to make a PR if you think that'll fix it!

[TwiN]

I created #748 in an attempt to address it, but I would appreciate if somebody (either @jerome-karabenli, @kevin7s-io, @h3mmy, @heathcliff26 or anybody reading this) could test it on their end and report back on whether #748 fixed it.

I've just built a container image; if you'd like to try it, pull twinproduction/gatus:experimental.

Note that the image in question is only built for linux/amd64.

[TwiN]

Works on Windows, but not on my Kubernetes cluster, even with the following configuration on the pods

          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - NET_RAW

influxdata/influxdata-docker#550 and influxdata/influxdata-docker#547 seems to have some information on what needs to be done to fix this.

Looking at containerd/containerd#6924, perhaps this will be fixed automagically too for Kubernetes 🤔

[heathcliff26]

I have tested it with podman and the experimental image works when running as root, but not in rootless mode.

I also tested running v5.10.0 as root since i didn't before, but it did not work.

So i guess the fix works, but still needs to have some privileges set.

[joryirving]

I'm currently experiencing this issue, where my config that was working in docker doesn't work in Kubernetes.

I tried a bunch of things, such as capabilities, and using the same SC I use in blackbox-exporter:

podSecurityContext:
  sysctls:
    - name: net.ipv4.ping_group_range
      value: "0 65536"

which also didn't work. I already have set

enable_unprivileged_ports = true
enable_unprivileged_icmp = true

and that doesn't appear to help either.

[h3mmy]

I created #748 in an attempt to address it, but I would appreciate if somebody (either @jerome-karabenli, @kevin7s-io, @h3mmy, @heathcliff26 or anybody reading this) could test it on their end and report back on whether #748 fixed it.

I've just built a container image; if you'd like to try it, pull twinproduction/gatus:experimental.

Note that the image in question is only built for linux/amd64.

SO sorry I missed this. I went ahead an tested the branch in #748 and it works in my k3s cluster.

This is a link to my HelmRelease: https://github.com/h3mmy/bloopySphere/blob/96329ee8e913168f11198920db4cd0f758b1ea68/cluster/apps/monitoring/gatus/app/helm-release.yaml

Important bits:

• container running as non-root
• dropped ALL capabilities
• disallow privilege escalation
• I do have an annotation to set the sysctls

annotations:
reloader.stakater.com/auto: "true"
# https://github.com/prometheus-community/pro-bing#linux
security.alpha.kubernetes.io/sysctls: net.ipv4.ping_group_range=0 2147483647

And the config I used as a test case: https://github.com/h3mmy/bloopySphere/blob/96329ee8e913168f11198920db4cd0f758b1ea68/cluster/apps/networking/traefik/external-services/nas-camelus.yaml

---

apiVersion: v1
kind: ConfigMap
metadata:
name: camelus-plexii-gatus-ep
namespace: networking
labels:
gatus.io/enabled: "true"
data:
config.yaml: |
endpoints:
- name: camelus-plexii-ping
group: infrastructure
url: icmp://${NAS_ADDRESS}
interval: 5m
ui:
hide-url: true
hide-hostname: true
conditions:
- "[CONNECTED] == true"
alerts:
- type: discord

Let me know if you'd like me to try any different arrangements for different scenarios.

[joryirving]

The experimental image resolved it for me. I didn't need the annotation.

[h3mmy]

The experimental image resolved it for me. I didn't need the annotation.

It may vary with host distribution and kernel security profiles. I'm not an expert though.

Do you have any security profiles enabled on your host? AppArmor, seccomp, SELinux, etc?

[joryirving]

I do not.

[TwiN]

The following pod security context (pod, not container) fixed the issue for me:

      securityContext:
        sysctls:
          - name: net.ipv4.ping_group_range
            value: 0 65536

Downside is that this fix is Kubernetes-specific, and releasing this as-is would break people deploying Gatus on Docker.

[joryirving]

Just wanted to revisit this, as I realized I was still on the experimental image.
I tried swapping to 5.12, but IMCP still isn't working. I've tried the securityContext but that isn't resolving it either.

I have the following on the pod:

            securityContext:
              allowPrivilegeEscalation: false
              readOnlyRootFilesystem: true
              capabilities: { drop: ["ALL"] }
              sysctls:
              - name: net.ipv4.ping_group_range
                value: 0 65536

and the following defaultPodOptions:

    defaultPodOptions:
      dnsConfig:
        options:
          - { name: ndots, value: "1" }
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        fsGroup: 65534
        fsGroupChangePolicy: OnRootMismatch
        seccompProfile: { type: RuntimeDefault }

I've tried putting the systctls in the defaultPodOptions.securityContext and commenting out the seccompProfile, but that hasn't worked either.

[enorasec]

I'm facing the same issue. I pulled the latest image (5.13.1) and configured it the sysctls. It's still not working for me. I set up the dnsutils container from k8s registry to ensure this would work on my kubernetes environment (Oracle Kubernetes Engine) with another container. I was able to show it being denied and then allowing with the sysctl securityContexts.

I also noticed when I set an endpoint to external ping (GitHub.com); it was successful with https:// but not with icmp://