Skip to content

Latest commit

 

History

History
121 lines (93 loc) · 5.38 KB

module-7-policy-debugging-ebpf.md

File metadata and controls

121 lines (93 loc) · 5.38 KB

Module 7 - Policy Debugging with eBPF

Debug Policy Enforcement in the eBPF maps

  1. First, find out the node that the client pod is running on:

    NODE_NAME=$(kubectl get pod -n client -o=jsonpath='{.items[0].spec.nodeName}')
CLIENT_POD_IP=$(kubectl get pod -n client -o=jsonpath='{.items[0].status.podIP}')
echo $CLIENT_POD_IP

  2. Then utilize the privileged pod that was deployed to get a shell into the node:

    PRIV_POD=$(kubectl get pods -o=jsonpath='{.items[?(@.spec.nodeName=="'"$NODE_NAME"'")].metadata.name}')
kubectl exec -it "$PRIV_POD" -- /bin/sh

    Once in, get into the host namespace:

    chroot /host

  3. Find the cali* interface associated with the client pod IP using ip route

    Assuming as an example, if CLIENT_POD_IP is 10.244.24.179 then do:

    ip route | grep 10.244.24.179

    This might give an output like:

    10.244.24.179 dev cali0cdd47bbdcb scope link

  4. Utilize the Calico calico-bpf tool to examine Calico Cloud's BPF maps. First, find the name of the calico-node pod of the host of the client pod:

    CNX_POD=$(kubectl get pods -ncalico-system -l k8s-app=calico-node -o=jsonpath='{.items[?(@.spec.nodeName=="'"$NODE_NAME"'")].metadata.name}')

  5. Dump the counters for the relevant cali- interface:

    kubectl exec -it -ncalico-system $CNX_POD -c calico-node -- calico-node -bpf counters dump --iface cali0cdd47bbdcb

    It should probably look something like this, showing no dropped packets (yet)

     +----------+--------------------------------+---------+--------+-----+
 | CATEGORY |              TYPE              | INGRESS | EGRESS | XDP |
 +----------+--------------------------------+---------+--------+-----+
 | Accepted | by Egress gateways             |       0 |      0 | N/A |
 |          | by another program             |       0 |      0 | N/A |
 |          | by failsafe                    |       0 |      0 | N/A |
 |          | by policy                      |     371 |    186 | N/A |
 | Dropped  | by policy                      |       0 |      0 | N/A |
 |          | failed decapsulation           |       0 |      0 | N/A |
 |          | failed encapsulation           |       0 |      0 | N/A |
 |          | incorrect checksum             |       0 |      0 | N/A |
 |          | malformed IP packets           |       0 |      0 | N/A |
 |          | packets hitting blackhole      |       0 |      0 | N/A |
 |          | route                          |         |        |     |
 |          | packets with unknown route     |       0 |      0 | N/A |
 |          | packets with unknown source    |       0 |      0 | N/A |
 |          | packets with unsupported IP    |       0 |      0 | N/A |
 |          | options                        |         |        |     |
 |          | too short packets              |       0 |      0 | N/A |
 | Total    | packets                        |    2044 |    683 | N/A |
 +----------+--------------------------------+---------+--------+-----+

  6. Try to send traffic from the client pod to management-ui again:

    kubectl -n client exec -it $(kubectl get po -n client -l role=client -ojsonpath='{.items[0].metadata.name}')  -- /bin/bash -c 'curl -m3 -I http://management-ui.management-ui'

    and then look at the counters again:

     +----------+--------------------------------+---------+--------+-----+
 | CATEGORY |              TYPE              | INGRESS | EGRESS | XDP |
 +----------+--------------------------------+---------+--------+-----+
 | Accepted | by Egress gateways             |       0 |      0 | N/A |
 |          | by another program             |       0 |      0 | N/A |
 |          | by failsafe                    |       0 |      0 | N/A |
 |          | by policy                      |     596 |    297 | N/A |
 | Dropped  | by policy                      |       4 |      1 | N/A |
 |          | failed decapsulation           |       0 |      0 | N/A |
 |          | failed encapsulation           |       0 |      0 | N/A |
 |          | incorrect checksum             |       0 |      0 | N/A |
 |          | malformed IP packets           |       0 |      0 | N/A |
 |          | packets hitting blackhole      |       0 |      0 | N/A |
 |          | route                          |         |        |     |
 |          | packets with unknown route     |       0 |      0 | N/A |
 |          | packets with unknown source    |       0 |      0 | N/A |
 |          | packets with unsupported IP    |       0 |      0 | N/A |
 |          | options                        |         |        |     |
 |          | too short packets              |       0 |      0 | N/A |
 | Total    | packets                        |    3275 |   1098 | N/A |
 +----------+--------------------------------+---------+--------+-----+

  7. All of the relevant rules can be viewed as well if needed:

    kubectl exec -it -ncalico-system $CNX_POD -c calico-node -- calico-node -bpf policy dump cali0cdd47bbdcb all

➡️ Module 8 - Cleanup

⬅️ Module 6 - Switching to eBPF Dataplane

↩️ Back to Main