Weave-net CrashLoopBackOff issue

[dynam1]

When we tried to deploy the kube-system with weave-net, the status of weave-net pod was CrashLoopBackOff. The events log of that weave-net pod is as follows.

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m17s                default-scheduler  Successfully assigned kube-system/weave-net-n986x to dynam-demo
  Normal   Pulling    2m17s                kubelet            Pulling image "docker.io/weaveworks/weave-kube:2.8.1"
  Normal   Pulled     2m14s                kubelet            Successfully pulled image "docker.io/weaveworks/weave-kube:2.8.1" in 2.696012035s
  Normal   Created    2m14s                kubelet            Created container weave-init
  Normal   Started    2m14s                kubelet            Started container weave-init
  Normal   Pulling    2m11s                kubelet            Pulling image "docker.io/weaveworks/weave-npc:2.8.1"
  Normal   Started    2m6s                 kubelet            Started container weave-npc
  Normal   Pulled     2m6s                 kubelet            Successfully pulled image "docker.io/weaveworks/weave-npc:2.8.1" in 5.100046682s
  Normal   Created    2m6s                 kubelet            Created container weave-npc
  Normal   Created    83s (x4 over 2m11s)  kubelet            Created container weave
  Normal   Started    83s (x4 over 2m11s)  kubelet            Started container weave
  Normal   Pulled     83s (x4 over 2m12s)  kubelet            Container image "docker.io/weaveworks/weave-kube:2.8.1" already present on machine
  Warning  BackOff    82s (x5 over 2m4s)   kubelet            Back-off restarting failed container

[bodom0015]

The events you've included seem to show that each container started up successfully, and then failed slightly later at runtime:

  Normal   Created    2m14s                kubelet            Created container weave-init
  Normal   Started    2m14s                kubelet            Started container weave-init
  Normal   Started    2m6s                 kubelet            Started container weave-npc
  Normal   Created    2m6s                 kubelet            Created container weave-npc
  Normal   Created    83s (x4 over 2m11s)  kubelet            Created container weave
  Normal   Started    83s (x4 over 2m11s)  kubelet            Started container weave

We can see a "Created" and "Started" event for each container, so the Pod should have logs that we can view as well. We can also see (from the x4 bit) that the weave container within the Pod is the one that is restarting due to errors.

Would you be willing to include the output of the weave container logs as well?

You should be able to retrieve the logs using the following:

kubectl logs -f -n kube-system $(kubectl get pods -n kube-system | grep weave-net | awk '{print $1}') -c weave

If the container has already restarted, you can retrieve the last set of logs from the restarted container with --previous:

kubectl logs --previous -f -n kube-system $(kubectl get pods -n kube-system | grep weave-net | awk '{print $1}') -c weave

[dynam1]

The log is
Network 10.244.0.0/16 overlaps with existing route 10.244.0.0/24 on host

The kubectl get pods result is

NAME                                                   READY   STATUS             RESTARTS   AGE
...
weave-net-9q2x4                                        1/2     CrashLoopBackOff   5          5m1s

[bodom0015]

Once you have the pod NAME, you can fetch the logs with kubectl logs -f <pod_name>:

kubectl logs -n kube-system -f weave-net-9q2x4 -c weave

For me (note that my pod name is different), I was unable to repeat these steps and weave appears to start:

ubuntu@lambert-test:~/kubeadm-bootstrap$ sudo kubectl logs -f weave-net-v7h5q -n kube-system -c weave
sudo: unable to resolve host lambert-test
DEBU: 2021/03/04 23:26:50.050332 [kube-peers] Checking peer "5e:2a:ec:84:96:b2" against list &{[]}
Peer not in list; removing persisted data
INFO: 2021/03/04 23:26:50.404511 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true http-addr:127.0.0.1:6784 ipalloc-init:consensus=0 ipalloc-range:10.244.0.0/16 metrics-addr:0.0.0.0:6782 name:5e:2a:ec:84:96:b2 nickname:lambert-test no-dns:true no-masq-local:true port:6783]
INFO: 2021/03/04 23:26:50.404673 weave  2.8.1
INFO: 2021/03/04 23:26:50.687380 Re-exposing 10.32.0.1/12 on bridge "weave"
INFO: 2021/03/04 23:26:50.702577 Bridge type is bridged_fastdp
INFO: 2021/03/04 23:26:50.702594 Communication between peers is unencrypted.
INFO: 2021/03/04 23:26:50.848174 Our name is 5e:2a:ec:84:96:b2(lambert-test)
INFO: 2021/03/04 23:26:50.848232 Launch detected - using supplied peer list: []
INFO: 2021/03/04 23:26:50.848358 Using "no-masq-local" LocalRangeTracker
INFO: 2021/03/04 23:26:50.848390 Checking for pre-existing addresses on weave bridge
INFO: 2021/03/04 23:26:50.848565 weave bridge has address 10.32.0.1/12
INFO: 2021/03/04 23:26:50.850695 [allocator 5e:2a:ec:84:96:b2] No valid persisted data
INFO: 2021/03/04 23:26:50.921800 Error checking version: Get "https://checkpoint-api.weave.works/v1/check/weave-net?arch=amd64&flag_docker-version=none&flag_kernel-version=4.4.0-127-generic&os=linux&signature=1H2Uh3xhdJFg8OVkhDyigc96i5jveCeUu6WfomiBPks%3D&version=2.8.1": dial tcp: lookup checkpoint-api.weave.works on 10.96.0.10:53: write udp 192.168.100.104:59519->10.96.0.10:53: write: operation not permitted
INFO: 2021/03/04 23:26:51.239564 [allocator 5e:2a:ec:84:96:b2] Initialising via deferred consensus
INFO: 2021/03/04 23:26:51.239750 Sniffing traffic on datapath (via ODP)
INFO: 2021/03/04 23:26:51.318659 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2021/03/04 23:26:51.318782 Listening for metrics requests on 0.0.0.0:6782
INFO: 2021/03/04 23:26:51.442573 [kube-peers] Added myself to peer list &{[{5e:2a:ec:84:96:b2 lambert-test}]}
DEBU: 2021/03/04 23:26:51.528975 [kube-peers] Nodes that have disappeared: map[]
INFO: 2021/03/04 23:26:51.540792 Assuming quorum size of 1
INFO: 2021/03/04 23:26:51.540913 adding entry 10.244.0.0/16 to weaver-no-masq-local of 0
INFO: 2021/03/04 23:26:51.540959 added entry 10.244.0.0/16 to weaver-no-masq-local of 0
INFO: 2021/03/04 23:26:51.821330 [allocator 5e:2a:ec:84:96:b2] Address 10.32.0.1/12 claimed by weave:expose - not in our range
10.244.0.1
DEBU: 2021/03/04 23:26:52.099712 registering for updates for node delete events
WARN: 2021/03/04 23:27:00.072571 [allocator]: Delete: no addresses for 9c7adec207bc06d46c3314883b23f682672ce61cb11091c1d500e231f41ff455
WARN: 2021/03/04 23:27:01.292188 [allocator]: Delete: no addresses for 19c8ab1681c7a38d49c2b535f196e025c40b89098438f8a561d98ac1340a6769

[dynam1]

Unfortunately, I could not fetch more detailed.

(venv-workbench) ubuntu@dynam-demo:~/kubeadm-bootstrap$ kubectl logs -f weave-net-9q2x4 -n kube-system -c weave
Network 10.244.0.0/16 overlaps with existing route 10.244.0.0/24 on host

Also, my weave pod was repeatedly restarted.
weave-net-9q2x4 1/2 CrashLoopBackOff 13 45m