kubedns not starting when using custom clusterDomain

[pasztorl]

What happened?

created vcluster with customized networking.advanced.clusterDomain then coredns failing to start
if i remove the clusterdomain settings vcluster seems to work normally

What did you expect to happen?

vcluster and coredns start normally

How can we reproduce it (as minimally and precisely as possible)?

vcluster create --values vcluster.values.yaml --connect=false my-cluster

Anything else we need to know?

coredns log output:

[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Service: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Namespace: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:1053
CoreDNS-1.11.3
linux/amd64, go1.21.11, a6338e9
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/ready: Still waiting on: "kubernetes"

Host cluster Kubernetes version

v1.30.6

vcluster version

0.21.1

VCluster Config

experimental:
  multiNamespaceMode:
    enabled: false

controlPlane:
  distro:
    k8s:
      enabled: true
      version: v1.30.6

  statefulSet:
    image:
      tag: 0.21.1

  ingress:
    enabled: true
    host: "my-cluster"
    # PathType is the path type of the ingress
    pathType: ImplementationSpecific
    labels: {}
    annotations:
      haproxy.org/ssl-passthrough: "true"

exportKubeConfig:
  context: my-cluster
  server: "https://my-cluster"

networking:
  advanced:
    clusterDomain: "k8s.my-cluster"

[pasztorl]

more info from vcluster logs:

2024-12-17 11:02:52 ERROR webhook/webhook.go:154 Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:vcluster-my-cluster:vc-my-cluster" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope {"component": "vcluster"}
2024-12-17 11:02:52 ERROR filters/authentication.go:73 Unable to authenticate the request {"component": "vcluster", "error": "tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vcluster-my-cluster:vc-my-cluster\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope", "errorCauses": [{"error": "tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vcluster-my-cluster:vc-my-cluster\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope"}]}
2024-12-17 11:02:53 ERROR webhook/webhook.go:154 Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:vcluster-my-cluster:vc-my-cluster" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope {"component": "vcluster"}
2024-12-17 11:02:53 ERROR filters/authentication.go:73 Unable to authenticate the request {"component": "vcluster", "error": "tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vcluster-my-cluster:vc-my-cluster\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope", "errorCauses": [{"error": "tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vcluster-my-cluster:vc-my-cluster\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope"}]}

the vcluster works normally if i remove networking. advanced. clusterDomain

[vardhaman-surana]

hi @pasztorl i am working on this and will try to raise a PR soon.