webhook requires hostNetwork: true on EKS with Calico CNI

[o-grigorev]

There is an EKS cluster with the Calico CNI installed. In this environment, the deployment of orchestra-login-portal fails when attempting to deploy any AuthenticationChain due to the webhook.

Internal error occurred: failed calling webhook "authmechs-openunison.tremolo.io": failed to call webhook: Post "https://openunison-openunison.openunison.svc:443/k8s/webhooks/v1/authmechs?timeout=5s": Address is not allowed

To work around this issue, it is required to patch the openunison deployment by adding hostNetwork: true for the Orchestra (OpenUnison) pods. This network issue is known to occur on EKS with Calico CNI, and you can find more details about it here.

Is it possible to add this parameter into kind: OpenUnison being possible to set it via helm chart.

Thanks

[mlbiam]

I've run into this issue a couple of times and could never figure out the issue. Didn't realize it had to do with an alternate CNI. Thanks! Added the flag network.enableHostNetwork to the values.yaml. When true, it sets the hostNetwork to true in OpenUnison's Deployment

[o-grigorev]

Hello @mlbiam ,
I figured out that Pod DNS policy should be changed as well, without that I got an issue with DNS resolution

[XNIO-1 task-2] ERROR ConfigSys - Could not process request
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison jakarta.servlet.ServletException: Could not execute request
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.ProxySys.doURI(ProxySys.java:112) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:141) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.auth.AuthMgrSys.doAuthMgr(AuthMgrSys.java:138) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:126) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.auth.AzSys.doAz(AzSys.java:139) ~[unison-sdk-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:111) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.auth.AuthSys.doAuth(AuthSys.java:140) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:105) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.ConfigSys.doConfig(ConfigSys.java:296) [unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.embedd.NextEmbSys.nextSys(NextEmbSys.java:93) [unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.filter.UnisonServletFilter.doFilter(UnisonServletFilter.java:299) [unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:276) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101) [undertow-servlet-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859) [undertow-core-2.3.7.Final.jar:2.3.7.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) [jboss-threads-2.3.6.Final.jar:2.3.6.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282) [xnio-api-3.8.9.Final.jar:3.8.9.Final]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.lang.Thread.run(Thread.java:829) [?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison Caused by: java.net.UnknownHostException: ouhtml-openunison.openunison.svc: Name or service not known
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:930) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.net.InetAddress.getAllByName(InetAddress.java:1386) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at java.net.InetAddress.getAllByName(InetAddress.java:1307) ~[?:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.14.jar:4.5.14]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.postProcess.UriRequestProcess.postProcess(UriRequestProcess.java:127) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:92) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.filters.SetNoCacheHeaders.doFilter(SetNoCacheHeaders.java:25) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:86) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.filters.XForward.doFilter(XForward.java:61) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.filter.HttpFilterChainImpl.nextFilter(HttpFilterChainImpl.java:86) ~[unison-server-core-1.0.37.jar:?]
openunison-openunison-66ccff9fbd-4gjv5 openunison-openunison 	at com.tremolosecurity.proxy.ProxySys.doURI(ProxySys.java:97) ~[unison-server-core-1.0.37.jar:?]

The fix is to change Orchestra pod's dnsPolicy to ClusterFirstWithHostNet

[spantaleev]

I'm also hitting this issue on a cluster configured with Kubespray, which uses Calico by default.

Besides the networkenableHostNetwork: true value change and patching /spec/template/spec/dnsPolicy in the openunison-openunison Deployment resource manually, those deploying via ArgoCD may wish to adjust their application like this to avoid it undoing their patch:

 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
   name: openunison
   namespace: argocd
 spec:
   project: default
   ignoreDifferences:
   - group: "admissionregistration.k8s.io"
     kind: "ValidatingWebhookConfiguration"
     jsonPointers:
     - /webhooks/0/clientConfig/caBundle
     - /webhooks/1/clientConfig/caBundle
     - /webhooks/2/clientConfig/caBundle
     - /webhooks/3/clientConfig/caBundle
     - /webhooks/4/clientConfig/caBundle
+
+  # Work around a Calico CNI issue.
+  # See: https://github.com/TremoloSecurity/OpenUnison/issues/788
+  - group: apps
+    kind: Deployment
+    jsonPointers:
+      - /spec/template/spec/dnsPolicy

---

It'd be great if the Helm chart provided a configuration value for dnsPolicy, so that we won't have to rest to such hacks.