UnknownHostException in Zookeeper Curator leaves Druid node in disconnected state. #3

elanv · 2018-11-13T07:18:35Z

Describe the bug
현재 Druid zookeeper를 kubernetes에서 운영하고 있습니다. kubernetes에서 동작중인 zookeeper 서버가 어떤 이유로 fail 상태에 들어가면 kubernetes는 zookeeper 노드를 삭제하고 다시 생성하며 이때 삭제된 zookeeper 서버의 DNS 레코드도 삭제후 다시 생성됩니다. (약 1분이내 복구) DNS 레코드가 일시적으로 삭제되면서 Zookeeper Curator에서 UnknownHostException이 발생하는 것으로 보이는데 이 예외가 발생한 Druid 노드는 여전히 running 상태이지만 다른 정상적인 zookeeper 서버가 있더라도 더 이상 Druid 클러스터에 등록하려는 시도를 하지 않으므로 disconnected 상태로 running 중인 상태가 됩니다.

To Reproduce
Steps to reproduce the behavior:

kubernetes에 zookeeper를 구동한다.
Druid Zookeeper 설정을 kubernetes가 생성한 zookeeper DNS name으로 지정한다.
Druid Zookeeper 서버를 일부 삭제한다.
Druid node 일부에서 Curator로부터 UnknownHostException이 발생하면 Druid 클러스터로부터 disconnected 상태가 되고 복구되지 않는다.

Expected behavior
Druid node가 Zookeeper connection fail 시 다른 정상 상태의 zookeeper 서버로의 연결을 retry하면서 연결에 성공하면 다시 Druid cluster로 복귀하게 되는 것처럼 특정 zookeeper 서버의 UnknownHostException 발생하더라도 정상 zookeeper로의 연결을 시도하여 자동으로 Druid 클러스터로 복구되기를 기대

Screenshots

Desktop (please complete the following information):

Additional context
Druid 서버 환경

Distro: CentOS 7.4
커널: 3.10.0-862.3.3.el7.x86_64

Related Druid issue: Unhandled Errors in Curator leave Druid Nodes in disconnected state apache/druid#6130

2018-11-10T13:57:14,143 INFO [Curator-Framework-0] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zk-zookeeper-0.zk-zookeeper-headless:2181,zk-zookeeper-1.zk-zookeeper-headless:2181,zk-zookeeper-2.zk-zookeeper-headless:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@285005b8
2018-11-10T13:57:14,143 ERROR [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.net.UnknownHostException: zk-zookeeper-2.zk-zookeeper-headless
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
        at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) ~[zookeeper-3.4.8.jar:3.4.8--1]
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:446) ~[zookeeper-3.4.8.jar:3.4.8--1]
        at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150) ~[curator-framework-2.10.0.jar:?]
        at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:91) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.10.0.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-11-10T13:57:23,144 INFO [Curator-Framework-0] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zk-zookeeper-0.zk-zookeeper-headless:2181,zk-zookeeper-1.zk-zookeeper-headless:2181,zk-zookeeper-2.zk-zookeeper-headless:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@285005b8
2018-11-10T13:57:23,145 ERROR [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.net.UnknownHostException: zk-zookeeper-2.zk-zookeeper-headless: Name or service not known
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_161]
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[?:1.8.0_161]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_161]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_161]
        at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) ~[zookeeper-3.4.8.jar:3.4.8--1]
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:446) ~[zookeeper-3.4.8.jar:3.4.8--1]
        at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150) ~[curator-framework-2.10.0.jar:?]
        at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:91) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) ~[curator-client-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.10.0.jar:?]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.10.0.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

The text was updated successfully, but these errors were encountered:

seoeun25 · 2018-11-14T07:58:47Z

apache druid에서 관련 이슈를 backport 하여 759-zookeeper 라는 브랜치를 생성하였습니다.
테스트 해보고 알려주시면, develop에 적용하도록 하겠습니다.

elanv · 2018-12-19T09:13:52Z

패치 적용 및 테스트를 해보았습니다.
패치 적용 전/후에 unknown host exception 발생 시 차이점 입니다.

패치 전: druid node가 running 상태를 유지하면서 zookeeper와 disconnected 상태가 지속됨. 세그먼트 생성은 할 수 없으나 broker를 통한 쿼리 기능은 정상 작동함.
패치 후: druid node 프로세스 자체가 종료되므로 쿼리 시 필요한 segment 를 가진 historical node가 종료된 경우 쿼리가 불가 하게됨.

클라우드 환경에서는 druid node가 종료되면 자동으로 복구가 시도되어 종료된 druid node가 재시작이 되고 zookeeper도 복구가 되었다면 druid도 복구가 되므로 이런 방식을 적용한 것이 아닌가 추측됩니다.

하지만 클라우드 환경에서는 아래 두 가지 조건이 충족되어야 안정적으로 복구가 될 것 같습니다.

zookeeper도 즉시 복구된 다는 보장이 있어야 합니다. zookeeper가 복구되지 않은 상태인 경우 druid node는 종료/재시작 루프에 빠지게 됩니다. 이 경우 패치 전/후 차이 비교에서 보시는 것처럼 쿼리 기능에 영향을 주게 됩니다.
복구된 zookeeper의 ip가 변경되지 않아야 합니다. 클라우드 환경에서는 재시작한 vm이나 컨테이너의 ip가 변경 될 수 있는데요, 아래 링크한 ZOOKEEPER-2184 이슈로 인해 Druid node는 ip가 변경된 zookeeper에 대한 reconnect에 실패하게 됩니다.

1번 문제에 대해서는 Curator ZookeeperFactory 를 확장한 방법이 있는 것 같구요..
https://issues.apache.org/jira/browse/CURATOR-229?focusedCommentId=16138245&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16138245

2번 문제를 해결하려면 zookeeper 버전 업데이트가 필요할 것 같습니다.
Zookeeper 3.4.13 버전에서 zookeeper ip 변경을 고려한 패치가 적용된 것 같고 아래 코멘트에 따르면 1번 문제도 해결해주는 것 같습니다.
CURATOR-229:
https://issues.apache.org/jira/browse/CURATOR-229?focusedCommentId=16639550&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16639550

제가 참고한 이슈 링크입니다.

CURATOR-229:
https://issues.apache.org/jira/browse/CURATOR-229

ZOOKEEPER-2184:
https://issues.apache.org/jira/browse/ZOOKEEPER-2184

kyungtaak assigned seoeun25 and navis Nov 13, 2018

kyungtaak transferred this issue from metatron-app/metatron-discovery Jun 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UnknownHostException in Zookeeper Curator leaves Druid node in disconnected state. #3

UnknownHostException in Zookeeper Curator leaves Druid node in disconnected state. #3

elanv commented Nov 13, 2018

seoeun25 commented Nov 14, 2018

elanv commented Dec 19, 2018 •

edited

Loading

UnknownHostException in Zookeeper Curator leaves Druid node in disconnected state. #3

UnknownHostException in Zookeeper Curator leaves Druid node in disconnected state. #3

Comments

elanv commented Nov 13, 2018

seoeun25 commented Nov 14, 2018

elanv commented Dec 19, 2018 • edited Loading

elanv commented Dec 19, 2018 •

edited

Loading