Unsupport using Etcd as the registry in shardingsphere proxy #33565

wuqiong818 · 2024-11-06T09:33:35Z

Bug Report

[ERROR] 2024-11-06 09:19:40.998 [Etcd-EventListener-0] c.g.common.eventbus.EventBus.default - Exception thrown by subscriber method renew(org.apache.shardingsphere.mode.event.dispatch.state.compute.WorkerIdEvent) on subscriber org.apache.shardingsphere.mode.manager.cluster.event.subscriber.dispatch.ComputeNodeStateSubscriber@3265ef81 when dispatching event: org.apache.shardingsphere.mode.event.dispatch.state.compute.WorkerIdEvent@7282af36
java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because "workerId" is null
at org.apache.shardingsphere.infra.instance.ComputeNodeInstanceContext.lambda$updateWorkerId$1(ComputeNodeInstanceContext.java:123)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at org.apache.shardingsphere.infra.instance.ComputeNodeInstanceContext.updateWorkerId(ComputeNodeInstanceContext.java:123)
at org.apache.shardingsphere.mode.manager.cluster.event.subscriber.dispatch.ComputeNodeStateSubscriber.renew(ComputeNodeStateSubscriber.java:78)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85)
at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:142)
at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:68)
at com.google.common.eventbus.Dispatcher$PerThreadQueuedDispatcher.dispatch(Dispatcher.java:109)
at com.google.common.eventbus.EventBus.post(EventBus.java:270)
at org.apache.shardingsphere.infra.util.eventbus.EventBusContext.post(EventBusContext.java:44)
at java.base/java.util.Optional.ifPresent(Optional.java:178)
at org.apache.shardingsphere.mode.manager.cluster.listener.GlobalMetaDataChangedListener.onChange(GlobalMetaDataChangedListener.java:39)
at org.apache.shardingsphere.mode.repository.cluster.etcd.EtcdRepository.lambda$dispatchEvent$3(EtcdRepository.java:191)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)

wuqiong818 · 2024-11-06T09:38:03Z

my configuratioin
global.yaml
`mode:
type: Cluster
repository:
type: etcd
props:
namespace: governance_ds
server-lists: http://124.156.192.247:11094
props:
timeToLiveSeconds: 30
connectionTimeout: 30

authority:
users:
- user: root@%
password: root
privilege:
type: ALL_PERMITTED

props:
sql-show: true`

wuqiong818 · 2024-11-06T09:42:08Z

my use version is ShardingSphere-Proxy:5.5.1

terrymanu · 2024-11-06T14:38:31Z

How to reproduce it?

wuqiong818 · 2024-11-07T03:16:39Z

An error was reported while I was loading the following configuration file in cluster mode.
I used ShardingSphere-Proxy:5.5.1

global.yaml

`mode:
type: Cluster
repository:
type: etcd
props:
namespace: governance_ds
server-lists: http://124.156.192.247:11094/
props:
timeToLiveSeconds: 30
connectionTimeout: 30

authority:
users:

user: root@%
password: root
privilege:
type: ALL_PERMITTED
props:
sql-show: true`

wuqiong818 · 2024-11-07T03:19:44Z

this photo is specific error info

terrymanu · 2024-11-07T04:51:44Z

It is only the log.
How did this log occur?

wuqiong818 · 2024-11-07T06:59:49Z

hello,Here's how i do it.

my image

i use docker start proxys，and this image written myself.The image contains some third-party packages, such as mysql connection、grpc ...
Image Name is yiwuqiong/sharding-etcd. You can search for this image on docker hub.
我的docker run command is
docker run -d \ --name shardingproxy1 \ -p 3307:3307 \ -e JVM_OPTS='-Xms256m -Xmx400m -Xss1m' \ -v /home/wangye/sharding/conf/:/opt/shardingsphere-proxy/conf \ yiwuqiong/sharding-etcd:3.0

my global.yaml

`mode:
type: Cluster
repository:
type: etcd
props:
namespace: governance_ds
server-lists: http://124.156.192.247:30004
props:
timeToLiveSeconds: 30
connectionTimeout: 30

authority:
users:
- user: root@%
password: root
privilege:
type: ALL_PERMITTED

props:
sql-show: true`

my database-sharding.yaml

databaseName: sharding_db

dataSources:
server_order0:
url: jdbc:mysql://124.156.192.247:19290/db_order
username: root
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1

server_order1:
url: jdbc:mysql://124.156.192.247:62195/db_order
username: root
password: 123456
connectionTimeoutMilliseconds: 30000
idleTimeoutMilliseconds: 60000
maxLifetimeMilliseconds: 1800000
maxPoolSize: 50
minPoolSize: 1

rules:

!SHARDING
autoTables:
t_order: #一对多，需要使用算法进行映射自动分片
actualDataSources: server_order${0..1}
shardingStrategy:
standard:
shardingColumn: order_no
shardingAlgorithmName: order_table_mod
keyGenerateStrategy:
column: id #表中需要留好列，shardingSphere不会帮助我们进行创建
keyGeneratorName: snowflake
t_order_item:
actualDataSources: server_order${0..1}
shardingStrategy:
standard:
shardingColumn: order_no
shardingAlgorithmName: order_item_table_mod
keyGenerateStrategy:
column: id #表中需要留好列，shardingSphere不会帮助我们进行创建
keyGeneratorName: snowflake
bindingTables:
- t_order, t_order_item
  shardingAlgorithms:
  order_table_mod:
  type: MOD
  props:
  sharding-count: 20
  order_item_table_mod:
  type: MOD
  props:
  sharding-count: 20
keyGenerators:
snowflake:
type: SNOWFLAKE
props:
worker-id: 1000 # 工作机器唯一标识，集群模式下自动生成，保证唯一
max-vibration-offset: 19
max-tolerate-time-difference-milliseconds: 10 #最大容器时钟回退时间，时钟回拨超过这个时间会报错
!BROADCAST
tables: # 广播表规则列表
- t_dict

operational process

I started multiple proxies
When I was shutting down one of the proxies，I see the following error message by docker logs <container_name>

we find java version: java21, full_version=21.0.4, full_path=/opt/java/openjdk/bin/java unadapted java version, please notice... The classpath is /opt/shardingsphere-proxy/conf:/opt/shardingsphere-proxy/conf:.:/opt/shardingsphere-proxy/lib/*:/opt/shardingsphere-proxy/ext-lib/* main class org.apache.shardingsphere.proxy.Bootstrap -1 /opt/shardingsphere-proxy/conf 0.0.0.0 false [INFO ] 2024-11-07 06:57:11.496 [main] i.v.c.spi.resolver.ResolverProvider - Using the default address resolver as the dns resolver could not be loaded [WARN ] 2024-11-07 06:57:40.288 [main] o.a.s.m.m.c.w.ClusterWorkerIdGenerator - It is unnecessary to configure worker-id in cluster mode, system assigned worker-id was 1 [ERROR] 2024-11-07 06:57:43.693 [main] o.a.s.d.p.c.l.PipelineContextManagerLifecycleListener - Dispatch enable pipeline job start event failed java.lang.IllegalArgumentException: Unsupported cluster type: etcd at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.lambda$createRegistryCenter$0(PipelineAPIFactory.java:150) at org.apache.shardingsphere.infra.exception.core.ShardingSpherePreconditions.checkState(ShardingSpherePreconditions.java:44) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.createRegistryCenter(PipelineAPIFactory.java:150) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.<init>(PipelineAPIFactory.java:141) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.lambda$getInstance$1(PipelineAPIFactory.java:155) at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.getInstance(PipelineAPIFactory.java:155) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory.getRegistryCenter(PipelineAPIFactory.java:109) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$ElasticJobAPIHolder.<init>(PipelineAPIFactory.java:123) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$ElasticJobAPIHolder.lambda$getInstance$0(PipelineAPIFactory.java:130) at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$ElasticJobAPIHolder.getInstance(PipelineAPIFactory.java:130) at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory.getJobConfigurationAPI(PipelineAPIFactory.java:89) at org.apache.shardingsphere.data.pipeline.core.listener.PipelineContextManagerLifecycleListener.dispatchEnablePipelineJobStartEvent(PipelineContextManagerLifecycleListener.java:74) at org.apache.shardingsphere.data.pipeline.core.listener.PipelineContextManagerLifecycleListener.onInitialized(PipelineContextManagerLifecycleListener.java:65) at org.apache.shardingsphere.mode.manager.ContextManager.<init>(ContextManager.java:75) at org.apache.shardingsphere.mode.manager.cluster.ClusterContextManagerBuilder.build(ClusterContextManagerBuilder.java:64) at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.createContextManager(BootstrapInitializer.java:62) at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.init(BootstrapInitializer.java:53) at org.apache.shardingsphere.proxy.Bootstrap.main(Bootstrap.java:64) [WARN ] 2024-11-07 06:57:43.694 [main] o.a.s.s.c.j.s.c.StatisticsCollectJobWorker - Can not collect statistics because of unsupported cluster type: etcd [INFO ] 2024-11-07 06:57:45.999 [main] o.a.s.p.v.ShardingSphereProxyVersion - Database type is MySQL, version is 8.0.40, database name is sharding_db [INFO ] 2024-11-07 06:57:46.001 [main] o.a.s.p.frontend.ssl.ProxySSLContext - Proxy frontend SSL/TLS is not enabled. [INFO ] 2024-11-07 06:57:46.295 [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy Cluster mode started successfully [ERROR] 2024-11-07 06:57:51.335 [Etcd-EventListener-0] o.a.s.m.r.c.etcd.EtcdRepository - Dispatch event failed java.util.concurrent.CompletionException: java.lang.NullPointerException: Cannot invoke "String.split(String)" because "attributes" is null at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.lang.NullPointerException: Cannot invoke "String.split(String)" because "attributes" is null at org.apache.shardingsphere.infra.instance.metadata.proxy.ProxyInstanceMetaData.<init>(ProxyInstanceMetaData.java:52) at org.apache.shardingsphere.infra.instance.metadata.InstanceMetaDataFactory.create(InstanceMetaDataFactory.java:41) at org.apache.shardingsphere.mode.manager.cluster.event.builder.ComputeNodeOnlineDispatchEventBuilder.createInstanceEvent(ComputeNodeOnlineDispatchEventBuilder.java:66) at org.apache.shardingsphere.mode.manager.cluster.event.builder.ComputeNodeOnlineDispatchEventBuilder.build(ComputeNodeOnlineDispatchEventBuilder.java:57) at org.apache.shardingsphere.mode.manager.cluster.listener.GlobalMetaDataChangedListener.onChange(GlobalMetaDataChangedListener.java:39) at org.apache.shardingsphere.mode.repository.cluster.etcd.EtcdRepository.lambda$dispatchEvent$3(EtcdRepository.java:191) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ... 3 common frames omitted [ERROR] 2024-11-07 06:57:57.339 [Etcd-EventListener-0] c.g.common.eventbus.EventBus.default - Exception thrown by subscriber method renew(org.apache.shardingsphere.mode.event.dispatch.state.compute.WorkerIdEvent) on subscriber org.apache.shardingsphere.mode.manager.cluster.event.subscriber.dispatch.ComputeNodeStateSubscriber@3dd570c3 when dispatching event: org.apache.shardingsphere.mode.event.dispatch.state.compute.WorkerIdEvent@2322bbce java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because "workerId" is null at org.apache.shardingsphere.infra.instance.ComputeNodeInstanceContext.lambda$updateWorkerId$1(ComputeNodeInstanceContext.java:123) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at org.apache.shardingsphere.infra.instance.ComputeNodeInstanceContext.updateWorkerId(ComputeNodeInstanceContext.java:123) at org.apache.shardingsphere.mode.manager.cluster.event.subscriber.dispatch.ComputeNodeStateSubscriber.renew(ComputeNodeStateSubscriber.java:78) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85) at com.google.common.eventbus.Subscriber $SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:142) at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:68) at com.google.common.eventbus.Dispatcher$PerThreadQueuedDispatcher.dispatch(Dispatcher.java:109) at com.google.common.eventbus.EventBus.post(EventBus.java:270) at org.apache.shardingsphere.infra.util.eventbus.EventBusContext.post(EventBusContext.java:44) at java.base/java.util.Optional.ifPresent(Optional.java:178) at org.apache.shardingsphere.mode.manager.cluster.listener.GlobalMetaDataChangedListener.onChange(GlobalMetaDataChangedListener.java:39) at org.apache.shardingsphere.mode.repository.cluster.etcd.EtcdRepository.lambda$dispatchEvent$3(EtcdRepository.java:191) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)

wuqiong818 · 2024-11-07T07:02:38Z

I feel very strange,It actually reports the error that Etcd is not supported,But there is no official documentation supporting Etcd.

linghengqian · 2024-11-07T07:20:12Z

I feel very strange,It actually reports the error that Etcd is not supported,But there is no official documentation supporting Etcd.

Your code block is too difficult for humans to read. Please use markdown syntax similar to ``` to surround the code block. See https://docs.github.com/zh/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks .
If your yiwuqiong/sharding-etcd:3.0 Docker Image is generated by a command like ./mvnw clean install -Prelease,docker -T1C -DskipTests -Djacoco.skip=true -Dcheckstyle.skip=true -Drat.skip=true -Dmaven.javadoc.skip=true -B, the key dependencies of Etcd are not in the Docker Image. You can notice this by looking at the pom.xml of org.apache.shardingsphere:shardingsphere-proxy-distribution:5.5.1.
The key dependency I'm referring to is io.vertx:vertx-grpc:4.5.1. This is documented in the pom.xml of org.apache.shardingsphere:shardingsphere-test-native:5.5.2-SNAPSHOT. I forgot where it is documented though.

       <dependency>
            <groupId>io.vertx</groupId>
            <artifactId>vertx-grpc</artifactId>
            <version>4.5.1</version>
        </dependency>

I would first recommend you to refer to ShardingSphere JDBC's unit test for this, namely org.apache.shardingsphere.test.natived.jdbc.modes.cluster.EtcdTest. If you have IntelliJ IDEA, Docker Engine and any JDK11+ installed, you can run the unit tests manually through IntelliJ IDEA without using the GraalVM Native Image.
The current issue looks more like a documentation issue.

wuqiong818 · 2024-11-07T07:37:02Z

oh,Sorry, this is my first time using issue.
My image is generated from a dockerfile, i used shardingSphere-prxoy.
I put the missing jar package in /opt/shardingsphere-proxy/ext-lib.
I read the documentation very carefully, but it didn't solve my problem

wuqiong818 · 2024-11-07T07:39:53Z

I feel very strange,It actually reports the error that Etcd is not supported,But there is no official documentation supporting Etcd.

Your code block is too difficult for humans to read. Please use markdown syntax similar to ``` to surround the code block. See https://docs.github.com/zh/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks .

If your yiwuqiong/sharding-etcd:3.0 Docker Image is generated by a command like ./mvnw clean install -Prelease,docker -T1C -DskipTests -Djacoco.skip=true -Dcheckstyle.skip=true -Drat.skip=true -Dmaven.javadoc.skip=true -B, the key dependencies of Etcd are not in the Docker Image. You can notice this by looking at the pom.xml of org.apache.shardingsphere:shardingsphere-proxy-distribution:5.5.1.

The key dependency I'm referring to is io.vertx:vertx-grpc:4.5.1. This is documented in the pom.xml of org.apache.shardingsphere:shardingsphere-test-native:5.5.2-SNAPSHOT. I forgot where it is documented though.
       <dependency>
            <groupId>io.vertx</groupId>
            <artifactId>vertx-grpc</artifactId>
            <version>4.5.1</version>
        </dependency>
I would first recommend you to refer to ShardingSphere JDBC's unit test for this, namely org.apache.shardingsphere.test.natived.jdbc.modes.cluster.EtcdTest. If you have IntelliJ IDEA, Docker Engine and any JDK11+ installed, you can run the unit tests manually through IntelliJ IDEA without using the GraalVM Native Image.

The current issue looks more like a documentation issue.

It's an error message that you can copy and put in a text editor

wuqiong818 · 2024-11-07T07:42:52Z

The team I'm on uses golang, so it uses proxy, and the registry leans toward Etcd

wuqiong818 · 2024-11-07T07:45:49Z

？？？

linghengqian · 2024-11-07T07:47:34Z

I'm trying to reproduce your point in a separate git. Normally, I wouldn't think http://124.156.192.247:11094 would be a usable ip.

wuqiong818 · 2024-11-07T07:50:14Z

At this time, the company Intranet, you need to change to your own etcd

wuqiong818 · 2024-11-07T07:51:59Z

You start with my image and global configuration file, then start multiple proxy nodes, and then close any one of them, and you'll see the error I mentioned。

wuqiong818 · 2024-11-07T07:56:06Z

I think it is caused by the following reasons：

This error occurs in communication between multiple nodes
When a node fails and disconnects, the other nodes or etcd do not know, and a null pointer exception occurs

linghengqian · 2024-11-07T09:24:51Z

I confirmed this is a bug at https://github.com/linghengqian/shardingsphere-proxy-etcd-test . And it is specific to shardingsphere proxy. shardingsphere jdbc is not affected by this.

git clone [email protected]:linghengqian/shardingsphere-proxy-etcd-test.git
cd ./shardingsphere-proxy-etcd-test/
docker compose up -d
docker compose logs --follow shardingsphere-proxy-custom-etcd

$ docker compose logs --follow shardingsphere-proxy-custom-etcd
shardingsphere-proxy-custom-etcd-1  | we find java version: java21, full_version=21.0.4, full_path=/opt/java/openjdk/bin/java
shardingsphere-proxy-custom-etcd-1  | unadapted java version, please notice...
shardingsphere-proxy-custom-etcd-1  | The port is 3308
shardingsphere-proxy-custom-etcd-1  | The classpath is /opt/shardingsphere-proxy/conf:.:/opt/shardingsphere-proxy/lib/*:/opt/shardingsphere-proxy/ext-lib/*
shardingsphere-proxy-custom-etcd-1  | main class org.apache.shardingsphere.proxy.Bootstrap 3308 /opt/shardingsphere-proxy/conf 0.0.0.0 false
shardingsphere-proxy-custom-etcd-1  | [INFO ] 2024-11-07 09:00:35.476 [main] i.v.c.spi.resolver.ResolverProvider - Using the default address resolver as the dns resolver could not be loaded
shardingsphere-proxy-custom-etcd-1  | [ERROR] 2024-11-07 09:00:41.107 [main] o.a.s.d.p.c.l.PipelineContextManagerLifecycleListener - Dispatch enable pipeline job start event failed
shardingsphere-proxy-custom-etcd-1  | java.lang.IllegalArgumentException: Unsupported cluster type: etcd
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.lambda$createRegistryCenter$0(PipelineAPIFactory.java:150)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.infra.exception.core.ShardingSpherePreconditions.checkState(ShardingSpherePreconditions.java:44)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.createRegistryCenter(PipelineAPIFactory.java:150)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.<init>(PipelineAPIFactory.java:141)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.lambda$getInstance$1(PipelineAPIFactory.java:155)
shardingsphere-proxy-custom-etcd-1  |   at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$RegistryCenterHolder.getInstance(PipelineAPIFactory.java:155)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory.getRegistryCenter(PipelineAPIFactory.java:109)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$ElasticJobAPIHolder.<init>(PipelineAPIFactory.java:123)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$ElasticJobAPIHolder.lambda$getInstance$0(PipelineAPIFactory.java:130)
shardingsphere-proxy-custom-etcd-1  |   at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory$ElasticJobAPIHolder.getInstance(PipelineAPIFactory.java:130)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.job.api.PipelineAPIFactory.getJobConfigurationAPI(PipelineAPIFactory.java:89)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.listener.PipelineContextManagerLifecycleListener.dispatchEnablePipelineJobStartEvent(PipelineContextManagerLifecycleListener.java:74)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.data.pipeline.core.listener.PipelineContextManagerLifecycleListener.onInitialized(PipelineContextManagerLifecycleListener.java:65)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.mode.manager.ContextManager.<init>(ContextManager.java:75)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.mode.manager.cluster.ClusterContextManagerBuilder.build(ClusterContextManagerBuilder.java:64)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.createContextManager(BootstrapInitializer.java:62)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.init(BootstrapInitializer.java:53)
shardingsphere-proxy-custom-etcd-1  |   at org.apache.shardingsphere.proxy.Bootstrap.main(Bootstrap.java:64)
shardingsphere-proxy-custom-etcd-1  | [WARN ] 2024-11-07 09:00:41.108 [main] o.a.s.s.c.j.s.c.StatisticsCollectJobWorker - Can not collect statistics because of unsupported cluster type: etcd
shardingsphere-proxy-custom-etcd-1  | [INFO ] 2024-11-07 09:00:41.202 [main] o.a.s.p.frontend.ssl.ProxySSLContext - Proxy frontend SSL/TLS is not enabled.
shardingsphere-proxy-custom-etcd-1  | [INFO ] 2024-11-07 09:00:41.226 [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy Cluster mode started successfully

The exception involved in this bug was introduced in Refactor PipelineContext to support JDBC mode in pipeline jobs #24883. From,

ModeConfiguration modeConfig = PipelineContext.getModeConfig();
return registryCenterInitializer.createRegistryCenter(modeConfig, PipelineMetaDataNode.getElasticJobNamespace());

To,

            PipelineContext pipelineContext = PipelineContextManager.getContext(contextKey);
            ModeConfiguration modeConfig = pipelineContext.getModeConfig();
            String elasticJobNamespace = PipelineMetaDataNode.getElasticJobNamespace();
            String clusterType = modeConfig.getRepository().getType();
            if ("ZooKeeper".equals(clusterType)) {
                return registryCenterInitializer.createZookeeperRegistryCenter(modeConfig, elasticJobNamespace);
            } else {
                throw new IllegalArgumentException("Unsupported cluster type: " + clusterType);
            }

ElasticJob has never actually supported registries other than zookeeper. See Use etcd3 as a registry shardingsphere-elasticjob#2221 .
@sandynz So from my point of view, shardingsphere proxy has never supported etcd as a registry. Only shardingsphere jdbc can use etcd as a registry. Is this the current situation?

wuqiong818 · 2024-11-07T09:35:00Z

I use is shardingsphere-proxy

wuqiong818 · 2024-11-07T09:42:11Z

Except for errors that do not support Etcd.
When Etcd is used as the persistent storage in cluster mode and multiple proxy nodes are deployed, the following error occurs if one proxy node fails and goes offline.

具体代码位置在etcd的watch方法中
classpath：apache-shardingsphere-5.5.1-src-release/mode/type/cluster/repository/provider/etcd/src/main/java/org/apache/shardingsphere/mode/repository/cluster/etcd/EtcdRepository.java

linghengqian · 2024-11-07T09:46:22Z

Except for errors that do not support Etcd.
When Etcd is used as the persistent storage in cluster mode and multiple proxy nodes are deployed, the following error occurs if one proxy node fails and goes offline.

@wuqiong818 This sounds as expected, because the shardingsphere proxy metadata is not initialized properly to the etcd server.
For convenience, I still use gcr.io/etcd-development/etcd:v3.5.10, which is the most commonly used Docker Image in the Etcd community, as the Docker Image to reproduce the error in https://github.com/linghengqian/shardingsphere-proxy-etcd-test. Google will shut down Container Registry next year, but Transition from gcr.io to a modern artifact repository etcd-io/etcd#15199 should not affect the current issue.
@zhaojinchao95 I'm personally not sure if this should be solved without elasticjob release. elasticjob's org.apache.shardingsphere.elasticjob.reg.base.CoordinatorRegistryCenter is certainly an SPI, but should Consider elasticjob governance center multiplex ShardingSphere governance center shardingsphere-elasticjob#2128 and Consider ElasticJob governance center multiplex ShardingSphere governance center #21380 update their descriptions to state the current status of using etcd in shardingsphere proxy?

wuqiong818 · 2024-11-07T09:52:39Z

Do you mean that proxy does not support etcd at present, only JDBC does? @linghengqian

linghengqian · 2024-11-07T09:54:19Z

Do you mean that proxy does not support etcd at present, only JDBC does? @linghengqian

From my perspective, this is true. Of course, this involves several external issues that have nothing to do with me, and I haven't paid attention to this before.

wuqiong818 · 2024-11-07T09:55:36Z

I don't see separate support for JDBC and Proxy in the latest official documentation.
it took me a lot of time to solve these problems

linghengqian · 2024-11-07T10:01:10Z

I don't see separate support for JDBC and Proxy in the latest official documentation.

it took me a lot of time to solve these problems

Feel free to consider submitting your fix PR to update the documentation. The ASF community is essentially a volunteer community, and outdated documentation is not something that is not allowed. Especially when you observe communities such as Hive that use ASF Confluence as a documentation site.

wuqiong818 · 2024-11-07T10:05:02Z

Thank you very much for your answer. @linghengqian
I'm just teasing that if the documentation were clear, it would get more developers to choose ShardingSphere

linghengqian · 2024-11-08T16:00:04Z

At this time, the company Intranet, you need to change to your own etcd

I realized from your usage that etcd's documentation is a bit outdated, and etcd docs are encouraging people to use something that has not been maintained for 4 years. I opened Remove all references to rkt in documentation etcd-io/website#912 for this reason.

wuqiong818 · 2024-11-10T07:24:27Z

At this time, the company Intranet, you need to change to your own etcd

I realized from your usage that etcd's documentation is a bit outdated, and etcd docs are encouraging people to use something that has not been maintained for 4 years. I opened Remove all references to rkt in documentation etcd-io/website#912 for this reason.

hello，I don't think it has much to do with where Etcd is deployed.
I can use etcdctl normal request On the same device.

linghengqian · 2024-11-11T20:47:07Z

Most people don't need to pay attention to the current issue. Multiple management issues of elasticjob prevent the current issue from being resolved.

terrymanu added the status: need more info label Nov 6, 2024

linghengqian self-assigned this Nov 7, 2024

linghengqian added status: pending and removed status: need more info labels Nov 7, 2024

linghengqian removed their assignment Nov 7, 2024

linghengqian added type: enhancement mode: cluster and removed status: pending labels Nov 7, 2024

linghengqian changed the title ~~Cannot invoke "java.lang.Integer.intValue()" because "workerId" is null~~ Support using Etcd as the registry in shardingsphere proxy Nov 7, 2024

linghengqian added in: proxy status: pending labels Nov 7, 2024

wuqiong818 changed the title ~~Support using Etcd as the registry in shardingsphere proxy~~ Unsupport using Etcd as the registry in shardingsphere proxy Nov 7, 2024

linghengqian mentioned this issue Nov 8, 2024

Remove all references to rkt in documentation etcd-io/website#912

Closed

linghengqian added status: volunteer wanted and removed status: pending labels Nov 8, 2024

linghengqian added status: blocked and removed status: volunteer wanted labels Nov 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unsupport using Etcd as the registry in shardingsphere proxy #33565

Unsupport using Etcd as the registry in shardingsphere proxy #33565

wuqiong818 commented Nov 6, 2024

wuqiong818 commented Nov 6, 2024 •

edited

Loading

wuqiong818 commented Nov 6, 2024

terrymanu commented Nov 6, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

terrymanu commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024 •

edited

Loading

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024 •

edited

Loading

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024 •

edited

Loading

linghengqian commented Nov 7, 2024 •

edited

Loading

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 8, 2024

wuqiong818 commented Nov 10, 2024 •

edited

Loading

linghengqian commented Nov 11, 2024

Unsupport using Etcd as the registry in shardingsphere proxy #33565

Unsupport using Etcd as the registry in shardingsphere proxy #33565

Comments

wuqiong818 commented Nov 6, 2024

Bug Report

wuqiong818 commented Nov 6, 2024 • edited Loading

wuqiong818 commented Nov 6, 2024

terrymanu commented Nov 6, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

terrymanu commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024 • edited Loading

my image

my global.yaml

my database-sharding.yaml

operational process

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024 • edited Loading

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024 • edited Loading

linghengqian commented Nov 7, 2024 • edited Loading

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 7, 2024

wuqiong818 commented Nov 7, 2024

linghengqian commented Nov 8, 2024

wuqiong818 commented Nov 10, 2024 • edited Loading

linghengqian commented Nov 11, 2024

wuqiong818 commented Nov 6, 2024 •

edited

Loading

wuqiong818 commented Nov 7, 2024 •

edited

Loading

wuqiong818 commented Nov 7, 2024 •

edited

Loading

wuqiong818 commented Nov 7, 2024 •

edited

Loading

linghengqian commented Nov 7, 2024 •

edited

Loading

wuqiong818 commented Nov 10, 2024 •

edited

Loading