Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

实时同步运行一段时间后挂掉 #363

Open
bibichuan opened this issue Sep 24, 2024 · 4 comments
Open

实时同步运行一段时间后挂掉 #363

bibichuan opened this issue Sep 24, 2024 · 4 comments

Comments

@bibichuan
Copy link

bibichuan commented Sep 24, 2024

我创建了两个实时同步的实例,从mysql80 到mysql80,刚创建的时候,可以实现实时同步,但是过了一天之后,就挂掉了
c9c0d0bad74f23fbdcbdeb0a328e6b95
这里是部分的日志文件
taskmanager_192.168.80.3_37823-2e2688_log.log

com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 60,200,415 milliseconds ago. The last packet sent successfully to the server was 60,200,415 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1402) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1387) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:386) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:795) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at org.apache.flink.connector.jdbc.statement.FieldNamedPreparedStatementImpl.executeBatch(FieldNamedPreparedStatementImpl.java:65) ~[flink-connector-jdbc-3.1.1-1.17.jar:3.1.1-1.17]
	at com.dtstack.chunjun.connector.jdbc.sink.PreparedStmtProxy.executeBatch(PreparedStmtProxy.java:305) ~[chunjun-connector-jdbc-base-1.12.5.jar:?]
	at com.dtstack.chunjun.connector.jdbc.sink.JdbcOutputFormat.writeMultipleRecordsInternal(JdbcOutputFormat.java:257) ~[chunjun-connector-jdbc-base-1.12.5.jar:?]
	at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.writeRecordInternal(BaseRichOutputFormat.java:500) ~[chunjun-core-1.12.5.jar:?]
	at com.dtstack.chunjun.sink.format.BaseRichOutputFormat.lambda$initTimingSubmitTask$1(BaseRichOutputFormat.java:460) ~[chunjun-core-1.12.5.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_412]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_412]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_412]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_412]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 60,200,415 milliseconds ago. The last packet sent successfully to the server was 60,200,415 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_412]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_412]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_412]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_412]
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:628) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:683) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:155) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.NativeSession.queryServerVariable(NativeSession.java:597) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1394) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	... 15 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_412]
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_412]
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_412]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_412]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_412]
	at com.mysql.cj.protocol.a.CompressedPacketSender.send(CompressedPacketSender.java:188) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.protocol.a.TimeTrackingPacketSender.send(TimeTrackingPacketSender.java:50) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.protocol.a.NativeProtocol.send(NativeProtocol.java:619) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:683) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:155) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.NativeSession.queryServerVariable(NativeSession.java:597) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	at com.mysql.cj.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:1394) ~[mysql-connector-j-8.0.31.jar:8.0.31]
	... 15 more
2024-09-24 10:45:42,208 WARN  com.dtstack.chunjun.dirty.log.LogDirtyDataCollector 
@baisui1981
Copy link
Member

看上面异常是在sink端抛出的异常,这样好办。只要在数据源配置一项中,设置autoReconnect=true参数应该就可以了
截屏2024-09-24 15 52 46

@baisui1981
Copy link
Member

用户通过添加以上reconnect参数经过一段时间测试发现没有效果,依然会断连接。
继续查看文档还需要额外添加一些参数,文档:https://www.cnblogs.com/yhtboke/p/5632757.html

autoReconnect=true&timeBetweenEvictionRunsMillis=26400&testWhileIdle=true&validationQuery=SELECT%201

@bibichuan
Copy link
Author

尝试找到数据库的配置文件 /usr/local/tis/docker-compose/data/cfg_repo/tis_plugin_config/db 目录,里面有各个数据库的配置文件,修改其中的xml,修改其中的 extraParams 参数,然后重启 tis,结果也是报错了:XmlPullParserException: entity reference name can not contain character =' (position: START_TAG seen

autoReconnect=true&timeBetweenEvictionRunsMillis=26400&testWhileIdle=true&validationQuery=SELECT%201

@baisui1981 baisui1981 reopened this Oct 4, 2024
@bibichuan
Copy link
Author

使用了DataX进行数据同步,同一个数据库,就不会有这个问题。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants