Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

同步时遇到大key报错,任务退出 #91

Open
pyb0815 opened this issue Aug 12, 2023 · 2 comments
Open

同步时遇到大key报错,任务退出 #91

pyb0815 opened this issue Aug 12, 2023 · 2 comments

Comments

@pyb0815
Copy link

pyb0815 commented Aug 12, 2023

被抛弃key数量到达阈值[1],exception reason[For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter]

同步时遇到大key报错,任务退出。日志信息如上

@pingxingshikong
Copy link
Member

redis是哪个版本的,能否给个测试数据样本?

@lyulong00
Copy link

redis:6.2.6,我们也出现了

2024-06-18T11:10:41.489884829Z 2024-06-18 11:10:41.489|[ WARN ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> KeyValueRdbSyncEventFilter#run:103 | taskId为[6CED8FCB5C2B461ABF34F645BB27B5BB]的任务全量同步结束..进入增量同步模式 time:[158:s]
2024-06-18T11:10:41.500083652Z 2024-06-18 11:10:41.499|[ INFO ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> RedisSocketReplicator#heartbeat:336 | heartbeat started.
2024-06-18T11:10:41.507800374Z 2024-06-18 11:10:41.507|[ WARN ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> KeyValueCommandSyncEventFilter#run:58 | taskId为[6CED8FCB5C2B461ABF34F645BB27B5BB]的任务增量同步开始..
2024-06-18T11:10:41.562375280Z 2024-06-18 11:10:41.562|[ WARN ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> TaskErrorUtils#brokenStatusAndLog:54 | 任务Id【6CED8FCB5C2B461ABF34F645BB27B5BB】任务启动/运行异常停止 ,Class【class syncer.syncerservice.sync.SendCommandWithOutQueue】,异常原因【被抛弃key数量到达阈值[1],exception reason[For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter]】
2024-06-18T11:10:41.562508621Z 2024-06-18 11:10:41.562|[ERROR ]|[6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10] -> SendCommandWithOutQueue#run:153 | [6CED8FCB5C2B461ABF34F645BB27B5BB]抛弃key:setmua:location:465873:1009587 ,class:[class syncer.syncerplusredis.cmd.impl.DefaultCommand]:原因[For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter]
2024-06-18T11:10:41.562968115Z syncer.syncerservice.exception.FilterNodeException: For input string: "QUEUED"->KeyValueCommandSyncEventFilter->KeyValueRdbSyncEventFilter->KeyValueEventDBMappingFilter->KeyValueDataAnalysisFilter->KeyValueTimeCalculationFilter
2024-06-18T11:10:41.562992311Z at syncer.syncerservice.filter.KeyValueTimeCalculationFilter.run(KeyValueTimeCalculationFilter.java:68)
2024-06-18T11:10:41.562998777Z at syncer.syncerservice.filter.KeyValueRunFilterChain.run(KeyValueRunFilterChain.java:42)
2024-06-18T11:10:41.563004338Z at syncer.syncerservice.sync.SendCommandWithOutQueue.run(SendCommandWithOutQueue.java:62)
2024-06-18T11:10:41.563009315Z at syncer.syncerservice.sync.RedisDataSyncTransmissionTask$1.onEvent(RedisDataSyncTransmissionTask.java:184)
2024-06-18T11:10:41.563014474Z at syncer.syncerplusredis.extend.replicator.listener.ValueDumpIterableEventListener.onEvent(ValueDumpIterableEventListener.java:53)
2024-06-18T11:10:41.563019721Z at syncer.syncerplusredis.replicator.AbstractReplicatorListener.doEventListener(AbstractReplicatorListener.java:92)
2024-06-18T11:10:41.563025424Z at syncer.syncerplusredis.replicator.AbstractReplicator.submitEvent(AbstractReplicator.java:105)
2024-06-18T11:10:41.563030947Z at syncer.syncerplusredis.RedisSocketReplicator$RedisSocketReplicatorRetrier.open(RedisSocketReplicator.java:532)
2024-06-18T11:10:41.563036605Z at syncer.syncerplusredis.replicator.AbstractReplicatorRetrier.retry(AbstractReplicatorRetrier.java:103)
2024-06-18T11:10:41.563041934Z at syncer.syncerplusredis.RedisSocketReplicator.open(RedisSocketReplicator.java:135)
2024-06-18T11:10:41.563046785Z at syncer.syncerplusredis.replicator.RedisReplicator.open(RedisReplicator.java:316)
2024-06-18T11:10:41.563050899Z at syncer.syncerservice.sync.RedisDataSyncTransmissionTask.run(RedisDataSyncTransmissionTask.java:194)
2024-06-18T11:10:41.563055585Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2024-06-18T11:10:41.563065049Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2024-06-18T11:10:41.563069482Z at java.lang.Thread.run(Thread.java:748)
2024-06-18T11:10:41.563073602Z 线程正准备关闭...6CED8FCB5C2B461ABF34F645BB27B5BB: threadPoolTaskExecutor-10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants