Skip to content

[improve][broker] Reduce the pressure from the transaction buffer in rolling restarts #1081

[improve][broker] Reduce the pressure from the transaction buffer in rolling restarts

[improve][broker] Reduce the pressure from the transaction buffer in rolling restarts #1081

Triggered via pull request July 23, 2024 13:43
Status Failure
Total duration 1h 18m 2s
Artifacts 35

pulsar-ci.yaml

on: pull_request
Preconditions
7s
Preconditions
Build and License check
14m 32s
Build and License check
Matrix: unit-tests
Build Pulsar docker image
26m 18s
Build Pulsar docker image
Matrix: Build Pulsar java-test-image docker image
CI - Unit - Upload Coverage
0s
CI - Unit - Upload Coverage
Run CodeQL Analysis
0s
Run CodeQL Analysis
Matrix: system-tests
Matrix: flaky-system-tests
Matrix: integration-tests
CI - System - Upload Coverage
0s
CI - System - Upload Coverage
CI - Integration - Upload Coverage
0s
CI - Integration - Upload Coverage
Build Pulsar on MacOS
0s
Build Pulsar on MacOS
OWASP dependency check
0s
OWASP dependency check
Delete system test docker image artifact
0s
Delete system test docker image artifact
Delete integration test docker image artifact
0s
Delete integration test docker image artifact
Pulsar CI checks completed
3s
Pulsar CI checks completed
Fit to window
Zoom out
Zoom in

Annotations

7 errors and 43 warnings
CI - Unit - Brokers - Broker Group 2
Process completed with exit code 1.
CI - Unit - Brokers - Broker Group 1
Process completed with exit code 1.
MLPendingAckStoreTest.beforeMethod: pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreTest.java#L82
--- An unexpected error occurred in the server --- Message: java.lang.Exception: Attempted to use a fenced managed ledger Stacktrace: org.apache.bookkeeper.mledger.ManagedLedgerException$ManagedLedgerFencedException: java.lang.Exception: Attempted to use a fenced managed ledger Caused by: java.lang.Exception: Attempted to use a fenced managed ledger at org.apache.bookkeeper.mledger.ManagedLedgerException$ManagedLedgerFencedException.<init>(ManagedLedgerException.java:80) at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncReadEntries(ManagedLedgerImpl.java:1888) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.asyncReadEntriesWithSkip(ManagedCursorImpl.java:808) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.asyncReadEntries(ManagedCursorImpl.java:788) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.asyncReadEntries(ManagedCursorImpl.java:782) at org.apache.pulsar.broker.service.persistent.MessageDeduplication.replayCursor(MessageDeduplication.java:178) at org.apache.pulsar.broker.service.persistent.MessageDeduplication.recoverSequenceIdsMap(MessageDeduplication.java:161) at org.apache.pulsar.broker.service.persistent.MessageDeduplication$4.openCursorComplete(MessageDeduplication.java:295) at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$7.operationComplete(ManagedLedgerImpl.java:1026) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$3.operationComplete(ManagedCursorImpl.java:734) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$3.operationComplete(ManagedCursorImpl.java:730) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$25.operationComplete(ManagedCursorImpl.java:2729) at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$25.operationComplete(ManagedCursorImpl.java:2725) at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$asyncUpdateCursorInfo$16(MetaStoreImpl.java:267) at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) at org.apache.bookkeeper.common.util.SingleThreadSafeScheduledExecutorService$SafeRunnable.run(SingleThreadSafeScheduledExecutorService.java:46) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840)
Pulsar CI checks completed
Process completed with exit code 1.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8ClientTests created 9 new threads. There are now 18 threads in total.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8SinkTests created 1 new threads. There are now 19 threads in total.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8AuthTests created 11 new threads. There are now 28 threads in total.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v8.ElasticSearch8SinkRawDataTests created 3 new threads. There are now 31 threads in total.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.v7.ElasticSearch7ClientTests created 2 new threads. There are now 33 threads in total.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.ElasticSearchBWCTests created 6 new threads. There are now 39 threads in total.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.opensearch.OpenSearchClientSslTests created 18 new threads. There are now 55 threads in total.
CI - Unit - Pulsar IO - Elastic Search
Summary: Tests in class org.apache.pulsar.io.elasticsearch.ElasticSearchRawRecordTests created 96 new threads. There are now 151 threads in total.
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.broker.delayed.bucket.BucketDelayedDeliveryTrackerTest created 1 new threads. There are now 10 threads in total.
CI - Unit - Brokers - Broker Group 2
Summary: Tests in class org.apache.pulsar.broker.delayed.InMemoryDeliveryTrackerTest created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorRollingRestartTest created 47 new threads. There are now 57 threads in total.
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.TestReplicationWorker created 17 new threads. There are now 27 threads in total.
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.BookieAutoRecoveryTest created 15 new threads. There are now 25 threads in total.
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorCheckAllLedgersTaskTest created 1 new threads. There are now 11 threads in total.
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorPeriodicCheckTest created 11 new threads. There are now 21 threads in total.
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AutoRecoveryMainTest created 19 new threads. There are now 29 threads in total.
CI - Unit - Pulsar Metadata
Summary: Tests in class org.apache.bookkeeper.replication.AuditorBookieTest created 28 new threads. There are now 38 threads in total.
CI - Unit - Brokers - Client Impl
Summary: Tests in class org.apache.pulsar.client.impl.ReaderTest created 8 new threads. There are now 17 threads in total.
CI - Unit - Brokers - Broker Group 3
Summary: Tests in class org.apache.pulsar.broker.admin.AdminApiTest created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.http.HttpSinkTest created 81 new threads. There are now 90 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.rabbitmq.source.RabbitMQSourceTest created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.rabbitmq.sink.RabbitMQSinkTest created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.kinesis.KinesisSinkTest created 9 new threads. There are now 19 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs3.sink.text.HdfsStringSinkTests created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs3.sink.seq.HdfsSequentialSinkTests created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs3.sink.seq.HdfsTextSinkTests created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs2.sink.text.HdfsStringSinkTests created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs2.sink.seq.HdfsSequentialSinkTests created 1 new threads. There are now 10 threads in total.
CI - Unit - Pulsar IO
Summary: Tests in class org.apache.pulsar.io.hdfs2.sink.seq.HdfsTextSinkTests created 1 new threads. There are now 10 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMappingTest created 1 new threads. There are now 11 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryChangeLedgerPathTest created 16 new threads. There are now 26 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.offload.filesystem.impl.FileSystemManagedLedgerOffloaderTest created 6 new threads. There are now 16 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.testclient.PerfClientUtilsTest created 12 new threads. There are now 22 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.broker.authentication.AuthenticationProviderAthenzTest created 1 new threads. There are now 11 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.broker.authentication.ProxySaslAuthenticationTest created 2 new threads. There are now 12 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageTest created 2 new threads. There are now 12 threads in total.
CI - Unit - Other
Summary: Tests in class org.apache.bookkeeper.mledger.impl.ManagedLedgerTest created 10 new threads. There are now 20 threads in total.
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.broker.transaction.pendingack.impl.MLPendingAckStoreTest created 73 new threads. There are now 82 threads in total.
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.broker.service.TopicGCTest created 2 new threads. There are now 84 threads in total.
CI - Unit - Brokers - Broker Group 1
Summary: Tests in class org.apache.pulsar.PulsarBrokerStarterTest created 4 new threads. There are now 88 threads in total.
CI - Integration - Messaging
Received request to deprovision: The request was cancelled by the remote provider.
CI - System - Pulsar Connectors - Process
Received request to deprovision: The request was cancelled by the remote provider.
Deprecation notice: v1, v2, and v3 of the artifact actions
The following artifacts were uploaded using a version of actions/upload-artifact that is scheduled for deprecation: "coverage_and_deps_inttest_CLI.tar.zst", "coverage_and_deps_inttest_METRICS.tar.zst", "coverage_and_deps_inttest_STANDALONE.tar.zst", "coverage_and_deps_inttest_TRANSACTION.tar.zst", "coverage_and_deps_systest_FUNCTION.tar.zst", "coverage_and_deps_systest_PULSAR_CONNECTORS_THREAD.tar.zst", "coverage_and_deps_systest_PULSAR_IO.tar.zst", "coverage_and_deps_systest_SCHEMA.tar.zst", "coverage_and_deps_systest_TIERED_FILESYSTEM.tar.zst", "coverage_and_deps_systest_TIERED_JCLOUD.tar.zst", "coverage_and_deps_unittest_BROKER_CLIENT_API.tar.zst", "coverage_and_deps_unittest_BROKER_CLIENT_IMPL.tar.zst", "coverage_and_deps_unittest_BROKER_GROUP_3.tar.zst", "coverage_and_deps_unittest_BROKER_GROUP_4.tar.zst", "coverage_and_deps_unittest_CLIENT.tar.zst", "coverage_and_deps_unittest_METADATA.tar.zst", "coverage_and_deps_unittest_OTHER.tar.zst", "coverage_and_deps_unittest_PROXY.tar.zst", "coverage_and_deps_unittest_PULSAR_IO.tar.zst", "coverage_and_deps_unittest_PULSAR_IO_ELASTIC.tar.zst", "coverage_and_deps_unittest_PULSAR_IO_KAFKA_CONNECT.tar.zst", "pulsar-java-test-image.zst", "pulsar-maven-repository-binaries.tar.zst", "pulsar-server-distribution.tar.zst", "pulsar-test-latest-version-image.zst". Please update your workflow to use v4 of the artifact actions. Learn more: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/

Artifacts

Produced during runtime
Name Size
Unit-BROKER_CLIENT_IMPL-dumps Expired
4.17 KB
Unit-BROKER_GROUP_1-dumps Expired
27.5 KB
Unit-BROKER_GROUP_1-surefire-reports Expired
33 MB
Unit-BROKER_GROUP_2-dumps Expired
7.91 KB
Unit-BROKER_GROUP_2-surefire-reports Expired
13.2 MB
Unit-BROKER_GROUP_3-dumps Expired
3.64 KB
Unit-METADATA-dumps Expired
38.3 KB
Unit-OTHER-dumps Expired
34.2 KB
Unit-PULSAR_IO-dumps Expired
75.2 KB
Unit-PULSAR_IO_ELASTIC-dumps Expired
47.6 KB
coverage_and_deps_inttest_CLI.tar.zst Expired
91.3 MB
coverage_and_deps_inttest_METRICS.tar.zst Expired
303 KB
coverage_and_deps_inttest_STANDALONE.tar.zst Expired
298 KB
coverage_and_deps_inttest_TRANSACTION.tar.zst Expired
308 KB
coverage_and_deps_systest_FUNCTION.tar.zst Expired
306 KB
coverage_and_deps_systest_PULSAR_CONNECTORS_THREAD.tar.zst Expired
327 KB
coverage_and_deps_systest_PULSAR_IO.tar.zst Expired
329 KB
coverage_and_deps_systest_SCHEMA.tar.zst Expired
310 KB
coverage_and_deps_systest_TIERED_FILESYSTEM.tar.zst Expired
91.2 MB
coverage_and_deps_systest_TIERED_JCLOUD.tar.zst Expired
302 KB
coverage_and_deps_unittest_BROKER_CLIENT_API.tar.zst Expired
2.13 MB
coverage_and_deps_unittest_BROKER_CLIENT_IMPL.tar.zst Expired
2.15 MB
coverage_and_deps_unittest_BROKER_GROUP_3.tar.zst Expired
2.16 MB
coverage_and_deps_unittest_BROKER_GROUP_4.tar.zst Expired
2.04 MB
coverage_and_deps_unittest_CLIENT.tar.zst Expired
645 KB
coverage_and_deps_unittest_METADATA.tar.zst Expired
251 KB
coverage_and_deps_unittest_OTHER.tar.zst Expired
9.9 MB
coverage_and_deps_unittest_PROXY.tar.zst Expired
127 KB
coverage_and_deps_unittest_PULSAR_IO.tar.zst Expired
366 KB
coverage_and_deps_unittest_PULSAR_IO_ELASTIC.tar.zst Expired
90.5 KB
coverage_and_deps_unittest_PULSAR_IO_KAFKA_CONNECT.tar.zst Expired
173 KB
pulsar-java-test-image.zst Expired
277 MB
pulsar-maven-repository-binaries.tar.zst Expired
293 MB
pulsar-server-distribution.tar.zst Expired
205 MB
pulsar-test-latest-version-image.zst Expired
1.72 GB