[fix][txn] Enable client without system topics permission to use transactions #314
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tionedTopic (cherry picked from commit 6e849fc)
…etrics gzip compression (apache#22576) (cherry picked from commit 997c8b9) (cherry picked from commit fe05e08)
…apache#22566) (cherry picked from commit 69839c7) (cherry picked from commit 701d4cf)
…n applicable (apache#22596) (cherry picked from commit 3b9602c) (cherry picked from commit d00d715)
apache#22600) (cherry picked from commit 5d9ccd4) (cherry picked from commit bfdbd20)
…pache#21640) (cherry picked from commit 7fdfbf4)
…ache#21380) (cherry picked from commit cb7c98a) (cherry picked from commit 8ecfd64)
…apache#22583) (cherry picked from commit 3de14c5) (cherry picked from commit 72833fe)
… replicateSubscriptionState (apache#22572) (cherry picked from commit a761b97) (cherry picked from commit 1dacca5)
…oad non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (apache#22580) (cherry picked from commit 340d60d) (cherry picked from commit 053c455)
(cherry picked from commit fd823f6)
…ache#22662) (cherry picked from commit a7e1fcd) (cherry picked from commit bfd4153)
… an orphan replicator in the previous topic owner (apache#21946) (cherry picked from commit 4924052) (cherry picked from commit 670aff0)
…e topic level replication (apache#22537) (cherry picked from commit d475655) (cherry picked from commit 076b55e)
… an orphan replicator in the previous topic owner (apache#21948) (cherry picked from commit b774666) (cherry picked from commit 6038bbf)
…ache#17524) (cherry picked from commit 93afd89) (cherry picked from commit 620fe9b)
…ache#22673) (cherry picked from commit ada31a9) (cherry picked from commit f2c4367)
… cache (apache#22679) Co-authored-by: Jiwe Guo <[email protected]> (cherry picked from commit 566330c) (cherry picked from commit f091bcb)
…che#22683) (cherry picked from commit 395a886)
…tion (apache#23179) (cherry picked from commit 7f04364) (cherry picked from commit 4196195)
…t when ServiceUnitState start (ExtensibleLoadManagerImpl only) (apache#23152) (cherry picked from commit 3053b64) (cherry picked from commit 9a090f7)
…the topic is a system topic (apache#23185) (cherry picked from commit 73433cd) (cherry picked from commit 09bc877)
… failed topic creation future (apache#23184) (cherry picked from commit 9edaa85) (cherry picked from commit 34c6fa0)
…with ongoing txn (apache#23189) (cherry picked from commit 94e1341) (cherry picked from commit b7ffa73)
…orrect failed time (apache#23199) Co-authored-by: fanjianye <[email protected]> (cherry picked from commit b661ec8) (cherry picked from commit b7f5026)
…ent (apache#23206) (cherry picked from commit 66e1a06) (cherry picked from commit 06ebe2a)
…pache#22372) Co-authored-by: xiangying <[email protected]> ### Motivation When a message is not set value, the `nullValue` message metadata should be true and change to false after the value is set. Otherwise, the message data will be set as a [] when the value is not set, that would cause the message data to be encoded and throw a `SchemaSerializationException` when calling `reconsumerLater`. ``` org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.SchemaSerializationException: Size of data received by IntSchema is not 4 at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1131) at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLater(ConsumerBase.java:467) at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLater(ConsumerBase.java:452) at org.apache.pulsar.client.api.ConsumerRedeliveryTest.testRedeliverMessagesWithoutValue(ConsumerRedeliveryTest.java:445) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139) at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:677) at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221) at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50) at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969) at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194) at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148) at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at org.testng.TestRunner.privateRun(TestRunner.java:829) at org.testng.TestRunner.run(TestRunner.java:602) at org.testng.SuiteRunner.runTest(SuiteRunner.java:437) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391) at org.testng.SuiteRunner.run(SuiteRunner.java:330) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256) at org.testng.TestNG.runSuitesLocally(TestNG.java:1176) at org.testng.TestNG.runSuites(TestNG.java:1099) at org.testng.TestNG.run(TestNG.java:1067) at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:65) at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105) Caused by: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.SchemaSerializationException: Size of data received by IntSchema is not 4 at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLater(ConsumerBase.java:462) ... 29 more Caused by: org.apache.pulsar.client.api.SchemaSerializationException: Size of data received by IntSchema is not 4 at org.apache.pulsar.client.impl.schema.IntSchema.validate(IntSchema.java:49) at org.apache.pulsar.client.impl.schema.AutoProduceBytesSchema.encode(AutoProduceBytesSchema.java:80) at org.apache.pulsar.client.impl.schema.AutoProduceBytesSchema.encode(AutoProduceBytesSchema.java:32) at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.lambda$value$3(TypedMessageBuilderImpl.java:157) at java.base/java.util.Optional.orElseGet(Optional.java:364) at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.value(TypedMessageBuilderImpl.java:156) at org.apache.pulsar.client.impl.ConsumerImpl.doReconsumeLater(ConsumerImpl.java:689) at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.doReconsumeLater(MultiTopicsConsumerImpl.java:550) at org.apache.pulsar.client.impl.ConsumerBase.reconsumeLaterAsync(ConsumerBase.java:574) ``` ### Modifications When a message is not set value, the `nullValue` message metadata should be true and change to false after the value is set. (cherry picked from commit f3c177e) (cherry picked from commit 0541176)
…itching ledgers (apache#23209) (cherry picked from commit 0a5cb51) (cherry picked from commit 57b0ca4)
This function was recently changed causing a failure in broker metrics.
…n remote cluster through replication (apache#23169) (cherry picked from commit 44f9860) (cherry picked from commit 9553c33)
…_disconnected_count" (apache#23213) (cherry picked from commit 09a16c2) (cherry picked from commit fbf5268)
This reverts commit d60f9d3.
This reverts commit 4322edf.
…rove Key_Shared observability (apache#23224) (cherry picked from commit 59424a8) (cherry picked from commit 766d2a4)
… specific subscription (apache#22861) Co-authored-by: duanlinlin <[email protected]> [PIP-359](apache#22902) Support custom message listener thread pool for specific subscription, avoid individual subscription listener consuming too much time leading to higher consumption delay in other subscriptions. <!-- ### Contribution Checklist - PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*. - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review. - Each pull request should address only one issue, not mix up code from multiple issues. - Each commit in the pull request has a meaningful commit message - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below. --> <!-- Either this PR fixes an issue, --> <!-- or this PR is one task of an issue --> <!-- If the PR belongs to a PIP, please add the PIP link here --> <!-- Details of when a PIP is required and how the PIP process work, please see: https://github.com/apache/pulsar/blob/master/pip/README.md --> ### Motivation In our scenario, there is a centralized message proxy service, this service will use the same PulsarClient instance to create a lot of subscription groups to consume many topics and cache messages locally.Then the business will pull messages from the cache of the proxy service. It seems that there is no problem, but during use, we found that when the message processing time of several consumer groups (listener mode) is very high, it almost affects all consumer groups responsible for the proxy service, causing a large number of message delays. By analyzing the source code, we found that by default, all consumer instances created from the same PulsarClient will share a thread pool to process message listeners, and sometimes there are multiple consumer message listeners bound to the same thread. Obviously, when a consumer processes messages and causes long-term blocking, it will cause the messages of other consumers bound to the thread to fail to be processed in time, resulting in message delays. Therefore, for this scenario, it may be necessary to support specific a message listener thread pool with consumer latitudes to avoid mutual influence between different consumers. <!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. --> ### Modifications Support custom message listener thread pool for specific subscription. <!-- Describe the modifications you've done. --> (cherry picked from commit 10f4e02) (cherry picked from commit c5846bb)
…the topic (apache#23237) (cherry picked from commit aee2ee5) (cherry picked from commit 311b6af)
|
oops, wrong target |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re-do of 26ef2f8 and 3fb1971 with adjustments to 3.x codebase.
Also see apache#18718
This change is not in the upstream and likely will not be as per https://lists.apache.org/thread/9wtfnzn6h0glyx5w6w69mmgsokl9xq86