[improve][broker, build, client, fn, io, ml, proxy, sql, test] Merge apache/branch-3.1 into 3.1_ds (II) #215
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(cherry picked from commit 1bf7371)
apache#21278) (cherry picked from commit 682eb36) # Conflicts: # pom.xml
…ear (apache#21302) (cherry picked from commit 70d086f)
(cherry picked from commit e76a86e) # Conflicts: # pulsar-common/src/test/java/org/apache/pulsar/common/nar/NarUnpackerTest.java
…pache#21329) (cherry picked from commit c883f50)
…AuthorizationProvider plugin (apache#20939) Fixed apache#20938
…apache#21338) Co-authored-by: Lari Hotari <[email protected]>
…g from BookKeeper to Pulsar. (apache#21340) There is no testing for AutoRecovery replication in Pulsar's current test suite, and we need to cover it. So migrate the replication testing from BookKeeper to Pulsar.
Fix zookeeper related flacky test (cherry picked from commit 9ab7417)
…re building (apache#21365) (cherry picked from commit 421c98a)
apache#21155) The client assumed the connection was inactive, but the Broker assumed the connection was fine. The Client tried to use a new connection to reconnect a producer, then got an error `Producer with name 'st-0-5' is already connected to topic`. - In a connection, the second connection waits for the first connection to complete\. But there is a bug that causes this mechanism to fail\. - If a producer uses a default name, the second registration will override the first one. But it can not override the first one if it uses a specified producer name\. I think this mechanism is to prevent a client from creating two producers with the same name. However, method `Producer.isSuccessorTo` has checked the `producer-id`, and the `producer-id` of multiple producers created by the same client are different. So this mechanism can be deleted. - For `issue 1`: If a producer with the same name tries to use a new connection, async checks the old connection is available. The producers related to the connection that is not available are automatically cleaned up. - For `issue 2`: - Fix the bug that causes a complete producer future will be removed from `ServerCnx`. - Remove the mechanism that prevents a producer with a specified name from overriding the previous producer. (cherry picked from commit bda16b6)
(cherry picked from commit 39235ed) # Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ClusterMigrationTest.java
### Motivation As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR. ### Modifications Support decoding UUID form AVRO or JSON schema. (cherry picked from commit 8c70943)
(cherry picked from commit e6cd11f)
) (cherry picked from commit ffc083b)
…le when using extensible load manager (apache#21213) (apache#21314)
…r migration tests (apache#21391) (cherry picked from commit e2c6c08)
…o after bkclient initialize (apache#20944)
…elete heartbeat topic (apache#21360) Co-authored-by: fanjianye <[email protected]> Co-authored-by: Jiwei Guo <[email protected]>
…MinNumRacksPerWriteQuorum not work problem. (apache#21327)
…d avoid recursive update error (apache#21282) (cherry picked from commit aecdb03)
…er (apache#21332) (cherry picked from commit ecd40e4)
(cherry picked from commit 643428b)
(cherry picked from commit fe2d61d)
### Issue: There is an NPE that causes the Future of Delay message indexes bucket deletion to be no longer complete, which leads to the topic deletion timeout. You can reproduce this issue by the test `testDeletePartitionedTopicIfCursorPropsEmpty` and `testDeleteTopicIfCursorPropsEmpty` ### Modifications Fix the NPE. (cherry picked from commit b2f2b53)
…which is not enable replication (apache#21495) ### Motivation [PIP 33](https://github.com/apache/pulsar/wiki/PIP-33%3A-Replicated-subscriptions) introduces a new concept ` Replicated subscriptions`. When a topic has a consumer (subscription) that enables replicated subscriptions, it will write markers into the original topic. Even if there is no replicated cluster configured for this topic, the mark will still be written. And that will make the backlog of the topic keep increasing. --- The mark will be written in the following two ways: 1. A scheduled task writes a marker at a fixed time interval if there are new messages published. https://github.com/apache/pulsar/blob/ea1fc0f20138bc35f54f55d32dabf3c3a3309c8e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/ReplicatedSubscriptionsController.java#L78-L86 https://github.com/apache/pulsar/blob/ea1fc0f20138bc35f54f55d32dabf3c3a3309c8e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/ReplicatedSubscriptionsSnapshotBuilder.java#L77-L86 2. Acknowledging message will trigger a check if the first snapshot is written and the mark delete position moves, if true, It will write a marker. https://github.com/apache/pulsar/blob/ea1fc0f20138bc35f54f55d32dabf3c3a3309c8e/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/ReplicatedSubscriptionsController.java#L114-L150 ### Modifications According to the topic policy to create or remove `ReplicatedSubscriptionsController` of this topic.
…21619) Co-authored-by: Jiwe Guo <[email protected]>
…he#21447) ### Motivation Since the files will be checked for `LICENSE` whose name is a suffix with `.conf`. But we can not add a LICENSE definition for the file `schema_example.conf` because it is a `JSON` file. Note: the file `schema_example.conf` is only used for the integration test `CLITest.java`. ### Modifications Rename `schema_example.conf` to `schema_example.json`.
… if the previous one is inactive (apache#21220) If a producer establishes a new connection when it is reconnecting, while the previous connection is now inactive, the initial request made on the new connection will fail. This failure will trigger the topic of cleaning up the inactive producers. However, upon making a second request, the producer will be able to successfully establish a connection and proceed with the operation. Make the initial request made on the new connection success. (cherry picked from commit 711b621)
### Motivation
When I did this test below and got the error "Topic not found".
```java
pulsarAdmin.topics().createNonPartitionedTopic("persistent://public/default/tp_1");
Consumer consumer = null;
Consumer consumer = pulsarClient.newConsumer()
.topic("persistent://public/default/tp_1")
.subscriptionName("s1")
.enableRetry(true)
.subscribe();
```
I do create the topic `persistent://public/default/tp_1` first but got a response "Topic not found", it is confusing.
The root cause is the retry letter topic `persistent://public/default/tp_1-sub1-RETRY` was not created.
### Modifications
clear the vague response if the topic is not founded.
(cherry picked from commit 1a024bc)
…n on non-persistent topic (apache#21592) Fixes apache#21568 Motivation Fix incorrect unack count when using shared subscription on non-persistent topic Modifications In the case of a non-persistent topic, the consumer does not send an ack to the broker (see org.apache.pulsar.client.impl.NonPersistentAcknowledgmentGroupingTracker# addAcknowledgment) To work around this, we can update unackedMessages when the broker sends a message to the consumer successfully.
Co-authored-by: Lari Hotari <[email protected]> Co-authored-by: xiangying <[email protected]> Co-authored-by: Jiwe Guo <[email protected]>
…endAsync (apache#21673) Signed-off-by: Zixuan Liu <[email protected]>
Signed-off-by: tison <[email protected]>
…anch-3.1 Conflicts: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java pulsar-io/alluxio/pom.xml pulsar-io/hbase/pom.xml pulsar-io/influxdb/pom.xml pulsar-io/jdbc/clickhouse/pom.xml pulsar-io/jdbc/core/pom.xml pulsar-io/jdbc/mariadb/pom.xml pulsar-io/jdbc/openmldb/pom.xml pulsar-io/jdbc/postgres/pom.xml pulsar-io/jdbc/sqlite/pom.xml pulsar-io/redis/pom.xml pulsar-io/solr/pom.xml pulsar-package-management/bookkeeper-storage/pom.xml pulsar-package-management/core/pom.xml pulsar-package-management/filesystem-storage/pom.xml pulsar-package-management/pom.xml testmocks/pom.xml
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
This PR merges apache/branch-3.1 into 3.1_ds.
3.1_ds is currently on par with apache/v3.1.1, this merge makes the branch on par with apache/branch-3.1 which at release v3.1.2
Modifications
Describe the modifications you've done.
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
doc-required(If you need help on updating docs, create a doc issue)
no-need-doc(Please explain why)
doc(If this PR contains doc changes)