MQ V2 Sink OSS release #65
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Upstream changes from sink connector
Contributes to: event-integration/qp-planning#10883 Signed-off-by: Joel Hanson <[email protected]>
Contributes to: event-integration/qp-planning#10883 Signed-off-by: Joel Hanson <[email protected]>
* fix: Move config to MQSinkConfig from MQSinkConnector Signed-off-by: Joel Hanson <[email protected]> * tests: New tests to check the duplicates Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * - Added assertj - refactored duplicate record test + added another for connect crash. * - Changed AuthIt tests to have specific host ports. - Refactored testcontainer out from DuplicateIT into AbstractJMSContextIT * feat: Make topic name and partition variables Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * Updated logging packages * feat: Intitial change for exactly once semantics Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: handle unprovided sidequeue scenario Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Intitial change for exactly once semantics Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: checkstyle changes Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * fix: Static variables should be in UPPER_CASE Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Latest changes for mq sink connector Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Update config property names and consume messages as TextMessage Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * Removed a error being sent to standard out. * feat: Test update based on feedback Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Rename JMSWriter to JMSWorker Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: map based only once delivery Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: tests for only-once delivery Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: check if the max tasks is 1 and throw an exception if not Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Update changes based on feedback from PR Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Remove max task check from connector code Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: exception testing for only-once Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * fix: Tests to check JMSRuntimeException and jsonException is only thrown Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * fix: failing tests Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * Updated MQ to remove vulnerability * feat: Rename JMSWriter to JMSWorker Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: map based only once delivery Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: tests for only-once delivery Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: check if the max tasks is 1 and throw an exception if not Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Update changes based on feedback from PR Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Remove max task check from connector code Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Check max task if the connector is configured with side queue Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * fix: rename writer to worker Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Update config value names to match MQ_EXACTLY_ONCE_STATE_QUEUE Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Update state queue exception handling Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Changes with the new configSupportsExactlyOnce method Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Update version to 2.0.0 (ibm-messaging#14) * feat: Update version to 1.6.0 Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Update version to 2.0.0 Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: New tests for atleast once behaviour with failing conditions. (ibm-messaging#18) * feat: New tests for only once to recreate faliure Contributes to: mhub/qp-planning#10748 Signed-off-by: Joel Hanson <[email protected]> * fix: fix licence formatting and new line at the end Contributes to: mhub/qp-planning#10748 Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Add missing Legal disclaimer (ibm-messaging#19) * feat: Add missing Legal disclaimer Contributes to: mhub/qp-planning#<issue number> Signed-off-by: Joel Hanson <[email protected]> * feat: Add current year to the legal disclaimer Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * feat: Add LICENSES Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: Updated to latest LICENSES Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * refactor: Move all the common test variables to a single class (ibm-messaging#20) Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: Joel Hanson <[email protected]> * fix: Review comment fixes (ibm-messaging#21) * fix: rename variable names Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: simplify the jsonprocessingexception failure Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Rename variable names and create method for readablity Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Move SinkRecordBuilder to utils folder Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: Add missing final Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: enable exactlyOnce to be true if tasks.max is not provided Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: spelling for commited -> committed Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: early exits are traced Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: Refactor handleException method Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Update version in BUG_REPORT Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: Rename class SinkRecordBuilder and remove temp variables Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: tidy up in test for clarity Contributes to: event-integration/eventstreams-planning#10774 Signed-off-by: Jonathan Hughes <[email protected]> Co-authored-by: Jonathan Hughes <[email protected]> * feat: Update exception handling similar to source connector (ibm-messaging#22) * fix: Update connectionException to JMSWorkerConnectionException Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Log stack trace instead of printStackTrace Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * refactor: split out large part of jmswoker into a seperate class (ibm-messaging#23) * refactor: split out large part of jmswoker into a seperate class Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * refactor: connect code is refactored have common method Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Move SSLContext related part to seperate class Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: Rename method name to be more meaningful Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Remove unwanted imports and java docs Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * refactor: Exception handling Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * refactor: simplify the put method Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: handle exception from top and clean up Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Formatting code, sorting imports, adding new tests for exceptions (ibm-messaging#24) Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * Update kafka binary to a working version (ibm-messaging#15) * feat: Formatting and updation of readme for exactly once configuration (ibm-messaging#27) * feat: Formatting and updation of read for exactly once configuration - .gitignore update to have the mq logs - Formatted the readme file - Added new documenation for the exactly once semantics - New sample file for the exactly-once connector config Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: Update 3.4.0 to 3.4.1 Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * fix: Update README, add license declaration, remove AbstractConfig (ibm-messaging#28) * fix: Update README, add license declaration, remove AbstractConfig Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: Change DestinationBuilderIT to MessagePropertyIT Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <[email protected]> * feat: attempt to fix e2es (ibm-messaging#29) * feat: attempt to fix e2es Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Jonathan Hughes <[email protected]> * feat: fixed tests Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Jonathan Hughes <[email protected]> * feat: revert from arm64 image from tests Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Jonathan Hughes <[email protected]> * fix: Update required version from 2.6.0 to 0.11.0 (ibm-messaging#30) Contributes to: event-integration/qp-planning#0 Signed-off-by: Joel Hanson <[email protected]> * feat: Update test to have meaning full variable names (ibm-messaging#31) Contributes to: event-integration/qp-planning#0 Signed-off-by: Joel Hanson <[email protected]> Co-authored-by: Joel Hanson <[email protected]> Co-authored-by: Nick Morgan <[email protected]> Co-authored-by: Jonathan Hughes <[email protected]>
* fix: Update packaged licenses Update packaged licenses Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: Neeraj Laad <[email protected]> Signed-off-by: neeraj-laad <[email protected]> * fix: Include license in pom.xml Include license in pom.xml Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <[email protected]> * fix: Move license folder under META-INF Move license folder under META-INF Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <[email protected]>
fix license in readme Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <[email protected]>
* feat: MQ Sink Refactor Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Priyanka.K.U [email protected] * feat: Test cases fixed Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U [email protected] * feat: Test cases updated Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U [email protected] * feat: updated the import Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U [email protected] * feat: added copyright header Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U [email protected] * chore: remove commented-out code Signed-off-by: Dale Lane <[email protected]> --------- Signed-off-by: Dale Lane <[email protected]> Co-authored-by: Dale Lane <[email protected]>
Contributes to: event-integration/eventstreams-planning#12432 Signed-off-by: Joel Hanson <[email protected]>
* feat: Update readme with appropriate mqmd info Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <[email protected]> * Apply suggestions from code review Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <[email protected]> Co-authored-by: Vignesh Kumar Elumalai <[email protected]> * feat: rearrange order of docs in readme Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <[email protected]> * feat: copyright year change and readme case fix Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <[email protected]> * change to note * remove specifically * Apply suggestions from code review --------- Signed-off-by: Joel Hanson <[email protected]> Co-authored-by: Vignesh Kumar Elumalai <[email protected]>
Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <[email protected]>
Contributes to: event-integration/eventstreams-planning#12621 Signed-off-by: Joel Hanson <[email protected]>
…/master' into mqv2-sink-ossrelease
Joel-hanson
requested changes
Aug 6, 2024
Contributes to: event-integration/eventstreams-planning#13003 Signed-off-by: Priyanka.K.U [email protected] feat: updated version in pom xml Contributes to: event-integration/eventstreams-planning#13003 Signed-off-by: Priyanka.K.U [email protected] feat: MQ Sink V2 OSS related changes Contributes to: event-integration/eventstreams-planning#13003 Signed-off-by: Priyanka.K.U [email protected]
priyankaku
force-pushed
the
mqv2-sink-ossrelease
branch
from
fec3967 to
ec7a4c4
Compare
Joel-hanson
reviewed
Aug 7, 2024
Comment on lines
+109
to
+117
| * Put the records in the sink. Usually this should send the records to the sink | ||
| * asynchronously and immediately return. | ||
| * <p> | ||
| * If this operation fails, the SinkTask may throw a | ||
| * {@link org.apache.kafka.connect.errors.RetriableException} to indicate that | ||
| * the framework should attempt to retry the same call again. Other exceptions | ||
| * will cause the task to be stopped immediately. | ||
| * {@link SinkTaskContext#timeout(long)} can be used to set the maximum time | ||
| * before the batch will be retried. |
There was a problem hiding this comment.
Just noticed this, all the comments are in a difficult to read manner.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Release MQ v2 sink connector under apache 2.0 license.
Fixes # (issue)
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Checklist