MQ V2 OSS release (#136)

* Only once (#23) squash and merge of exactly once work, history is available in the onlyOnce-history branch * feat: Update mq image to latest (#45) Contributes to: event-integration/qp-planning#10883 Signed-off-by: Joel Hanson <[email protected]> * fix: Include appropriate license in packaging (#46) * fix:L Include appropriate license in packaging Include appropriate license in packaging Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad<[email protected]> Signed-off-by: neeraj-laad <[email protected]> * fix: Move Licenses folder under META-INF Move Licenses folder under META-INF Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <[email protected]> * fix: update license in README update license in README Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <[email protected]> * fix: fix the no-auth MQ integration tests (#48) We use the latest tag MQ image in our integration tests. In December, MQ's container image removed the no-auth svrconn channel so this breaks the connector tests that try to make connections to MQ without credentials. This commit adds a custom MQSC script to configure the queue manager to restore the previous behaviour. The MQSourceTaskAuthIT tests still test the ability to connect to a queue manager with auth credentials, so this commit means we still test connecting with and without credentials. Contributes-to: event-integration/eventstreams-planning#12427 Signed-off-by: Dale Lane <[email protected]> * refactor: delegate config validation to Kafka Connect (#50) * feat: pull in changes from origin Contributes to: event-integration/eventstreams-planning#12467 Signed-off-by: Joel Hanson <[email protected]> * feat: Fix suggested review comments Contributes to: event-integration/eventstreams-planning#12467 Signed-off-by: Joel Hanson <[email protected]> --------- Signed-off-by: Joel Hanson <[email protected]> * fix: avoid dead-locking the SourceTask if all commits aren't received (#51) * chore: comment updates - no functional changes Fixed some weird line-wrapping in the existing comments, edited comments that were technically inaccurate, and added some extra clarification to comments ahead of changes in the next commit. Signed-off-by: Dale Lane <[email protected]> * test: new unit tests that illustrate desired behaviour poll() should quickly return an empty list if we are still waiting for a previous batch to complete, but throw an exception if this happens too many times. Signed-off-by: Dale Lane <[email protected]> * fix: avoid deadlocking poll calls when commitRecord isnt called Signed-off-by: Dale Lane <[email protected]> * chore: prepare new release I also removed an invalid config option that was included in the pom.xml at some point. Signed-off-by: Dale Lane <[email protected]> * feat: Address review comments Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <[email protected]> * fix: Update kafka connect-api and connect-json version Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <[email protected]> --------- Signed-off-by: Dale Lane <[email protected]> Signed-off-by: Joel Hanson <[email protected]> Co-authored-by: Joel Hanson <[email protected]> * feat: Automate mq source release (#52) Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <[email protected]> * fix: travis release (#53) Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Joel Hanson <[email protected]> * feat: Addressing review comments Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] feat: updated versions in pom.xml Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] feat: Addressing review comments Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] feat: Addressing review comments Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] Updated BUG Report Template Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] feat: updated copyright year 2024 Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] feat: addressed the review comments and updated the version to 2.1.0 Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] feat: updated the License folder Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] feat: Updated License and Travis for MQ 2.0.2 Contributes to: event-integration/eventstreams-planning#12988 Signed-off-by: Priyanka.K.U [email protected] --------- Signed-off-by: Joel Hanson <[email protected]> Signed-off-by: neeraj-laad <[email protected]> Signed-off-by: Dale Lane <[email protected]> Signed-off-by: Priyanka.K.U [email protected] Co-authored-by: Joseph Vullo <[email protected]> Co-authored-by: Joel Hanson <[email protected]> Co-authored-by: Neeraj Laad <[email protected]> Co-authored-by: Dale Lane <[email protected]> Co-authored-by: Priyanka K U <[email protected]>
ibm-messaging · Aug 7, 2024 · bbb1102 · bbb1102
1 parent 99fc0ff
commit bbb1102
Show file tree

Hide file tree

Showing 57 changed files with 5,628 additions and 1,247 deletions.
diff --git a/.github/ISSUE_TEMPLATE/BUG-REPORT.yml b/.github/ISSUE_TEMPLATE/BUG-REPORT.yml
@@ -57,7 +57,8 @@ body:
       label: Version
       description: What version of our software are you running?
       options:
-        - 1.3.5 (Default)
+        - 2.1.0 (Default)
+        - 1.3.5 
         - older (<1.3.5)
     validations:
       required: true

diff --git a/.gitignore b/.gitignore
@@ -15,4 +15,8 @@ target/
 *.iws
 
 # Visual Studio Code
-.vscode/
+.vscode/
+
+mqjms.log.*
+/src/main/resources/simplelogger.properties
+.envrc
diff --git a/Dockerfile b/Dockerfile
@@ -5,7 +5,7 @@ FROM alpine as builder
 RUN apk update
 RUN apk --no-cache add curl
 
-RUN curl -L "https://downloads.apache.org/kafka/3.4.1/kafka_2.12-3.4.1.tgz" -o kafka.tgz
+RUN curl -L "https://downloads.apache.org/kafka/3.6.2/kafka_2.12-3.6.2.tgz" -o kafka.tgz
 RUN mkdir /opt/kafka \
     && tar -xf kafka.tgz -C /opt/kafka --strip-components=1
 
@@ -27,4 +27,4 @@ EXPOSE 8083
 
 USER esuser
 
-ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]
+ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]
diff --git a/LICENSE b/LICENSE
@@ -175,7 +175,7 @@ Apache License
 
    END OF TERMS AND CONDITIONS
 
-   Copyright 2017 IBM Corporation
+   Copyright 2017, 2024 IBM Corporation
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

diff --git a/README.md b/README.md
diff --git a/config/mq-source-exactly-once.json b/config/mq-source-exactly-once.json
@@ -0,0 +1,19 @@
+{
+    "name": "mq-source-exactly-once",
+    "config":
+    {
+        "connector.class": "com.ibm.eventstreams.connect.mqsource.MQSourceConnector",
+        "tasks.max": "1",
+        "topic": "<TOPIC>",
+
+        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
+        "value.converter": "org.apache.kafka.connect.storage.StringConverter",
+
+        "mq.queue.manager": "<QUEUE_MANAGER>",
+        "mq.connection.name.list": "<CONNECTION_NAME_LIST>",
+        "mq.channel.name": "<CHANNEL_NAME>",
+        "mq.queue": "<QUEUE>",
+        "mq.record.builder": "com.ibm.eventstreams.connect.mqsource.builders.DefaultRecordBuilder",        
+        "mq.exactly.once.state.queue": "<EXACTLY_ONCE_STATE_QUEUE>"
+    }
+}
diff --git a/config/mq-source.properties b/config/mq-source.properties
@@ -1,4 +1,4 @@
-# Copyright 2017, 2020 IBM Corporation
+# Copyright 2017, 2020, 2023, 2024 IBM Corporation
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,6 +16,8 @@ name=mq-source
 connector.class=com.ibm.eventstreams.connect.mqsource.MQSourceConnector
 
 # You can increase this for higher throughput, but message ordering will be lost
+# Exactly-once message delivery requires tasks.max to be 1
+# (see README.md for more details).
 tasks.max=1
 
 # The name of the target Kafka topic - required
@@ -37,6 +39,10 @@ mq.channel.name=
 # The name of the source MQ queue - required
 mq.queue=
 
+# This is required for exactly-once delivery. If not supplied, message delivery will be at-least-once.
+# (see README.md for more details).
+# mq.exactly.once.state.queue=
+
 # The user name for authenticating with the queue manager - optional
 # mq.user.name=
 

diff --git a/pom.xml b/pom.xml
@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!--
- * Copyright 2017, 2020, 2023 IBM Corporation
+ * Copyright 2017, 2020, 2023, 2024 IBM Corporation
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -13,15 +13,14 @@
  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  *  See the License for the specific language governing permissions and
  *  limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0"
-    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.ibm.eventstreams.connect</groupId>
     <artifactId>kafka-connect-mq-source</artifactId>
     <packaging>jar</packaging>
-    <version>1.3.5</version>
+    <version>2.1.0</version>
     <name>kafka-connect-mq-source</name>
     <organization>
         <name>IBM Corporation</name>
@@ -46,13 +45,13 @@
         <dependency>
             <groupId>org.apache.kafka</groupId>
             <artifactId>connect-api</artifactId>
-            <version>3.4.0</version>
+            <version>3.6.2</version>
             <scope>provided</scope>
         </dependency>
         <dependency>
             <groupId>org.apache.kafka</groupId>
             <artifactId>connect-json</artifactId>
-            <version>3.4.0</version>
+            <version>3.6.2</version>
             <scope>provided</scope>
         </dependency>
 
@@ -81,14 +80,14 @@
         </dependency>
         <dependency>
             <groupId>org.slf4j</groupId>
-            <artifactId>slf4j-log4j12</artifactId>
+            <artifactId>slf4j-api</artifactId>
             <version>2.0.7</version>
-            <scope>test</scope>
         </dependency>
         <dependency>
             <groupId>org.slf4j</groupId>
-            <artifactId>slf4j-api</artifactId>
+            <artifactId>slf4j-simple</artifactId>
             <version>2.0.7</version>
+            <scope>test</scope>
         </dependency>
         <!-- tests in src/integration depend on a running MQ queue manager -->
         <!--  in a container, configured using org.testcontainers          -->
@@ -98,10 +97,34 @@
             <version>1.17.6</version>
             <scope>test</scope>
         </dependency>
+
+        <dependency>
+            <groupId>org.assertj</groupId>
+            <artifactId>assertj-core</artifactId>
+            <version>3.24.2</version>
+            <scope>test</scope>
+        </dependency>
+
     </dependencies>
 
     <build>
         <plugins>
+            <plugin>
+                <artifactId>maven-resources-plugin</artifactId>
+                <version>3.3.1</version>
+                <configuration>
+                    <outputDirectory>${project.build.outputDirectory}/META-INF</outputDirectory>
+                    <resources>
+                        <resource>
+                            <directory>${project.basedir}</directory>
+                            <includes>
+                                <include>LICENSE</include>
+                            </includes>
+                        </resource>
+                    </resources>
+                </configuration>
+            </plugin>
+
             <plugin>
                 <artifactId>maven-compiler-plugin</artifactId>
                 <version>3.1</version>
@@ -121,6 +144,9 @@
                     <systemPropertyVariables>
                         <connectorVersion>${project.version}</connectorVersion>
                     </systemPropertyVariables>
+                    <additionalClasspathElements>
+                        <additionalClasspathElement>src/test/resources</additionalClasspathElement>
+                    </additionalClasspathElements>
                 </configuration>
             </plugin>
 
@@ -133,6 +159,9 @@
                     <systemPropertyVariables>
                         <connectorVersion>${project.version}</connectorVersion>
                     </systemPropertyVariables>
+                    <additionalClasspathElements>
+                        <additionalClasspathElement>src/test/resources</additionalClasspathElement>
+                    </additionalClasspathElements>
                 </configuration>
                 <executions>
                     <execution>
@@ -187,7 +216,6 @@
                 </executions>
             </plugin>
 
-
             <!-- generate test code coverage report -->
             <plugin>
                 <groupId>org.jacoco</groupId>
@@ -200,8 +228,7 @@
                             <goal>prepare-agent</goal>
                         </goals>
                         <configuration>
-                            <destFile>
-                                ${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</destFile>
+                            <destFile>${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</destFile>
                             <propertyName>surefire.jacoco.args</propertyName>
                         </configuration>
                     </execution>
@@ -212,10 +239,9 @@
                             <goal>report</goal>
                         </goals>
                         <configuration>
-                            <dataFile>
-                                ${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</dataFile>
-                            <outputDirectory>
-                                ${project.reporting.outputDirectory}/jacoco-unit-test-coverage-report</outputDirectory>
+                            <dataFile>${project.build.directory}/jacoco-output/jacoco-unit-tests.exec</dataFile>
+                            <outputDirectory>${project.reporting.outputDirectory}/jacoco-unit-test-coverage-report
+                            </outputDirectory>
                         </configuration>
                     </execution>
                     <execution>
@@ -225,8 +251,7 @@
                             <goal>prepare-agent</goal>
                         </goals>
                         <configuration>
-                            <destFile>
-                                ${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</destFile>
+                            <destFile>${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</destFile>
                             <propertyName>failsafe.jacoco.args</propertyName>
                         </configuration>
                     </execution>
@@ -237,10 +262,10 @@
                             <goal>report</goal>
                         </goals>
                         <configuration>
-                            <dataFile>
-                                ${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</dataFile>
+                            <dataFile>${project.build.directory}/jacoco-output/jacoco-integration-tests.exec</dataFile>
                             <outputDirectory>
-                                ${project.reporting.outputDirectory}/jacoco-integration-test-coverage-report</outputDirectory>
+                                ${project.reporting.outputDirectory}/jacoco-integration-test-coverage-report
+                            </outputDirectory>
                         </configuration>
                     </execution>
                     <execution>
@@ -269,18 +294,18 @@
                         </goals>
                         <configuration>
                             <dataFile>${project.build.directory}/jacoco-output/merged.exec</dataFile>
-                            <outputDirectory>
-                                ${project.reporting.outputDirectory}/jacoco-merged-test-coverage-report</outputDirectory>
+                            <outputDirectory>${project.reporting.outputDirectory}/jacoco-merged-test-coverage-report
+                            </outputDirectory>
                         </configuration>
                     </execution>
                 </executions>
             </plugin>
+
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-checkstyle-plugin</artifactId>
                 <version>3.2.0</version>
                 <configuration>
-                    <encoding>UTF-8</encoding>
                     <consoleOutput>true</consoleOutput>
                     <failsOnError>true</failsOnError>
                     <violationSeverity>warning</violationSeverity>
@@ -301,8 +326,6 @@
                 </executions>
             </plugin>
         </plugins>
-
-        <!-- some integration tests use the custom mqsc file in the resources directory -->
         <testResources>
             <testResource>
                 <directory>src/integration/resources</directory>

diff --git a/src/assembly/package.xml b/src/assembly/package.xml
@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!--
- * Copyright 2018 IBM Corporation
+ * Copyright 2018, 2024 IBM Corporation
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -37,6 +37,7 @@
         <excludes>
           <exclude>**/copyright-exclude</exclude>
           <exclude>META-INF/maven/**</exclude>
+          <exclude>META-INF/LICENSE*</exclude>
         </excludes>
       </unpackOptions>
       <useTransitiveFiltering>true</useTransitiveFiltering>