Build refactoring for generic connector

[planetf1]

• Import base egeria pom for dependency constraints
• Ensure new connector is build, packaged into a maven artefact, and released

[planetf1]

At some point we need to update the gradle build files/cicd to publish a maven artefact - update here if/when you need help with that/are ready to do it.

The hardest part is probably deciding the name(s) of the artifacts(assuming we keep them all distinct)

This will mean the connector gets published -- even as a snapshot -- so it will be downloadable. Later we can consider released versions.

At that time we should probably add an 'enforcedPlatform' reference in the top level gradle file -- this will cause us to use the same versions as egeria (this is how we did it in the hive megastore connector)

[planetf1]

Agreed in team meeting 30 Aug that I will make the change to get the new connectors published as maven artifacts

[planetf1]

After the recent changes, we generate an action attachment containing the three connectors:

[planetf1]

These are all being published - but with names based on the directory name. Either we need to move the directory to use as the default component name (our usual pattern in egeria) or explicitly specify the project name (probably the simplest for now)

^^ These can be accessed NOW (since I merged the PR that I wrote before my vacation, but was awaiting review so only merged this morning)

I'll update the artifact names later in the week

[planetf1]

^^ @dwolfson @lpalashevski @mandy-chessell

[dwolfson]

@planetf1 Thanks - I looked in both Maven Central and Sonatype for a jdbc.snapshot.jar but didn't see one? Am I looking for the wrong thing or looking in the wrong place?

[planetf1]

The artifacts are only available on the SNAPSHOT repository, since we have not yet done a release for the database connectors (at least not the new ones). I did ask at the f2f but the consensus was not yet.

In the snapshot URL, as an example, the resource connector can currently be found under:

https://oss.sonatype.org/service/local/repositories/snapshots/content/org/odpi/egeria/jdbc-resource-connector/3.13-SNAPSHOT/

However, as these are snapshots, all files include the timestamp ,making it awkward to fetch

So you can issue a query like

wget http://oss.sonatype.org/service/local/artifact/maven/redirect\?r\=snapshots\&g\=org.odpi.egeria\&a\=jdbc-resource-connector\&v\=3.13-SNAPSHOT\&c\=jar

What this shows, however, is that the actual connector is not published ie:

We have javadoc, sources, but no actual code ;-)

Thanks for spotting.

[planetf1]

Connectors now being published with new names:

• egeria-connector-resource-jdbc
• egeria-connector-integration-jdbc
• egeria-connector-event-mapper-polling-database

And...

• egeria-database-connectors is no longer published (as it is just the build parent)
• egeria-connector-postgres has been left with the existing name

As an example we can see this also now includes the jar correctly:

At this point the http query isn't working - this may depend on an index update (or another bug?)

[planetf1]

The http query will work -- for the jar, do not specify a classifier ie:

• http://oss.sonatype.org/service/local/artifact/maven/redirect\?r\=snapshots\&g\=org.odpi.egeria\&a\=egeria-connector-integration-jdbc\&v\=3.13-SNAPSHOT
• http://oss.sonatype.org/service/local/artifact/maven/redirect\?r\=snapshots\&g\=org.odpi.egeria\&a\=egeria-connector-resource-jdbc\&v\=3.13-SNAPSHOT

Note that both jars are currently built without dependencies. The integration connector does require the resource connector. Both require a jdbc connector for the database being used. I've not verified the runtime behaviour to see if anything else extra is required, though there should be no change here from previously

^ @davidradl @dwolfson @lcpopa

[planetf1]

(extra points if anyone spots my error in the current code. cosmetic only!)

[planetf1]

Inspecting .module file -- this is a gradle addition that provides additional metadata for gradle consumers

The top level metadata looks good:

  "component": {
    "group": "org.odpi.egeria",
    "module": "egeria-connector-integration-jdbc",
    "version": "3.13-SNAPSHOT",
    "attributes": {
      "org.gradle.status": "integration"
    }

The java jar is properly represented with:

• A configuration confirming we have a library, using java 11, of type jar / java-api
• Also sources & javadoc listed, with appropriate metadata
• Files of all 3 are correct
• correct dependencies for runtime

Checking the pom

• correct group/artifactid/version
• apache 2 license
• developer list (only me currently: should be updated)
• scm links
• dependencyManagement with scope - and including our current pom import
• good description
• no explicit packaging type (maven defaults to 'jar' so this is correct)

And the files

• javadoc, sources, jar - all look good content/name, non-zero.

And the maven-metadata.xml

• timestamps, versions & list of files (pom, sources jar, module,java jar, javadoc jar) all look correct.

By manual inspection I do not see issues..... at least when publishing to local maven repo

[planetf1]

Checking retrieving the jar via curl:

➜  newc curl -L -X GET "http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-resource-jdbc&v=3.13-SNAPSHOT" --output egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   134  100   134    0     0    740      0 --:--:-- --:--:-- --:--:--   757
100   242  100   242    0     0     96      0  0:00:02  0:00:02 --:--:--   250
100  4550  100  4550    0     0   1741      0  0:00:02  0:00:02 --:--:--     0
➜  newc ls
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
➜  newc file egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar: Zip archive data, at least v1.0 to extract, compression method=deflate
➜  newc jar -tf egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
org/
org/odpi/
org/odpi/openmetadata/
org/odpi/openmetadata/adapters/
org/odpi/openmetadata/adapters/connectors/
org/odpi/openmetadata/adapters/connectors/resource/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnectorProvider.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector$JdbcConnectorAsDataSource.class
➜  newc

This looks ok (repeat for any other connectors)

Next we check with the proper maven tools - and can see that again, the maven artifact is correctly downloaded:

➜  newc ls
➜  newc mvn dependency:get -DrepoUrl=https://oss.sonatype.org/content/repositories/snapshots -Dartifact=org.odpi.egeria:egeria-connector-resource-jdbc:3.13-SNAPSHOT -Ddest=egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:get (default-cli) @ standalone-pom ---
[WARNING] repositoryUrl parameter is deprecated. Use remoteRepositories instead
[INFO] Resolving org.odpi.egeria:egeria-connector-resource-jdbc:jar:3.13-SNAPSHOT with transitive dependencies
Downloading from temp: https://oss.sonatype.org/content/repositories/snapshots/org/odpi/egeria/egeria-connector-resource-jdbc/3.13-SNAPSHOT/maven-metadata.xml
Downloaded from temp: https://oss.sonatype.org/content/repositories/snapshots/org/odpi/egeria/egeria-connector-resource-jdbc/3.13-SNAPSHOT/maven-metadata.xml (1.4 kB at 2.4 kB/s)
[WARNING] destination/dest parameter is deprecated: it will disappear in future version.
[INFO] Copying /Users/jonesn/.m2/repository/org/odpi/egeria/egeria-connector-resource-jdbc/3.13-SNAPSHOT/egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar to /private/tmp/newc/egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
[WARNING] Notice transitive dependencies won't be copied.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.976 s
[INFO] Finished at: 2022-11-07T14:21:05Z
[INFO] ------------------------------------------------------------------------
➜  newc ls
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
➜  newc file *jar
egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar: Zip archive data, at least v1.0 to extract, compression method=deflate
➜  newc jar -tf egeria-connector-resource-jdbc-3.13-SNAPSHOT.jar
META-INF/
META-INF/MANIFEST.MF
org/
org/odpi/
org/odpi/openmetadata/
org/odpi/openmetadata/adapters/
org/odpi/openmetadata/adapters/connectors/
org/odpi/openmetadata/adapters/connectors/resource/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector$JdbcConnectorAsDataSource.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnectorProvider.class
org/odpi/openmetadata/adapters/connectors/resource/jdbc/JdbcConnector.class

[planetf1]

In summary - for this repo's publish flow:

• The maven repo looks valid for both jars, and artifacts with optional classifiers (javadoc, sources, jar-with-dependencies)
• modules files add additional metadata for gradle clients, without impacting maven clients
• The browser on oss.sonatype.org is not working with module files (despite above!)
• our other gradle repos are the same
• the correct sonatype REST api call works to download the snapshot
• the maven client works to download the snapshot

More broadly we need to think about usability

• Should we publish connectors anywhere else?
  • direct download?
  • combined container image I am a little concerned we may have an isolation concern without custom classloader, albeit not so bad for out connectors -- but this still doesn't help 3rd party)
• we should document the format of the URLs to use (this should be in general docs -- but also very explicitly in this repo)
• do we need to add any alternative mechanism for connector download such as
  • symbolic name (precanned name 'jdbc' which expands out to the fully qualified download link)
  • option to run a user-supplied script to retrieve the artifact

I suggest these general issues are followed up in
- egeria-docs (docs)
- egeria-charts (additional capabilities for helm charts)
- egeria (changes to egeria container, or maybe a new repo for a combined image!)
- here (specific docs on this connector - ie download example for using jdbc connector)
- via a team call/open discussion to see how we can do better

Does that help @dwolfson @mandy-chessell ?

[planetf1]

Tested with postgres connectors:

Using a file ~/etc/jdbc.yaml:

extralibs:
  - url: http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-resource-jdbc&v=3.13-SNAPSHOT"
    filename: egeria-connector-resource-jdbc.jar
  - url: http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-integration-jdbc&v=3.13-SNAPSHOT"
    filename: egeria-connector-integration-jdbc.jar
  - url: https://jdbc.postgresql.org/download/postgresql-42.5.0.jar
    filename: postgresql.jar

We can install an instance of the helm chart ie:

helm install lab egeria/odpi-egeria-lab --set jupyter.tokenPlain='s3cr3t!' --set strimzi.enabled=false --devel -f ~/etc/jdbc.yaml

This

• uses a helm instance name of 'lab'
• Uses the odpi-egeria-lab on the (configured) egeria repository
• specifies a hardcoded password for jupyter
• does not deploy the strimzi operator (as I am using multiple egeria charts, and have deployed manually)
• Uses the development version charts
• Specifies the additional override above (all the settings could be moved into that file)

After the container is started, we can see the connector is present:

➜  etc kubectl logs lab-odpi-egeria-lab-core-0 -c init-connector
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100   134  100   134    0     0    685      0 --:--:-- --:--:-- --:--:--   687
100   787    0   787    0     0   1488      0 --:--:-- --:--:-- --:--:--  3306
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100   134  100   134    0     0    853      0 --:--:-- --:--:-- --:--:--   858
100   793    0   793    0     0   1625      0 --:--:-- --:--:-- --:--:--  1625
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 1021k  100 1021k    0     0   714k      0  0:00:01  0:00:01 --:--:--  714k
Downloads complete.

➜  etc kubectl exec -it lab-odpi-egeria-lab-core-0 -- /bin/ls -la /deployments/server/extralib
Defaulted container "egeria" out of: egeria, init-connector (init)
total 1044
drwxrwsrwx. 2 root       1000840000    4096 Nov  7 15:16 .
drwxr-xr-x. 1 root       root          4096 Nov  7 15:16 ..
-rw-r--r--. 1 1000840000 1000840000     793 Nov  7 15:16 egeria-connector-integration-jdbc.jar
-rw-r--r--. 1 1000840000 1000840000     787 Nov  7 15:16 egeria-connector-resource-jdbc.jar
-rw-r--r--. 1 1000840000 1000840000 1046274 Nov  7 15:16 postgresql.jar
➜  etc

[planetf1]

Update on above - the connectors are the wrong content .The reason is that the specific URL specified utilizes a query which results in a http redirect as a response. In the charts we do not follow redirects, so the http redirect is downloaded, not the target of the redirect..

See odpi/egeria-charts#207

[planetf1]

This error was a user problem - missing quotes in the URL which then failed (incorrect end result, not a failure....) on the curl request
Will look at whether any additional error checking can be done in the helm charts

The correct jdbc.yaml is:

extralibs:
  - url: "http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-resource-jdbc&v=3.13-SNAPSHOT"
    filename: egeria-connector-resource-jdbc.jar
  - url: "http://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=org.odpi.egeria&a=egeria-connector-integration-jdbc&v=3.13-SNAPSHOT"
    filename: egeria-connector-integration-jdbc.jar
  - url: "https://jdbc.postgresql.org/download/postgresql-42.5.0.jar"
    filename: postgresql.jar

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

this was completed as part of the original work