Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build against aws-sdk-core. #267

Merged
merged 1 commit into from
Dec 13, 2024
Merged

Build against aws-sdk-core. #267

merged 1 commit into from
Dec 13, 2024
+5 −4

Conversation

pecastro
Copy link
Contributor

Description

Builds opensearch plugin that can be deployed against the more recent versions of logstash.

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has documentation added
  • Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Solves this:

root@625ca0aeeae7:/usr/share/logstash# DEBUG=1 bin/logstash-plugin install file:///usr/share/logstash/logstash-integration-aws-0.1.0.pre.gem.zip
Using bundled JDK: /usr/share/logstash/jdk
Processing jvm.options file at `/usr/share/logstash/config/jvm.options`
Appending jvm options from environment LS_JAVA_OPTS
Using GEM_HOME=/usr/share/logstash/vendor/bundle/jruby/3.1.0
Using GEM_PATH=/usr/share/logstash/vendor/bundle/jruby/3.1.0
DEBUG: exec /usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/lib/pluginmanager/main.rb install file:///usr/share/logstash/logstash-integration-aws-0.1.0.pre.gem.zip
Local file: /usr/share/logstash/logstash-integration-aws-0.1.0.pre.gem.zip
Installing with strategy: LogStash::PluginManager::PackInstaller::Local
Installing file: /usr/share/logstash/logstash-integration-aws-0.1.0.pre.gem.zip
Pack uncompressed to /tmp/studtmp-e7f0ee38a2427ecc981c9fecfe62f873854f97aa0c9df5916e21076d5fa3
Installing, logstash-integration-aws, version: 0.1.0.pre file: /tmp/studtmp-e7f0ee38a2427ecc981c9fecfe62f873854f97aa0c9df5916e21076d5fa3/logstash-integration-aws-0.1.0.pre.gem
Found changes from the lockfile, re-resolving dependencies because the dependencies in your gemfile changed, you added a new platform to your gemfile
Resolving dependencies.....................................
Install successful
root@625ca0aeeae7:/usr/share/logstash# DEBUG=1 bin/logstash-plugin install file:///usr/share/logstash/logstash-output-opensearch-2.0.3-java.gem.zip
Using bundled JDK: /usr/share/logstash/jdk
Processing jvm.options file at `/usr/share/logstash/config/jvm.options`
Appending jvm options from environment LS_JAVA_OPTS
Using GEM_HOME=/usr/share/logstash/vendor/bundle/jruby/3.1.0
Using GEM_PATH=/usr/share/logstash/vendor/bundle/jruby/3.1.0
DEBUG: exec /usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/lib/pluginmanager/main.rb install file:///usr/share/logstash/logstash-output-opensearch-2.0.3-java.gem.zip
Local file: /usr/share/logstash/logstash-output-opensearch-2.0.3-java.gem.zip
Installing with strategy: LogStash::PluginManager::PackInstaller::Local
Installing file: /usr/share/logstash/logstash-output-opensearch-2.0.3-java.gem.zip
Pack uncompressed to /tmp/studtmp-85e6cce49abf76bf6b4104545d318070d64e86c8fbeb3e51d8c93029ab7a
Installing, logstash-output-opensearch, version: 2.0.3 file: /tmp/studtmp-85e6cce49abf76bf6b4104545d318070d64e86c8fbeb3e51d8c93029ab7a/dist/logstash-output-opensearch-2.0.3-java.gem
Found changes from the lockfile, re-resolving dependencies because the dependencies in your gemfile changed, you added a new platform to your gemfile
Resolving dependencies......................................................................................
Bundler::VersionConflict: Could not find gem 'aws-sdk (~> 3)', which is required by gem 'logstash-output-opensearch (= 2.0.3)', in any of the sources.
            start at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/resolver.rb:53
          resolve at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/definition.rb:279
      add_sources at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:38
             each at org/jruby/RubyArray.java:1981
  each_with_index at org/jruby/RubyEnumerable.java:1214
      add_sources at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:31
        generate! at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:19
         generate at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:15
          to_lock at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/definition.rb:344
             lock at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/definition.rb:294
           inject at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:99
        temporary at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/settings.rb:131
           inject at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:76
          inject! at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:59
            chdir at org/jruby/RubyDir.java:472
          inject! at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:58
          execute at /usr/share/logstash/lib/pluginmanager/pack_installer/local.rb:57
          execute at /usr/share/logstash/lib/pluginmanager/install.rb:48
              run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
          execute at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11
              run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
              run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:133
           <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64
Bundler::Molinillo::VersionConflict: Unable to satisfy the following requirements:

- `aws-sdk (~> 3)` required by `logstash-output-opensearch (2.0.3) (java)`
  raise_error_unless_state at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/vendor/molinillo/lib/molinillo/resolution.rb:317
       unwind_for_conflict at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/vendor/molinillo/lib/molinillo/resolution.rb:299
                       tap at <internal:uri:classloader:/jruby/kernel/kernel.rb>:19
       unwind_for_conflict at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/vendor/molinillo/lib/molinillo/resolution.rb:297
     process_topmost_state at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/vendor/molinillo/lib/molinillo/resolution.rb:257
                   resolve at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/vendor/molinillo/lib/molinillo/resolution.rb:182
                   resolve at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/vendor/molinillo/lib/molinillo/resolver.rb:43
                     start at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/resolver.rb:32
                   resolve at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/definition.rb:279
               add_sources at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:38
                      each at org/jruby/RubyArray.java:1981
           each_with_index at org/jruby/RubyEnumerable.java:1214
               add_sources at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:31
                 generate! at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:19
                  generate at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/lockfile_generator.rb:15
                   to_lock at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/definition.rb:344
                      lock at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/definition.rb:294
                    inject at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:99
                 temporary at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/bundler/settings.rb:131
                    inject at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:76
                   inject! at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:59
                     chdir at org/jruby/RubyDir.java:472
                   inject! at /usr/share/logstash/lib/pluginmanager/bundler/logstash_injector.rb:58
                   execute at /usr/share/logstash/lib/pluginmanager/pack_installer/local.rb:57
                   execute at /usr/share/logstash/lib/pluginmanager/install.rb:48
                       run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                   execute at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11
                       run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                       run at /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/clamp-1.0.1/lib/clamp/command.rb:133
                    <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64

dblock
dblock requested changes Dec 10, 2024
View reviewed changes
Copy link
Member

@dblock dblock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also update release-notes/logstash-output-opensearch-release-notes-next.md please?

(I think we should move release notes into a CHANGELOG like other projects in this org and add a workflow that checks that the file is updated in every PR, but that's beyond the scope of this.)

logstash-output-opensearch.gemspec Outdated
@@ -11,7 +11,7 @@ signing_key_path = "gem-private_key.pem"

Gem::Specification.new do |s|
s.name = 'logstash-output-opensearch'
s.version = '2.0.3'
s.version = '2.0.4.pre'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make this 2.1.0 since we're changing dependencies.

@pecastro pecastro force-pushed the build-with-aws-sdk-core branch from 9414c40 to 9be76ab Compare December 11, 2024 17:43
dblock
dblock reviewed Dec 11, 2024
View reviewed changes
release-notes/logstash-output-opensearch-release-notes-next.md Outdated

Compatible with OpenSearch 1.3+, 2.0+

### Bug Fixes

* Fix warning retrieving cluster UUID with Amazon OpenSearch Serverless (#237)
* Install against most recent versions of logstash by building with [aws-sdk-core](https://rubygems.org/gems/aws-sdk-core).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include the PR number in your new line as well.

@pecastro pecastro force-pushed the build-with-aws-sdk-core branch from 9be76ab to 5426e45 Compare December 12, 2024 11:48
@dblock
Copy link
Member

dblock commented Dec 12, 2024

Thanks, check the failing test above @pecastro? Thanks for hanging in here with me.

@pecastro
Build plugin against aws-sdk-core.
90304e8 
Signed-off-by: Paulo E. Castro <[email protected]>
Signed-off-by: Paulo E. Castro <[email protected]>
@pecastro pecastro force-pushed the build-with-aws-sdk-core branch from 5426e45 to 90304e8 Compare December 13, 2024 11:20
@pecastro
Copy link
Contributor Author

It WOOM.
Comms issue?

pecastro@localhost /download/logstash-output-opensearch[build-with-aws-sdk-core]$ git rev-parse HEAD
90304e8042a05e72c1fbda710f150b8b59bea59e
pecastro@localhost /download/logstash-output-opensearch[build-with-aws-sdk-core]$ export LOGSTASH_VERSION=7.16.3
pecastro@localhost /download/logstash-output-opensearch[build-with-aws-sdk-core]$ time ./scripts/unit-test/docker-setup.sh
WARN[0000] /download/logstash-output-opensearch/scripts/unit-test/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
WARN[0000] /download/logstash-output-opensearch/scripts/unit-test/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Building 6.9s (15/15) FINISHED                                                                                                                                                                docker:default
 => [logstash internal] load build definition from Dockerfile                                                                                                                                               0.1s
 => => transferring dockerfile: 690B                                                                                                                                                                        0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG docker.elastic.co/logstash/logstash-oss:${LOGSTASH_VERSION} results in empty or invalid base image name (line 2)                                   0.1s
 => [logstash internal] load metadata for docker.elastic.co/logstash/logstash-oss:7.16.3                                                                                                                    1.1s
 => [logstash internal] load .dockerignore                                                                                                                                                                  0.1s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [logstash 1/9] FROM docker.elastic.co/logstash/logstash-oss:7.16.3@sha256:dcba93055cf40b0d81239a41b4eb568e2e79411382d7219c061cb7082216420c                                                              0.0s
 => [logstash internal] load build context                                                                                                                                                                  0.1s
 => => transferring context: 26.76kB                                                                                                                                                                        0.0s
 => CACHED [logstash 2/9] COPY --chown=logstash:logstash Gemfile /usr/share/plugins/plugin/Gemfile                                                                                                          0.0s
 => CACHED [logstash 3/9] COPY --chown=logstash:logstash *.gemspec VERSION* version* /usr/share/plugins/plugin/                                                                                             0.0s
 => CACHED [logstash 4/9] RUN cp /usr/share/logstash/logstash-core/versions-gem-copy.yml /usr/share/logstash/versions.yml                                                                                   0.0s
 => CACHED [logstash 5/9] RUN gem install bundler -v '< 2'                                                                                                                                                  0.0s
 => CACHED [logstash 6/9] WORKDIR /usr/share/plugins/plugin                                                                                                                                                 0.0s
 => CACHED [logstash 7/9] RUN bundle install --with test ci                                                                                                                                                 0.0s
 => [logstash 8/9] COPY --chown=logstash:logstash . /usr/share/plugins/plugin                                                                                                                               0.2s
 => [logstash 9/9] RUN bundle exec rake vendor                                                                                                                                                              4.4s
 => [logstash] exporting to image                                                                                                                                                                           0.3s 
 => => exporting layers                                                                                                                                                                                     0.3s 
 => => writing image sha256:6bda115d0533add2fbc93d16df8ff09979ae2d1ef1a099df44ba002b2e2f9b64                                                                                                                0.0s 
 => => naming to docker.io/library/unit-test-logstash                                                                                                                                                       0.0s
 => [logstash] resolving provenance for metadata file                                                                                                                                                       0.0s

real    0m7.206s

@dblock
Copy link
Member

dblock commented Dec 13, 2024 •
edited
Loading

I think retrying hits different (real) failures. They are possibly unrelated to this change, but unfortunately we have to fix them :(

 > [logstash 7/9] RUN bundle install --with test ci:
10.67 Could not find gem 'nio4r (~> 2.0)', which is required by gem 'puma (~> 5, >=
10.67 5.6.2)', in any of the sources.
10.67 
10.67 Bundler could not find compatible versions for gem "rspec":
10.67   In Gemfile:
10.67     logstash-devutils java was resolved to 0.0.13, which depends on
10.67       rspec (~> 2.14.0) java
10.67 
10.67 Could not find gem 'rspec (~> 2.14.0)', which is required by gem
10.67 'logstash-devutils', in any of the sources.

@pecastro
Copy link
Contributor Author

That's another one I've just ran successfully on my local machine 🤷‍♂️

[pecastro@stealthix logstash-output-opensearch]$ scripts/opensearch/docker-setup.sh
WARN[0000] /download/logstash-output-opensearch/scripts/opensearch/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
WARN[0000] /download/logstash-output-opensearch/scripts/opensearch/docker-compose.override.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 3/3
 ✔ Container opensearch-logstash-1     Removed                                                                                                                                                              0.0s 
 ✔ Container opensearch-integration-1  Removed                                                                                                                                                              0.0s 
 ✔ Network opensearch_default          Removed                                                                                                                                                              0.5s 
WARN[0000] /download/logstash-output-opensearch/scripts/opensearch/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
WARN[0000] /download/logstash-output-opensearch/scripts/opensearch/docker-compose.override.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Building 2.3s (22/22) FINISHED                                                                                                                                                                docker:default
 => [integration internal] load build definition from Dockerfile.opensearch                                                                                                                                 0.1s
 => => transferring dockerfile: 374B                                                                                                                                                                        0.0s
 => [logstash internal] load build definition from Dockerfile                                                                                                                                               0.2s
 => => transferring dockerfile: 690B                                                                                                                                                                        0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG docker.elastic.co/logstash/logstash-oss:${LOGSTASH_VERSION} results in empty or invalid base image name (line 2)                                   0.2s
 => [integration internal] load metadata for docker.io/opensearchproject/opensearch:1.3.4                                                                                                                   0.7s
 => [logstash internal] load metadata for docker.elastic.co/logstash/logstash-oss:8.3.2                                                                                                                     1.3s
 => [integration internal] load .dockerignore                                                                                                                                                               0.1s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [integration 1/2] FROM docker.io/opensearchproject/opensearch:1.3.4@sha256:60f811cd4c71deb76db4415f2c18711a786d93308ee0fc7a176c6e3e1ff3d842                                                             0.0s
 => CACHED [integration 2/2] RUN if [ "true" != "true" ] ; then /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security; fi                                                                  0.0s
 => [integration] exporting to image                                                                                                                                                                        0.1s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => writing image sha256:08e6ed04076382f3da1a320ed5af21f9834fd24b5a97f13816ed07423de322aa                                                                                                                0.0s
 => => naming to docker.io/library/opensearch-integration                                                                                                                                                   0.0s
 => [logstash internal] load .dockerignore                                                                                                                                                                  0.2s
 => => transferring context: 2B                                                                                                                                                                             0.0s
 => [integration] resolving provenance for metadata file                                                                                                                                                    0.0s
 => [logstash 1/9] FROM docker.elastic.co/logstash/logstash-oss:8.3.2@sha256:003ebd400bf2bab97c8d562094cfe94dabd10c392cb2aa40403e93df7e35040d                                                               0.0s
 => [logstash internal] load build context                                                                                                                                                                  0.1s
 => => transferring context: 10.78kB                                                                                                                                                                        0.0s
 => CACHED [logstash 2/9] COPY --chown=logstash:logstash Gemfile /usr/share/plugins/plugin/Gemfile                                                                                                          0.0s
 => CACHED [logstash 3/9] COPY --chown=logstash:logstash *.gemspec VERSION* version* /usr/share/plugins/plugin/                                                                                             0.0s
 => CACHED [logstash 4/9] RUN cp /usr/share/logstash/logstash-core/versions-gem-copy.yml /usr/share/logstash/versions.yml                                                                                   0.0s
 => CACHED [logstash 5/9] RUN gem install bundler -v '< 2'                                                                                                                                                  0.0s
 => CACHED [logstash 6/9] WORKDIR /usr/share/plugins/plugin                                                                                                                                                 0.0s
 => CACHED [logstash 7/9] RUN bundle install --with test ci                                                                                                                                                 0.0s
 => CACHED [logstash 8/9] COPY --chown=logstash:logstash . /usr/share/plugins/plugin                                                                                                                        0.0s
 => CACHED [logstash 9/9] RUN bundle exec rake vendor                                                                                                                                                       0.0s
 => [logstash] exporting to image                                                                                                                                                                           0.0s
 => => exporting layers                                                                                                                                                                                     0.0s
 => => writing image sha256:0e751cd71305fb116bac7d4af6c48ddfc965beed5111fa704d5ca5cf940cf3b9                                                                                                                0.0s
 => => naming to docker.io/library/opensearch-logstash                                                                                                                                                      0.0s
 => [logstash] resolving provenance for metadata file

@dblock
Copy link
Member

dblock commented Dec 13, 2024

I retried it and it ran. YOLO.

@dblock dblock merged commit 856763c into opensearch-project:main Dec 13, 2024
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dblock dblock dblock approved these changes

@dlvenable dlvenable Awaiting requested review from dlvenable dlvenable is a code owner

@oeyh oeyh Awaiting requested review from oeyh oeyh is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pecastro @dblock