Backport PR #16026 to 8.13: Save name came through ENV vars to let Logstash decide using value from either keystore or ENV.

[github-actions]

Backport PR #16026 to 8.13 branch, original message:

---

Release notes

What does this PR do?

This PR fixes the historical bug: when running Logstash in a docker container with ENV variables, env2yaml tool saves ENV ${KEY}=${VALUE} by resolving its actual value. This will be an issue when keystore has a same ${KEY} where the expected behavior is to be keystore precedence.

Why is it important/What is the impact to the user?

If users are using same ${KEY} in both keystore and docker ENV, they may see the behavior where keystore ${KEY}=VALUE will be used.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

• pull this PR
• run rake artifact:docker -> creates an docker image (see the logs below)
• run the container with docker run --rm --name={preferredName} -e API_ENABLED=false -e {dockerImageHashBuiltByRake}
• add API_ENABLED same key to the keystore with value of true, bin/logstash-keystore add API_ENABLED
• rerun the docker container
• check the Logstash logs that API server will be executed because Logstash uses keystore API_ENABLED=true
• got to container and check the config/logstash.yml file. Before the fix, logstash.yml would contain api.enabled: false lines. After the fix it is, api.enabled: ${API_ENABLED}

Related issues

• Closes Inconsistent order of resolving ${VAR} value when running Logstash with docker #15766

Use cases

Screenshots

Logs

Using system java: /usr/bin/java
....
[docker] Building docker image
../vendor/jruby/bin/jruby -S erb -T "-" created_date="2024-03-25T12:32:27-07:00" elastic_version="8.14.0-SNAPSHOT" arch="arm64" version_tag="8.14.0-SNAPSHOT" image_flavor="oss" local_artifacts="true" templates/Dockerfile.erb > "/Users/mashhur/Dev/elastic/logstash/build/Dockerfile-oss" ;   ../vendor/jruby/bin/jruby -S erb -T "-" created_date="2024-03-25T12:32:27-07:00" elastic_version="8.14.0-SNAPSHOT" arch="arm64" version_tag="8.14.0-SNAPSHOT" image_flavor="full" local_artifacts="true" templates/Dockerfile.erb > "/Users/mashhur/Dev/elastic/logstash/build/Dockerfile-full" ;   ../vendor/jruby/bin/jruby -S erb -T "-" created_date="2024-03-25T12:32:27-07:00" elastic_version="8.14.0-SNAPSHOT" arch="arm64" version_tag="8.14.0-SNAPSHOT" image_flavor="ubi8" local_artifacts="true" templates/Dockerfile.erb > "/Users/mashhur/Dev/elastic/logstash/build/Dockerfile-ubi8" ; 
docker run --rm \
	  -v "/Users/mashhur/Dev/elastic/logstash/docker/data/logstash/env2yaml:/usr/src/env2yaml" \
		-e GOARCH=arm64 -e GOOS=linux \
		-w /usr/src/env2yaml golang:1 go build -o /usr/src/env2yaml/env2yaml-arm64
Unable to find image 'golang:1' locally
1: Pulling from library/golang
...
Digest: sha256:0b55ab82ac2a54a6f8f85ec8b943b9e470c39e32c109b766bbc1b801f3fa8d3b
Status: Downloaded newer image for golang:1
go: downloading gopkg.in/yaml.v2 v2.4.0
docker run --rm \
	  -v "/Users/mashhur/Dev/elastic/logstash/docker/data/logstash/env2yaml:/usr/src/env2yaml" \
		-e GOARCH=amd64 -e GOOS=linux \
		-w /usr/src/env2yaml golang:1 go build -o /usr/src/env2yaml/env2yaml-amd64
go: downloading gopkg.in/yaml.v2 v2.4.0
docker run --rm -d --name=logstash-docker-artifact-server \
	           -p 8000:8000 --expose=8000 -v /Users/mashhur/Dev/elastic/logstash/build:/mnt \
	           python:3 bash -c 'cd /mnt && python3 -m http.server'
Unable to find image 'python:3' locally
3: Pulling from library/python
...
Digest: sha256:336461f63f4eb1100e178d5acbfea3d1a5b2a53dea88aa0f9b8482d4d02e981c
Status: Downloaded newer image for python:3
c0788dc1c4055d4822d64dfb189f8a82c5d81aefed259b9ddc4013e94195d183
timeout 120 bash -c 'until curl -s localhost:8000 > /dev/null; do sleep 1; done'
docker build --network=host -t docker.elastic.co/logstash/logstash-full:8.14.0-SNAPSHOT -f /Users/mashhur/Dev/elastic/logstash/build/Dockerfile-full data/logstash || \
	  (docker kill logstash-docker-artifact-server; false); \
	docker tag docker.elastic.co/logstash/logstash-full:8.14.0-SNAPSHOT docker.elastic.co/logstash/logstash:8.14.0-SNAPSHOT;
[+] Building 2.5s (19/19) FINISHED                                                                                                                                                       
 => [internal] load build definition from Dockerfile-full                                                                                                                           0.1s
 => => transferring dockerfile: 3.26kB                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                   0.1s
 => => transferring context: 2B                                                                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                     1.8s
 => [ 1/14] FROM docker.io/library/ubuntu:20.04@sha256:80ef4a44043dec4490506e6cc4289eeda2d106a70148b74b5ae91ee670e9c35d                                                             0.0s
 => [internal] load build context                                                                                                                                                   0.0s
 => => transferring context: 3.16MB                                                                                                                                                 0.0s
 => CACHED [ 2/14] RUN for iter in {1..10}; do export DEBIAN_FRONTEND=noninteractive && apt-get update -y && apt-get upgrade -y && apt-get install -y procps findutils tar gzip &&  0.0s
 => CACHED [ 3/14] RUN groupadd --gid 1000 logstash &&     adduser --uid 1000 --gid 1000     --home /usr/share/logstash --no-create-home     logstash                               0.0s
 => CACHED [ 4/14] RUN  curl -Lo - http://localhost:8000/logstash-8.14.0-SNAPSHOT-linux-$(arch).tar.gz |     tar zxf - -C /usr/share &&     mv /usr/share/logstash-8.14.0-SNAPSHOT  0.0s
...
 => [14/14] RUN chmod 0755 /usr/local/bin/docker-entrypoint                                                                                                                         0.1s
 => exporting to image                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                             0.0s
 => => writing image sha256:e89b6d0d04e262f5fa3b51378ca23982756a6b585af411b917dfd2010122806a                                                                                        0.0s
 => => naming to docker.elastic.co/logstash/logstash-full:8.14.0-SNAPSHOT                                                                                                           0.0s
docker kill logstash-docker-artifact-server
logstash-docker-artifact-server

[mashhurs]

LGTM!

[elastic-sonarqube]

Quality Gate failed

Failed conditions

2 New issues

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b43b72d