Add ES exporter to application.yaml

[akeller]

Description

Open discussion - https://camunda.slack.com/archives/C074T8BTTM2/p1721078565460779?thread_ts=1720034668.231519&cid=C074T8BTTM2

Takes the following instructions, packaged with the release artifacts, and makes them more consumable:

How to run
==========

Prerequisites:
1. Download Elasticsearch 8.9.2 (from https://www.elastic.co/downloads/elasticsearch)
2. For non-production cases, disable Elasticsearch's security packages by setting the xpack.security.* configuration options to false in ELASTICSEARCH_HOME/config/elasticsearch.yml
3. Start Elasticsearch by running ELASTICSEARCH_HOME/bin/elasticsearch (or ELASTICSEARCH_HOME\bin\elasticsearch.bat on Windows)

Start Camunda:
1. Run bin/camunda (or bin\camunda.bat on Windows)

When should this change go live?

• This is a bug fix, security concern, or something that needs urgent release support.
• This is already available but undocumented and should be released within a week.
• This on a specific schedule and the assignee will coordinate a release with the DevEx team. (apply hold label or convert to draft PR)
• This is part of a scheduled alpha or minor. (apply alpha or minor label)
• There is no urgency with this change and can be released at any time.

PR Checklist

• My changes are for an already released minor and are in /versioned_docs directory.
• My changes are for the next minor and are in /docs directory (aka /next/).

• My changes require an Engineering review, and I've assigned the Engineering DRI or delegate.
• My changes require a technical writer review, and I've assigned @camunda/tech-writers as a reviewer.

[github-actions]

👋 🤖 🤔 Hello! Did you make your changes in all the right places?

These files were changed only in docs/. You might want to duplicate these changes in versioned_docs/version-8.5/.

• docs/guides/react-components/install-plain-java.md

You may have done this intentionally, but we wanted to point it out in case you didn't. You can read more about the versioning within our docs in our documentation guidelines.

[akeller]

For reviewers (@christinaausley @conceptualshark @pepopowitz)- I'm looking for the following, feedback-wise:

• Is it overkill to include the entire application.yaml because of spacing/nesting purposes? Do you have a suggestion for how to make these easy to explain & maintain if the monorepo (source of truth) doesn't include the exporter by default?
  • If this is too challenging to explain & maintain, I'd like to push back on the idea that this should be a default and users who want to run Zeebe by itself should remove the exporter.
• Does breaking up Step 1 (installation step) by technology make sense? ES needs to be running for Camunda to start successfully anyway.
  • Implementing Step 1 as a reusable component seems to cause the headers to not show up in the right-hand sidebar, but the header links appear to work... was this just weird on my computer, or is this the case for you too? Will this be confusing?

Please set aside some time to run this on your local machine or work together to see how it runs. Let's set a deadline to do this by next Friday (July 26). This is a high priority (part of the getting started experience), but not urgent. If the urgency increases, I'll move forward as-is and open a new PR based on your feedback.

[pepopowitz]

Most of my feedback is non-blocking, but one in particular (marked "blocking") is preventing me from moving forward to test this out. I am not clear on which specific settings I need to change for elasticsearch.

[conceptualshark]

Running alpha2 seemed to result in a version mismatch for me, but running alpha3 went fine:

io.camunda.operate.schema.SchemaStartup - Schema startup failed: Error(s) in index schema: Newer version(s) for operation (8.4.0) already exists: [8.4.1]

[akeller]

@romansmirnov thoughts on this?

[romansmirnov]

Assumption: The same ES instance has been used, but first, alpha3 was started, creating an index in version 8.4.1, and then alpha2 was started, expecting the index to be in version 8.4.0.

Can you please confirm my understanding? If so, I don't recommend using the same ES instance and switching from a newer C8 version to an older C8 version.

[akeller]

@conceptualshark I don't know if we should expect people using the same ES instance (purposefully or accidentally). Can I give you a follow up here to maybe add a small troubleshooting section or think more about this experience for new users?

[conceptualshark]

The only thing I can think of that would have impacted this is having started C8SM Run on this machine (it was not running at the time). When I tested this PR, I ran ES + alpha2, failed with the above error, stopped both, then reinstalled and ran ES + alpha3, so I am not sure why an index (or even ES instance) would have been maintained.

Unfortunately, I can't add anything for resolving it, because it still doesn't work on my machine (I had to run it in a VM).

[pepopowitz]

I took a second pass. As I was finishing, I noticed in Slack that I may have been asked to use alpha3, but I went through with alpha2. Oops? This might be a worthless review.

In the end, I can see a login screen for operate at localhost:8080 (though I don't know what credentials to log in with). I see localhost:8080/tasklist trying to serve something, but it's basically a blank gray page. I am wondering if it would serve an actual tasklist if I had logged into localhost:8080.

[conceptualshark]

I ran through this whole process in UTM to isolate it from the build errors I was having on my machine and: success!

I changed the ES security config xpack.security.enabled: false prior to running ES for the first time, and it worked fine. I also no longer got schema errors running alpha2, so can confirm that worked on a fresh/isolated install as well.

My assumption is testing the C8Run script caused some versioning/config issues there, which might just be good to have in mind if we end up with folks attempting multiple methods of installing/running.

[christinaausley]

Is it overkill to include the entire application.yaml because of spacing/nesting purposes?

I don't love this, because users will only scroll so far down a page in general, and it can make the page a bit messy. Are we able to link to it, or include it in something like a <details> element or collapsible/expandable code snippet? I've seen this a few times across our docs.

If not, is there a specific reason why users aren't able to find these details in elements like these already?

Fun fact, I guess you can use <details> elements in GitHub?? I only know because I just typed it and got the following 😄 :

Some details here.

[akeller]

Hi All - thanks for the reviews and feedback. I will do one more pass today to close this out.

Related, we will swap in C8SM Run to be our single option for getting started with SM. If you have more input on that part of the conversation, please go to #e2e-docs-experience-getting-started-guides. Regardless, it's important if we do ever need to have both of these, we can easily pull these changes back in and offer parallel paths.

[conceptualshark]

I think this is in good shape! If we're eager to merge, my suggestion isn't blocking.

[conceptualshark]

🎉

Feedback addressed