Changes required to install Elasticsearch 6 on staging

[AmitPhulera]

https://dimagi.atlassian.net/browse/SAAS-16062

Changes required to install Elasticsearch 6 on staging

Environments Affected

Staging

[zwets]

If I may chime in on this change, I would suggest also fixing this:

diff --git a/src/commcare_cloud/ansible/roles/elasticsearch/templates/config/jvm.options.j2 b/src/commcare_cloud/ansible/roles/elasticsearch/templates/config/jvm.options.j2
index 36761736b..50ba49a01 100644
--- a/src/commcare_cloud/ansible/roles/elasticsearch/templates/config/jvm.options.j2
+++ b/src/commcare_cloud/ansible/roles/elasticsearch/templates/config/jvm.options.j2
@@ -54,7 +54,7 @@
 14-:-XX:InitiatingHeapOccupancyPercent=30
 
 ## JVM temporary directory
--Djava.io.tmpdir=${ES_TMPDIR}
+-Djava.io.tmpdir={{ elasticsearch_data_dir }}/tmp
 
 ## heap dumps
 
@@ -67,7 +67,7 @@
 -XX:HeapDumpPath={{ elasticsearch_data_dir }}/logs/heapdump.hprof
 
 # specify an alternative path for JVM fatal error logs
--XX:ErrorFile={{ elasticsearch_data_dir }}logs/hs_err_pid%p.log
+-XX:ErrorFile={{ elasticsearch_data_dir }}/logs/hs_err_pid%p.log
 
 ## JDK 8 GC logging
 8:-XX:+PrintGCDetails

Regarding the fixes: jvm.options is a config file for the JVM, not a shell script, so -Djava.io.tmpdir=${ES_TMPDIR} results in a directory that is literally called /opt/elasticsearch-5.6.16/${ES_DIR} 😆

I did not submit my local fix) of simply commenting the line out, because on many systems the default java.io.tmpdir (/tmp, under which ES creates a unique directory jna--{number}) is mounted noexec, whereas apparently ES wants to execute scripts from it.

Clearly then this is an ES bug: rather than make deployers redirect the JVM tmpdir, they should do their JNA execution in some directory of their own, for instance simply tmp under their data directory. My fix above does exactly that.

Regardless of whether you stick to the directory called $ES_TMPDIR or call it tmp as I suggest, if it is not under /tmp you'll probably need to take care of its clean-up. Not sure if that's currently taken care of?

The second change in my diff above seemed obvious to me by just looking at it.

[AmitPhulera]

@zwets Thank you for your suggestions. They were definitely an oversight from our end. I have incorporated them in eb11aa5.

[gherceg]

One small comment but otherwise looks good

[gherceg]

Just going off of this, these settings seemed to also apply to ES5 and therefore aren't strictly an ES2 setting? Are they still safe to remove?

[AmitPhulera]

Yes this is safe to remove and is intentionally removed. This was there to in ES 5 to support if someone was using groovy scripts which are deprecated.
If someone installs commcare now they will not need es5 with groovy scripts so it is safe to remove for ES5 as well.