LD_LIBRARY_PATH override stomps EMR settings

[CloudNiner]

Leading to the following exception in any job where Spark attempts to compress data:

ERROR GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)
    at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2306)

This was verified by running the following:

[hadoop@ip-172-31-7-166 ~]$ spark-shell --master local --driver-java-options -Djava.library.path=/usr/local/lib
...
scala> System.getProperty("java.library.path")
res0: String = /usr/local/lib

versus

[hadoop@ip-172-31-7-166 ~]$ spark-shell --master local
...
scala> System.getProperty("java.library.path")
res0: String = /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

We override this setting so that executors are able to find the GDAL bindings installed by the bootstrap script. I'd expect to have the spark cluster configured such that all of the native EMR installed libs as well as GDAL are available on the LD_LIBRARY_PATH

[pomadchin]

Things to keep in mind:

1. override LD_LIBRARY_PATH is enough, it was well tested with GDAL ingests in a VLM performance repo

2. Probably this will work (needs to be checked):

EmrConfig("spark-env").withProperties(
  "LD_LIBRARY_PATH" -> "/usr/local/lib:$LD_LIBRARY_PATH"
)