[DRILL-7803] Docker based developer build environment

[nielsbasjes]

DRILL-7803: Docker based developer build environment

Description

I was asked to review a change in Drill ( #2112 ).
When I tried to build the software it turns out that a lot of the build fails over the fact that I have Java 11 installed on my machine by default.

To "fix" this I copied/changed the Docker based build environment I initially created for Hadoop (many years ago) and recently updated for Apache Beam ( apache/beam#13308 | BEAM-10891 )

Because I think it makes sense to have this helper more easily available I put it in this pull request.

This is just an additional tool to make setting up the build environment for Drill a lot easier.

Documentation

Only a small addition for developers who want to build the sourcecode locally.

Testing

On a "empty" (only git and docker are needed) Linux machine with Docker installed (i.e. RedHat/CentOS probably won't work) in the root of the Drill source tree run the start-build-env.sh script.
After a while (depends on machine and network speed) a bash command prompt should appear with all needed tools to build Apache Drill installed.

Testing is then simply running something like mvn clean verify to show everything works.

[cgivre]

@nielsbasjes
Thanks for the contribution and the code review. If you can fix the license headers, I'll approve this. (Drill is picky about them)
Best,
-- C

[nielsbasjes]

NOTE: Doing a full test run to ensure it is correct now.

[nielsbasjes]

Having trouble getting the JDBC test to pass that uses the EmbeddedMySQL.
The required dependencies are there but it still fails during the setup of the test.

[INFO] Running org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! - in org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT
[ERROR] org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT  Time elapsed: 0.001 s  <<< ERROR!
com.wix.mysql.exceptions.CommandFailedException: Command 'CREATE USER 'mysqlUser'@'%' IDENTIFIED BY 'mysqlPass';' on schema 'information_schema' failed with message 'Stream closed'
	at org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT.initMysql(TestJdbcPluginWithMySQLIT.java:70)
Caused by: java.io.IOException: Stream closed
	at org.apache.drill.exec.store.jdbc.TestJdbcPluginWithMySQLIT.initMysql(TestJdbcPluginWithMySQLIT.java:70)

[nielsbasjes]

At this point the entire build passes with one exception: TestJdbcPluginWithMySQLIT

[cgivre]

@nielsbasjes
I'm trying this out on my Mac. I'll report back if I get the same error.
Thanks for the contribution!

I didn't see the latest comment about the MySQLIT error.

[nielsbasjes]

Should be fixed now.
Added libncurses5 as indicated here wix-incubator/wix-embedded-mysql#116
Now this test passes when run in isolation.
Going to do a full rerun now to be sure.

Doing mvn clean install -Drat.skip=false -Dlicense.skip=false

[nielsbasjes]

Added MAVEN_OPTS settings from the Release documentation

[nielsbasjes]

It worked on my machine

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Drill Root POM 1.19.0-SNAPSHOT:
[INFO] 
[INFO] Apache Drill Root POM .............................. SUCCESS [  6.664 s]
[INFO] tools/Parent Pom ................................... SUCCESS [  0.495 s]
[INFO] tools/freemarker codegen tooling ................... SUCCESS [  3.996 s]
[INFO] Drill Protocol ..................................... SUCCESS [  5.408 s]
[INFO] Common (Logical Plan, Base expressions) ............ SUCCESS [  6.158 s]
[INFO] Logical Plan, Base expressions ..................... SUCCESS [  6.717 s]
[INFO] exec/Parent Pom .................................... SUCCESS [  1.274 s]
[INFO] exec/memory/Parent Pom ............................. SUCCESS [  0.283 s]
[INFO] exec/memory/base ................................... SUCCESS [  4.175 s]
[INFO] exec/rpc ........................................... SUCCESS [  2.831 s]
[INFO] exec/Vectors ....................................... SUCCESS [01:35 min]
[INFO] contrib/Parent Pom ................................. SUCCESS [  0.514 s]
[INFO] contrib/data/Parent Pom ............................ SUCCESS [  0.315 s]
[INFO] contrib/data/tpch-sample-data ...................... SUCCESS [  1.862 s]
[INFO] metastore/Parent Pom ............................... SUCCESS [  0.366 s]
[INFO] metastore/Drill Metastore API ...................... SUCCESS [  6.318 s]
[INFO] metastore/Drill Iceberg Metastore .................. SUCCESS [ 11.373 s]
[INFO] exec/Java Execution Engine ......................... SUCCESS [22:56 min]
[INFO] exec/JDBC Driver using dependencies ................ SUCCESS [01:12 min]
[INFO] JDBC JAR with all dependencies ..................... SUCCESS [ 22.328 s]
[INFO] Drill-on-YARN ...................................... SUCCESS [  7.593 s]
[INFO] metastore/Drill RDBMS Metastore .................... SUCCESS [  9.037 s]
[INFO] contrib/kudu-storage-plugin ........................ SUCCESS [  6.361 s]
[INFO] contrib/http-storage-plugin ........................ SUCCESS [ 13.912 s]
[INFO] contrib/opentsdb-storage-plugin .................... SUCCESS [ 12.608 s]
[INFO] contrib/mongo-storage-plugin ....................... SUCCESS [ 22.082 s]
[INFO] contrib/hbase-storage-plugin ....................... SUCCESS [01:29 min]
[INFO] contrib/jdbc-storage-plugin ........................ SUCCESS [01:00 min]
[INFO] contrib/hive-storage-plugin/Parent Pom ............. SUCCESS [  0.423 s]
[INFO] contrib/hive-storage-plugin/hive-exec-shaded ....... SUCCESS [ 19.934 s]
[INFO] contrib/hive-storage-plugin/core ................... SUCCESS [03:34 min]
[INFO] contrib/kafka-storage-plugin ....................... SUCCESS [03:53 min]
[INFO] contrib/drill-udfs ................................. SUCCESS [ 23.610 s]
[INFO] contrib/format-syslog .............................. SUCCESS [ 10.682 s]
[INFO] contrib/format-hdf5 ................................ SUCCESS [ 14.429 s]
[INFO] contrib/format-spss ................................ SUCCESS [ 11.198 s]
[INFO] contrib/ltsv-format-plugin ......................... SUCCESS [ 10.541 s]
[INFO] contrib/format-esri ................................ SUCCESS [ 11.475 s]
[INFO] contrib/format-excel ............................... SUCCESS [ 13.508 s]
[INFO] contrib/druid-storage-plugin ....................... SUCCESS [  7.319 s]
[INFO] Packaging and Distribution Assembly ................ SUCCESS [ 25.364 s]
[INFO] contrib/mapr-format-plugin ......................... SUCCESS [  6.800 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  40:49 min
[INFO] Finished at: 2020-11-15T15:00:32Z
[INFO] ------------------------------------------------------------------------

[cgivre]

LGTM +1