Run the pipeline on a small / medium / large genome

[davidonlaptop]

With updated versions of the BDGenomics pipeline (snap, adam, avocado), use a small / medium / large genome to validate the new images and the orchestration scripts. We'll compare these results with the results from S. Bonami and the BDGenomics papers.

References

TODO: Find the data used in the Snap / Adam / Avocado papers.

[flangelier]

Avocado is always failing at the moment...

[davidonlaptop]

what's the error message?

[davidonlaptop]

(let's leave comments here, for future tracking)

François Langelier 2:35 PM (22 minutes ago)
to David
2015-03-22 17:29:25 ERROR Executor:96 - Exception in task 0.0 in stage 1.0 (TID 16)
java.lang.OutOfMemoryError: GC overhead limit exceeded

[davidonlaptop]

Potential solution: https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded

[codingtony]

If you can run it with jvisualvm attached you will have more idea of what
is using the memory and what is the usage of the heap in general.

What JVM parameters you are currently using ?

-tony

On Mon, Mar 23, 2015 at 3:13 PM, David Lauzon [email protected]
wrote:

Potential solution:
https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded

—
Reply to this email directly or view it on GitHub
#11 (comment).

[davidonlaptop]

Currently, it is the default settings. I think there is no enough memory allocated to the spark workers.

Francois will is describing the steps-by-steps to reproduce the problem.

[sebastienbonami]

@flangelier I think I encountered the same problem as you with Avocado! Change the values of the two lines below in the file bin/avocado-submit. 4g is probably more than what you have available in memory and the JVM can't start.

--conf spark.executor.memory=${AVOCADO_EXECUTOR_MEMORY:-4g} \
--driver-memory ${AVOCADO_DRIVER_MEMORY:-4g} \

See: https://github.com/bigdatagenomics/avocado/blob/master/bin/avocado-submit#L56-58