Rendered image appears as a single dot for very large networks

[nilsoberg]

Sometimes nodes in very large clusters are stacked on top of each other after layout, when using the non-OpenCL version of prefuse force directed layout. The OpenCL version works well. This occurs in Cytoscape 3.8 and 3.10 (snapshot 0714).

This occurs in Cytoscape as well as through the CyREST API. The output in the terminal shows there is a stack overflow:

15:24:24.742 WARN  [qtp1898635594-610] Network larger than view threshold
Network factory creation finished in 171048 msec.
java.lang.StackOverflowError
		at prefuse.util.force.NBodyForce.calcMass(NBodyForce.java:289)
		at prefuse.util.force.NBodyForce.calcMass(NBodyForce.java:289)
		at prefuse.util.force.NBodyForce.calcMass(NBodyForce.java:289)
....
		at prefuse.util.force.NBodyForce.forceHelper(NBodyForce.java:324)
		at prefuse.util.force.NBodyForce.forceHelper(NBodyForce.java:349)
		at prefuse.util.force.NBodyForce.forceHelper(NBodyForce.java:349)
....

An example of this type of file is here:

https://efi.igb.illinois.edu/databases/very_large_ssn.zip

370 MB zip file, 4.6 GB uncompressed.

First referenced in #81.

[bdemchak]

Thank you for this. I have loaded this file onto my test machine (10GB RAM, multiple core Xeon), and I see that Cytoscape hangs because of low JavaVM memory.

I'd like to reproduce this. How much RAM would I need?

Your error report is sufficiently detailed that I can guess that you're running into a difference between the CL and non-CL prefuse algorithms where the CL prefuse uses memory more efficiently.

If that's the case, I would post this to the Cytoscape support group as a bug, and I would also wonder whether adding RAM would improve the non-CL execution. Are you able to try this?

FYI, here is the error I see when loading your XGMLL file:

java.io.IOException: Could not parse XGMML file.
at org.cytoscape.io.internal.read.xgmml.GenericXGMMLReader.run(GenericXGMMLReader.java:166) ~[?:?]
at org.cytoscape.work.internal.task.JDialogTaskManager$TaskRunnable.innerRun(JDialogTaskManager.java:337) ~[!/:?]
at org.cytoscape.work.internal.task.JDialogTaskManager$TaskRunnable.run(JDialogTaskManager.java:352) [!/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.RuntimeException: Out of memory error caught. The network being loaded is too large for the current memory allocation. Use the -Xmx flag for the java virtual machine to increase the amount of memory available, e.g. java -Xmx1G cytoscape.jar -p apps ....
at org.cytoscape.io.internal.read.xgmml.GenericXGMMLReader.readXGMML(GenericXGMMLReader.java:251) ~[?:?]
at org.cytoscape.io.internal.read.xgmml.GenericXGMMLReader.run(GenericXGMMLReader.java:163) ~[?:?]
... 7 more

[nilsoberg]

I'm running these with large amounts of RAM and that avoids the parsing issue. That particular network required 170 GB to load; the JAVA_OPTS were set to "-Xms350G -Xmx350G". This is running on a system with the Xeon E5-2680 CPUs (10 years old) and with Cytoscape 3.10 took 1.5 hours to run. This happens on the desktop version as well, so it's likely not in py4cytoscape. Could I ask which project I should submit this bug report to?

[bdemchak]

Hi, Nils --

This is a very impressive RAM requirement ... I don't think my laptop will deal with it, even if I max it out. :)

Feel free to submit here: https://cytoscape.org/bug-report.html

I can work on it with you if you can create an example that shows the problem in a 10GB RAM. My machine is 32GB, but I'd be willing to max it at 64GB if that would help you along.

[nilsoberg2]

I submitted a bug report. I will see if I can get something smaller to replicate. No guarantees due to time constraints -- working on automating the generation of 150,000+ images using Cytoscape...