Slow loading in Local instance of SEQR

[Tulasi08]

Hi Team,

We have actually set up the local instance of SEQR in our machine which is 16 GB RAM, 8 CPUs, 718 Gb disk space ( root + external disk). We have uploaded 3 projects as of now containing 180, 36 and 1 samples respectively. Please find the attached screenshot of disk status from SEQR UI:

The software was working fine. But from the last week, the search results of project variant search is taking long time to load and the download option is also been disabled and also shows an error as shown in screenshot.

When I restart the docker, it works fine for 2 searches and the error shows up again.

Do you have any idea on why this is happening and what can be done to resolve this issue?

Thanks and Regards,
Tulasi S

[hanars]

Hi Tulsai,

Please provide all the logs from both the seqr container and the elasticsearch container. This is not the sort of issue that can be debugged without more detailed information.

Best,
Hana

[Tulasi08]

Thanks Hana,

Sure, Please find attached the log files for your reference. Also wanted to check with you if our machine configuration is not enough and if should increase the size and cores of our machine. We already have total of 352 samples in different projects and we are expecting more than 1000 or more samples to be added in future. There are 3 to 4 analyst working at the same time currently and are mostly doing the project variant search. If you think the configuration of our machine is problem, could you please suggest on what the configuration must be according to our usage as described above? Looking forward to hear from you!

Thanks and Regards,
Tulasi S
logs_elasticsearch.txt
logs_1.zip

[hanars]

Can you also provide a screenshot of your full elasticsearch status page? Also, can you attach the docker config file you are using?

Also wanted to check with you if our machine configuration is not enough and if should increase the size and cores of our machine

So from the bit of the status page you showed above you look fine because you are using 22% of disk and 12% of memory, which means you should have plenty available. However, I am seeing a lot of out of memory errors in your logs. Its possible that you are using so much of the memory that ES is crashing and restarting, so by the time you took this screenshot memory usage is back down to something readable. You definitely do not need more disk space, but its possible the more memory would be helpful

[Tulasi08]

Sure Hana,
Here is the full screenshot of the Eleasticsearch status page and also the docker-compose.yml file that we use. It would be great if you could give us some numbers on how much more memory we would require (your guess/suggesstion) as we don't have any idea regarding the same?

The memory we have right now is:

docker-compose.txt

Thanks and Regards,
Tulasi S

[hanars]

Hi Tulasi,

So I think a big problem you have is the number of indices per project. seqr is designed to have all the samples in a project joint called together and loaded together in one index. While you technically can do what you did here and have it work in theory, this is not the recommended way to use seqr and therefore it is likely contributing to your search issues. For you next steps, I would recommend joint calling your data together and loading one index per project, and then deleting all the old indices.

As you are not loading your ddata in the supported or recommended way, I I can not make any recommendations on what kind of memory you would need to be able to handle searching across so many indices like this.

Best
Hana

[Tulasi08]

Thank you so much for your valuable suggestions Hana.

Regards,
Tulasi S