You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is really an organization-wide issue, for now keeping it in morphodict.
I've just checked out the RAM consumption of our machines, and it seems that it goes about as follows:
speech-db 4GB
itwewina.app 11.9 GB
non-itwewina sssttt.altlab.dev 4.9GB
backend+frontend refactorings 3.6GB
semantic-explorer 1.2GB
korp portions 0.2GB
This doesn't count itwewina.dev. But when we ramp up a parallel itwewina.dev version with a full dictionary, we run close to the memory limits of our server, which could eventually lead to unexpected service drops. The itwewina.dev service stopped processing requests on its own, likely because of memory constraints, and I've decided to take it down at least until a reboot of the production itwewina (at a time less likely to disrupt others), perhaps even until after CILLDI ends. But most importantly, we need to address the memory issue with urgency. Some suggestions (first one is the most straightforward):
Requesting a machine with more memory for the server from Compute Canada (say, 48GB at minimum, ideally 64GB RAM o more)
Limiting the per-docker container maximum amount of memory and the memory available for uWSGI (python server), perhaps even looking at alternatives to using uWSGI, and ensuring that such restrictions would trigger image restarts that would keep services going instead of randomly stopping processing requests. Note that memory restrictions may not be feasible if all the memory is required for computation instead of caching.
Optimize the apps' use of memory (from profiling, etc.), and document where most of the memory consumption is currently going to.
In parallel, it seems that the generation of sound recordings for paradigms can impose a considerable load on speech-db CPU usage (bursts per-request, which are not an issue currently but would scale to a problem if many people are using them). We should consider separating the service and functionality of providing recordings for dictionary purposes from the validation and recording services provided by speech-db, to avoid the former taking down the latter.
The text was updated successfully, but these errors were encountered:
This is really an organization-wide issue, for now keeping it in morphodict.
I've just checked out the RAM consumption of our machines, and it seems that it goes about as follows:
speech-db
4GBitwewina.app
11.9 GBsssttt.altlab.dev
4.9GBbackend+frontend
refactorings 3.6GBsemantic-explorer
1.2GBkorp
portions 0.2GBThis doesn't count
itwewina.dev
. But when we ramp up a parallelitwewina.dev
version with a full dictionary, we run close to the memory limits of our server, which could eventually lead to unexpected service drops. Theitwewina.dev
service stopped processing requests on its own, likely because of memory constraints, and I've decided to take it down at least until a reboot of the production itwewina (at a time less likely to disrupt others), perhaps even until after CILLDI ends. But most importantly, we need to address the memory issue with urgency. Some suggestions (first one is the most straightforward):In parallel, it seems that the generation of sound recordings for paradigms can impose a considerable load on
speech-db
CPU usage (bursts per-request, which are not an issue currently but would scale to a problem if many people are using them). We should consider separating the service and functionality of providing recordings for dictionary purposes from the validation and recording services provided by speech-db, to avoid the former taking down the latter.The text was updated successfully, but these errors were encountered: