platform stability

[sanderegg]

reduce issues with the platform on master/staging/production. Includes cases referring:

• deploys malfunctions (when something goes wrong in the deployed environs)
• design upgrades of key component aiming greater stability
• monitoring & diagnostics tooling (configuring or implementing tools to detect issues or help investigating them)
• maintenance services: ops services to maintain running framework stable. E.g. cron jobs to clean hosts/databases/storage; auto-restarting unhealthy services; etc
• testing services/tooling: to ensure stability during development e.g. new e2e, mock fixtures, ...

[sanderegg]

Update on Dim Sum sprint for review

Current status

• improved logging from oSparc micro-services to report errors in async tasks, reduce noise
• enhanced sidecar refactored with async libraries to overcome stability issues and responsiveness to rabbitMQ heartbeat
• new monitoring and diagnostic modules that measure's service health (currently only in webserver service)
  • set-up diagnose tool in webserver to detect async exception, slow tasks and better analysis of memory consumption - link to prometheus/grafana
  • auto-restart webserver service when handling of tasks slows down too much
• increased test coverage
  • added unit/integration testing of sidecar
  • new pytest-plugin reusable by any service aims to increase tests coverage
  • improved e2e testing (parallel / series streams)
• new python docker image base: moved from alpine to debian-slim to gain stability and build speed (based on published study).

Ongoing development

• add separate traefik.io instance to handle internal reverse-proxying (remove task from webserver) => reduces traffic load in webserver
• increase unit/integration testing coverage (using the new pytest plugin)
• fix code coverage metric
• use new monitoring to explore the cause of slowdown (still not fully resolved)
• test webserver scaling
• extend python docker-image base to webserver

[sanderegg]

Update on mid-sprint Zhongzi

Current status

Separated internal reverse proxy from webserver microservice:

• now a separate instance of traefik takes care of this task Is1304/add internal traefik instance for reverse proxy #1313 .
• Modifications on deployment architecture #59
• Modifications on 3D-viewer service #36.

Ongoing development

• improve garbage collector: remove anonymous users storage space/database entries garbage collection of anonymous storage space #1415
• improve garbage collector: clean up services around when webserver crashes when the webserver crashes, the garbage collection of services/project is not done #1307
• fix code coverage metric
• use new monitoring to explore the cause of slowdown (still not fully resolved)
• test webserver scaling
  extend python docker-image base to webserver

[sanderegg]

Update on sprint Chrigel Maurer

Current status

• improved garbage collector: Cleanup of orphaned services #1509
• zero downtime deployment: set docker swarm parameters to follow "zero downtime migration" article #1451
• active logout removes services after faster timeout: Platform stability enhancements #1527
• fixes in CC services to improve stability: [bug] Is479/cc stability #1500
• fixes in storage when Blackfynn is down: storage fails when blackfynn API is down #1506
• director nenew cache in background: director: renew cache in the background instead of clearing cache #1496
• libraries upgrade
• synchronisation of all deploys ops service versions (master, staging/production)
• new deployment of staging platform
• new release v0.0.21
• improvements on grafana monitoring dashboards (hard-disk left + Metrics)

Ongoing development

Open points

• new sidecar (for dynamic services, improve platform stability when 3rd party service is bad, security, access rights, simpler integration of services)
• test webserver scaling
• fix code coverage metric (unit testing)
• use new monitoring to explore the cause of slowdown (still not fully resolved)

[sanderegg]

Update on sprint Huo Guo

Current status

• improvements on synchronisation mechanism to deploy 3rd party services to each platform (speed, reliability) #37
• cookiecutter enhancements (fix some code-generation issues, added docker base images) #33
• updated osparc-ops repository with latest changes from master deployment #69
• updated osparc-ops for osparc api new endpoint (now deployed on master) Set public API entrypoints and set traffik to route it to api-gateway service #1474
• fix code coverage metric generation maintenance fix codecov reports #1568

Ongoing development

• new sidecar (for dynamic services, improve platform stability when 3rd party service is bad, security, access rights, simpler integration of services) Change how dynamic services run - dy-sidecar #244

Open points

• test webserver scaling
• use new monitoring to explore the cause of slowdown (still not fully resolved)
• filter e2e testing from metrics Filter metrics from e2e testing #1561
• metrics monitoring issue (data size to generate some metrics causes issue) #1599

[GitHK]

Update on sprint Da Jia

Current status

• fixed issue with volume mounting in director bugfix: setup of mounts #1740
• fixed performance issue in sidecar docker volume mounts Bugfix/comp services see all inputs #1734
• new migration service for automatic update of the postgreSQL database New Migration service #1571 + Is1571/migration service #1714
• fixes with sidecar for gpu-enabled services Fixing issues with nvidia-smi containers having the same name #1727
• additional e2e testing with jupyterNotebook+jupyterLab e2e: JupyterNB + JupyterLab #1689 + Extend jupyters e2e #1693
• fixed sidecar async code error handling FIX: catching exceptions does not work outside asyncio layer #1694
• fixed sidecar unit test which failed randomly FIX: catching exceptions does not work outside asyncio layer #1694
• trapped two previously unhandled errors which caused computational services to be frozen Fixes some error with the services #1692
• computational services no longer result frozen when external resources are unavailable Fixes some error with the services #1692
• sidecar will report more rich errors in the UI logs for better issue reporting and faster debugging Making services more debuggable #1690
• sidecar retries on pull errors, avoids connectivity issues with the registry Sidecar retries pull on error #1688
• it is now possible to totally delete users by setting their role to GUEST Garbage Collector removes manually marked GUEST users #1683
• storage service speed improved by an order of magnitude Improving storage performance #1659
• director service only lists services from its stack Fix director service discovery #1538

Ongoing development

• fixing issue with deleting users marked as GUEST garbage collector system overhaul and extras #1724
• CC tutorial fails in master CC fails in master: sync outputs issue #1479
• fix issue with sleepers tutorial failing in master-deploy sleepers tutorial failing in master-deploy #1752
• garbage collect data generated by computational services Garbage collect sidecars shared folder #1747
• new sidecar (for dynamic services, improve platform stability when 3rd party service is bad, security, access rights, simpler integration of services) Change how dynamic services run - dy-sidecar #244

Open points

• long delays fetching s3 data when runs increased Long delay fetching s3 data when runs are increased #1638
• test webserver scaling Webserver scaling #1522
• use new monitoring to explore the cause of slowdown (still not fully resolved)
• filter e2e testing from metrics Filter metrics from e2e testing #1561
• metrics monitoring issue (data size to generate some metrics causes issue) #1599

[pcrespov]

Update on sprint Nephelai

A snapshot of the stability cases follows:

The board above is accessible via zenhub (requires github account)

[sanderegg]

Update on sprint Wankel

Current status

• Fixed redis version in deploys (fix redis version to version 5.0.9-alpine #1913)
• Added image digests to postgres docker image (Adding image digests to databases #1914)
• Fixing duplicate key error in comp_pipelines table (Fixing duplicate key error in comp_pipeline #1934)
• Added timeout for director/filemanager (juyters have enough time to upload and download files in ports) (added timeout for the director and filemanager #1936)
• Ensure only one DB connection acquisition per request in FastAPI servers (ensure only one connection acquisition per request #1966)
• Fix issue closing studies (Do not close/unlock study until it is completely stopped #1937 + Is1875/issue jupyter #1972)
• Fix issue GUI not showing correctly output port updates + persistency of outputs (#312 + Is1875/issue jupyter #1972)
• Fix study will fail after deleting a computation service (#367)
• Fix garbage collector deleting guest users An eager GC deletes guest users while getting created #1853 + bugfix for #1853: garbage collector deletes guest users #1928

Ongoing development

Open points

• long delays fetching s3 data when runs increased Long delay fetching s3 data when runs are increased #1638
• test webserver scaling Webserver scaling #1522
• monitoring to explore the cause of slowdown (still not fully resolved)
• filter e2e testing from metrics Filter metrics from e2e testing #1561
• metrics monitoring issues Metrics for "Total number of anonymous users who are redirected to OSPARC from the SPARC portal" is empty #1915 Metrics: CPU consumption issue the metrics seems to reset #1923 monitoring on production shows issues with regard to metrics CPU consumption #1599

[sanderegg]

Update on sprint Alfred Büchi

Current status

• Report errors during user cleanup Report errors during user cleanup #2013
• Fixed creating a group breaks a study creating a group in the frontend breaks the sudy #2038
• Improve test of e2e in local development Enhancement/improve e2e usage #2056
• Fixed async call to start pipelines comp_task requests are sequentially committed for each node in the project #2043
• Configured more meaningful timeouts for http clients timeouts for http clients #1918
• Fix closing of study (lock issue) Improve closing study workflow #2004
• Avoid setting null on Filepicker output Fix: Avoid setting null on file File Picker's output label #2000

Ongoing development

Open points

• long delays fetching s3 data when runs increased Long delay fetching s3 data when runs are increased #1638
• test webserver scaling Webserver scaling #1522
• monitoring to explore the cause of slowdown (still not fully resolved)
• filter e2e testing from metrics Filter metrics from e2e testing #1561
• metrics monitoring issues Metrics for "Total number of anonymous users who are redirected to OSPARC from the SPARC portal" is empty #1915 Metrics: CPU consumption issue the metrics seems to reset #1923 monitoring on production shows issues with regard to metrics CPU consumption #1599
• node ports to retry upload/download in case of failure node_ports should have retry policies when upload/download fails #2035

[sanderegg]

Update on sprint Chronos

Done

• bugfix: director-v2 is implicitely converting float values to int, preventing kember from running bugfix: director-v2 is implicitely converting float values to int, preventing kember from running #2081
• bugfix: gpu computational service not scheduled correctly bugfix: gpu computational service not scheduled correctly #2090
• bugfix:Fixes/stop services bugfix:Fixes/stop services #2088
• bugfix: kember-viewer fix kember-viewer #2096
• bugfix: access port type using proper attribute bugfix: access port type using proper attribute #2097
• E2E tests: added new tests More e2e tests #2083
• P2P tests: added kember test add kember to p2p testing #2082
• bugfix: address issue acquiring MPI lock in sidecar bugfix: addressed issues when acquiring mpi_lock in sidecar #2093
• E2E testing: start template instead of service e2e isolve fix: Start template instead of service #2100
• bugfix: running one sleeper in a pipeline that contain a filepicker or a dynamic service fails running one sleeper in a pipeline that contain a filepicker or a dynamic service fails #2087
• bugfix: pipeline without computational service is not created in comp_tasks table bugfix: pipeline without computational service is not created in comp_tasks table #2105
• UI: Tags display gets cut in study cards Add unzip 2d cc #396
• issue: "Error saving the study" constantly displaying "Error saving the study" constantly displaying #2113

Ongoing

Open points

• long delays fetching s3 data when runs increased Long delay fetching s3 data when runs are increased #1638
• test webserver scaling Webserver scaling #1522
• filter e2e testing from metrics Filter metrics from e2e testing #1561
• metrics monitoring issues Metrics for "Total number of anonymous users who are redirected to OSPARC from the SPARC portal" is empty #1915 Metrics: CPU consumption issue the metrics seems to reset #1923 monitoring on production shows issues with regard to metrics CPU consumption #1599
• node ports to retry upload/download in case of failure node_ports should have retry policies when upload/download fails #2035

[sanderegg]

Update on sprint The White Rabbit

Done

• Fixes on catalog/services section of the web-API Fixes on catalog/services section of the web-API #2151
• e2e: added MPI test e2e: Add MPI test #2149
• fixes awaitables and broad-exceptions in garbage collector BugFix: fixes awaitables and broad-exceptions in garbage-collector #2104
• Fixes aiopg hanging or raising error Fixes aiopg hanging or raising psycopg2.OperationalError suddenly in gh-actions CI #2138

Ongoing

• Maintenance tasks
• API-to-end testing API-to-end testing #1580
• update python to 3.8 update osparc-simcore to python 3.7 or even 3.8 #1412
• improve scheduling of computational tasks Scheduler: use Director-v2 as scheduler and distribute tasks through Celery #2164

Open points

• increase postgres clients increase postgres clients #2012
• test service resilience Test services resilience #1039
• images registry: failures in images at the registry Images Registry: Failures in images at the registry #2020
• long delays fetching s3 data when runs increased Long delay fetching s3 data when runs are increased #1638
• sidecar logs when pulling Sidecar logs error to pull sleepers during tests/integration/computation/test_computation #1187
• test webserver scaling Webserver scaling #1522
• filter e2e testing from metrics Filter metrics from e2e testing #1561
• metrics monitoring issues Metrics for "Total number of anonymous users who are redirected to OSPARC from the SPARC portal" is empty #1915 Metrics: CPU consumption issue the metrics seems to reset #1923 monitoring on production shows issues with regard to metrics CPU consumption #1599
• node ports to retry upload/download in case of failure node_ports should have retry policies when upload/download fails #2035

[sanderegg]

Update on sprint Red Panda

Done

• Bugfix: fixed access to storage in shared studies Study Collaborators have no access to the Study files #2041

• Bugfix: disallow saving a non opened project after disconnection Disallow saving a non-opened project #2225

• Bugfix: fix archiving/unarchiving using too much memory Avoids memory issues with big files while extracting archives #2192 Fixing wrong file name #2197 Final bugs for state_puller #2200

• Bugfix: shared studies can overwrite services data Fix/simcore sdk error messages #2226

• Bugfix: zero downtime when deploying osparc Zero time restart #2214 missing changes #2229 reduce traefik latency check, and remove unnecessary warnings when a pipeline does not exist #2224

• Bugfix: dynamic services starting behavior Fix dependencies did not start #2219

• UI/UX: message when browser window is too small Screen size message #2217

• Bugfix: frontend cache gets corrupted bugfix: cached resources with multiple keys #2216

• Bugfix: cannot change tags on templates adding and removing tags from templates is now possible #2211

• Bugfix: Handle 500s coming from director-v2 gracefully Webserver: handle 500s from director-v2 gracefully #2202

• Bugfix: storage would crash if metadata would not exist if metadata file did not exist it would crash #2201

• E2E: improvement on jupyters testing wait for notebook to complete #2190 Extending data_manager #2196 E2e wait for notebook2 #2198 E2e wait for notebook3 #2199

• Bugfix: parsing classifiers BugFix: Fixes error parsing classifiers from database #2195

Ongoing

• Maintenance tasks
• API-to-end testing API-to-end testing #1580
• update python to 3.8 update osparc-simcore to python 3.7 or even 3.8 #1412
• improve scheduling of computational tasks Scheduler: use Director-v2 as scheduler and distribute tasks through Celery #2164

Open points

• increase postgres clients increase postgres clients #2012
• test service resilience Test services resilience #1039
• images registry: failures in images at the registry Images Registry: Failures in images at the registry #2020
• long delays fetching s3 data when runs increased Long delay fetching s3 data when runs are increased #1638
• sidecar logs when pulling Sidecar logs error to pull sleepers during tests/integration/computation/test_computation #1187
• test webserver scaling Webserver scaling #1522
• filter e2e testing from metrics Filter metrics from e2e testing #1561
• metrics monitoring issues Metrics for "Total number of anonymous users who are redirected to OSPARC from the SPARC portal" is empty #1915 Metrics: CPU consumption issue the metrics seems to reset #1923 monitoring on production shows issues with regard to metrics CPU consumption #1599
• node ports to retry upload/download in case of failure node_ports should have retry policies when upload/download fails #2035

[sanderegg]

The list of changes for stability/maintenance tasks will be moved to osparc-issues#428

[KZzizzle]

should we close this issue since everything is now under ITISFoundation/osparc-issues#428

[sanderegg]

yes