Roadmap to FastAPI

[mvdbeek]

• Make run.sh use the new stack (exclusively ?): Adjust run.sh and run_tests.sh to use gunicorn + fastapi by default #10885
• Replace/update mentions of uwsgi as appropriate in:
  • https://docs.galaxyproject.org/en/master/admin/framework_dependencies.html Touch up framework dependency documentation #13379
  • https://docs.galaxyproject.org/en/master/admin/config.html @ic4f Remove uwsgi mention from debug config option #13426 Update config docs (uwsgi >> gunicorn) #13456
  • https://docs.galaxyproject.org/en/master/admin/config_logging.html @ic4f
  • https://docs.galaxyproject.org/en/master/admin/nginx.html @mvdbeek [22.01] Document gunicorn and nginx #13436
  • https://docs.galaxyproject.org/en/master/admin/apache.html @mvdbeek [22.01] Document gunicorn and apache #13439
  • https://docs.galaxyproject.org/en/master/admin/scaling.html @mvdbeek [22.01] Drop uWSGI from scaling documentation #13381
  • https://docs.galaxyproject.org/en/master/admin/cluster.html @mvdbeek [22.01] Update jobs and cluster documentation #13440
  • https://docs.galaxyproject.org/en/master/admin/jobs.html @mvdbeek [22.01] Update jobs and cluster documentation #13440
  • https://docs.galaxyproject.org/en/master/admin/reports.html @ic4f Drop uWSGI from Galaxy Reports documentation #13406
  • https://docs.galaxyproject.org/en/master/admin/special_topics/interactivetools.html @mvdbeek
  • https://docs.galaxyproject.org/en/master/admin/special_topics/interactive_environments.html (remove or merge proxy instructions with interactivetools docs) @mvdbeek [22.01] Drop interactive environment build instructions from docs #13442
  • https://docs.galaxyproject.org/en/master/admin/special_topics/performance_tracking.html @mvdbeek [22.01] Update performance tracking for Gunicorn #13441
  • .k8s_ci.Dockerfile
  • contrib/galaxy_supervisor.conf (drop?)
  • doc/source/admin/config_logging.rst ("... identifying the uWSGI worker ID and mule ID ...")
  • doc/source/admin/migrating_to_gunicorn.md (update for 22.05)
  • doc/source/admin/proxy_package_layout.rst
  • doc/source/admin/scaling.md (update for 22.05)
  • doc/source/admin/special_topics/interactivetools.rst ("Native, out-of-the box support for InteractiveTools in Galaxy via uWSGI proxying")
  • lib/galaxy/config/sample/job_conf.xml.sample_advanced
  • pyproject.toml
  • Ansible Galaxy
  • https://training.galaxyproject.org/training-material/topics/admin/ @hexylena @natefoo @Slugger70
• Write a "why fastAPI" document [22.01] Add Migrating from uWSGI to Gunicorn and FastAPI document #13480
• Write a "how to migrate" section [22.01] Add Migrating from uWSGI to Gunicorn and FastAPI document #13480
• Create gravity schema & include sample in galaxy.yml.sample [22.01] Include gravity in sample config #13531
• Write a FAQ ?
• Get together with @gmauro @hexylena @natefoo @Slugger70 and update playbooks and test servers
• Remove uwsgi section from galaxy.yml, port is going to be set in gravity section
• Remove paste entry points and scripts, remove .github/workflows/api_paste.yaml and GALAXY_TEST_USE_UVICORN [22.05] Remove uWSGI / WSGI support #13071
• Remove duplicated API controllers [22.05] Drop redundant legacy API endpoints  #13582

I will update this as we go along.

[natefoo]

 https://docs.galaxyproject.org/en/master/admin/special_topics/interactivetools.html (can I convince @hexylena or @natefoo to take this on ? It may include adding the nodejs proxy to the circusd config)

Yes.

[mvdbeek]

@natefoo @dannon
I'm working on removing all the uwsgi/paster cruft (https://github.com/galaxyproject/galaxy/compare/dev...mvdbeek:drop_wsgi_mode?expand=1) and updating run.sh, and ran into one thing I should've thought about but didn't ... do we want to provide a circusd instance per thing (galaxy, toolshed, reports) ? That seems like the only way we can just do ./run.sh or ./run_tool_shed.sh without coordinating which services to start. Or are we ok with one circusd just for Galaxy, and the rest will continue to start the process directly ? I guess I'd prefer that since it's less work, but I think we need a cron-style task for the tool shed already ... or I'll have to refactor that one toolshed cron task which I don't think has to be a cron task in the first place ...

[natefoo]

We have discussed this some on matrix, but for the record, I'm not really concerned about the other applications, fwiw. We actively discourage anyone from running a TS, so reports is the only question here, which many production servers also do not run. We could allow people to optionally run it from the Galaxy supervisor I guess?

[mvdbeek]

Alright, made a lot of progress, and the remaining tasks we can do in the 22.05 cycle

[nsoranzo]

Q: Should we drop merge_duplicate_keys=True from _order_load_path() in lib/galaxy/config/config_manage.py ? It was needed because uWSGI's YAML config allowed duplicated keys. But if we drop it, I think we cannot simply ignore the uwsgi: section of migrated galaxy.yml files, but we would need to ask admins to remove it, otherwise parsing may fail.

[mvdbeek]

But if we drop it, I think we cannot simply ignore the uwsgi: section of migrated galaxy.yml files, but we would need to ask admins to remove it, otherwise parsing may fail.

I think we should slightly restructure the file. For the conversion path I think it's fine if we rewrite the config file without the uwsgi section, for the validation part it's fine to raise a validation error.

[bgruening]

Impressive work! I guess we can close this?

[mvdbeek]

I think there's a little bit of docs left to do, I'll look over this