TODO Migration

For the docker/eu-central-1/python3 migration, and in the aftermath

Migration

• Local serving via make -f Makefile.frankfurt serve doesn't work (https://jira.swisstopo.ch/browse/BGDIINF_SB-2446)
• Replace dev, int and prod.env thought local.env (other vars are set by docker-compose and k8s) (#3896)
• Keep docker-compose.yaml or use only docker run? If no, templates generation cleanup (#3905)
• Remove the proxy stuff in the apache conf (no proxy, the app is served though mod_wsgiapache module) (#3897 )
• Secrets: pgpass and pgsuer, opentrans_api_key in gopass/summon (https://jira.swisstopo.ch/browse/BGDIINF_SB-2414)
• Port cleanup SERVER_PORT vs APACHE_PORT (https://jira.swisstopo.ch/browse/BGDIINF_SB-2415)
• Use requirements_py3.txt to install dependencies within the docker image, instead of make setup (https://jira.swisstopo.ch/browse/BGDIINF_SB-2416) (#3917)
• Check, which envsubsts are needed in docker-entrypoint.sh and the environ target of Makefile.frankfurt. Could some be cleaned-up? (https://jira.swisstopo.ch/browse/BGDIINF_SB-2417)
• Clean-up the env var list in .env.default and set the according list of env vars per staging in the k8s repository (https://jira.swisstopo.ch/browse/BGDIINF_SB-2418)
• Various log handler in the Pylons settings (base.ini, local.ini, etc) files: still make sense?
• Python test files (nose) still using development.ini (hardcoded). (https://jira.swisstopo.ch/browse/BGDIINF_SB-2428)
• For now using tests from Makefile in the CI, we should use at some point Makefile.frankfurt when a all test passe (https://jira.swisstopo.ch/browse/BGDIINF_SB-2427)
• Dockerfile optimisation (layers, caching, etc) (https://jira.swisstopo.ch/browse/BGDIINF_SB-2426)
• Docker file must use Makefile.frankfurt
• .git in dockerignore (1.1 GB of history) and git hash version stuff (how to get git hashes in docker when ther is no git repository anymore)(https://github.com/geoadmin/mf-chsdi3/pull/3900)
• Fix the failing tests in Makefile.frankfurt (https://jira.swisstopo.ch/browse/BGDIINF_SB-2425)
• Check for resources in Frankfurt (S3, pg-replica). Wait for now, as the DB is not very much in sync (failing tests)
• Docker image tag for local build (make dockerbuild dockerrun)? Not getting the images from the ECR (https://jira.swisstopo.ch/browse/BGDIINF_SB-2424)
• This VERSION, DEPLOY_TARGET variable in docker-entrypoint.sh ?? (#3903, #)
• potomo and translate, needed, in github? (#3901)
• IIPIMAGE was was that? -> ltclm was a predecessor of statified images in s3 aerialimages, failed poc not used anymore
• Compiling CSS (for doc): installing node (#3908)
• Statics files (doc, rss) (#3904)
• make legends? Still used? (#3912)
• Cleanup/clarify app_version and VERSION (use GIT_HASH? What is it for?) (https://jira.swisstopo.ch/browse/BGDIINF_SB-2423)

Failing tests

https://sys-api3.dev.bgdi.ch/rest/services/api/MapServer/ch.bafu.nabelstationen/legend https://sys-api3.dev.bgdi.ch/rest/services/api/MapServer?searchText=wasser https://sys-api3.dev.bgdi.ch/rest/services/api/MapServer/ch.bafu.nabelstationen/RIG/htmlPopup https://sys-api3.dev.bgdi.ch/rest/services/api/SearchServer?features=ch.bfs.gebaeude_wohnungs_register&type=featuresearch&searchText=isabelle

test_identify_bbox_offset (tests.integration.test_identify_service.TestIdentifyService) ... ERROR test_identify_one_timeinstant_several_layers (tests.integration.test_identify_service.TestIdentifyService) ... ERROR test_identify_several_timeinstants (tests.integration.test_identify_service.TestIdentifyService) ... ERROR test_feature_grid_valid_feature (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_htmlpopup_invalid_id_format (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_htmlpopup_time (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_htmlpopup_valid (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_htmlpopup_valid_lv95 (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_htmlpopup_valid_topic_all (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_htmlpopup_valid_with_callback (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_htmlpopup_with_iframeservice (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_iframehtmlpopup (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_none_layer (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_searchField_error (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_searchField_none (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_several_features (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_several_features_geojson (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_two_layer (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_unsupported_srid (tests.integration.test_features_service.TestFeaturesView) ... ERROR test_validate_wmtscapabilities (tests.integration.test_wmtscapabilities.TestWmtsCapabilitiesView) ... ERROR test_wrong_epsg_wmtscapabilities (tests.integration.test_wmtscapabilities.TestWmtsCapabilitiesView) ... ERROR test_wrong_map_wmtscapabilities (tests.integration.test_wmtscapabilities.TestWmtsCapabilitiesView) ... ERROR

Check for production

• Check logging in Kibana
• JSON logs (or later)
• JSON logs with Apache not viable, see https://jira.swisstopo.ch/browse/BGDIINF_SB-2459, ticket might be reopened if move from apache to gunicorn
• Load tests
• Performance tests
• Monitoring

Cleanup

• Write e2e tests (https://jira.swisstopo.ch/browse/BGDIINF_SB-2480)
• Remove https://github.com/geoadmin/mf-chsdi3/blob/master/tests/e2e/test_varnish.py (no mapproxy anymore)

Once Frankfurt is in production

• Remove all deployment scripts of Ireland. (https://jira.swisstopo.ch/browse/BGDIINF_SB-2454)
• Remove py2 support (https://jira.swisstopo.ch/browse/BGDIINF_SB-2496)
• For the docker image to be built quickier don't package the application with setup.cfg and setup.py. Note this can/should be done together with pipenv (https://jira.swisstopo.ch/browse/BGDIINF_SB-2506).
• Use pipenv, yapf, pipfile, etc. (split python libraries into runtime and develop (was the case in py2), https://jira.swisstopo.ch/browse/BGDIINF_SB-2506 and https://jira.swisstopo.ch/browse/BGDIINF_SB-2507)
• Use nose2 instead of nosetests (https://jira.swisstopo.ch/browse/BGDIINF_SB-2497)
• Upgrade python3 version to 3.9 (currently 3.7). Both for local development and within docker image (https://jira.swisstopo.ch/browse/BGDIINF_SB-2498)
• remove the guard- target (ltshb: which guard target ? I would keep them)
• Check if apache is still needed or can be removed/replaced by gunicorn, if not removed then cleanup and simplify the apache config (https://jira.swisstopo.ch/browse/BGDIINF_SB-2486)
• If moved to gunicorn reopened https://jira.swisstopo.ch/browse/BGDIINF_SB-2459
• Remove wmts capabilities and tests(https://jira.swisstopo.ch/browse/BGDIINF_SB-1825)
• Remove service-search and tests
• Migrate wmts and wms e2e tests
• Migrate all e2e/intergration tests into infra-e2e-tests
• Check and cleanup docker user and permissions (start root process as geodata ?) (https://jira.swisstopo.ch/browse/BGDIINF_SB-2471)
• Rename Makefile.frankfurt
• remove templates/info.json.mako (as static/info.json is being rendered differently)
• remove python2 support from makos. F.ex. on kgs.mako, zeitreihe.mako, statistikwasserkraftanlagen.mako, lubis.mako, lubis_terra.mako, lubis_schraegaufnahme.mako (generall searching for #python2/3) (https://jira.swisstopo.ch/browse/BGDIINF_SB-2496)
• We should try to use the same python lib for all download. Currently we have http2, the stand urllib and requests. First we will have to define a best-practice and then hamonise the makos based on this.
• Check if the KML related code is still needed, and, if not, remove it (https://jira.swisstopo.ch/browse/BGDIINF_SB-2456)
• Re-enable accept-language e2e tests in infra-e2e-tests (search for TODO: Not working in IRELAND, wait for FRANKFURT migration)