Skip to content

Commit

Permalink
Merge branch 'master' into dmr/replace-production-couch-node-2
Browse files Browse the repository at this point in the history
  • Loading branch information
dannyroberts committed Oct 2, 2023
2 parents 6414850 + aed117f commit b2b4e37
Show file tree
Hide file tree
Showing 11 changed files with 113 additions and 30 deletions.
27 changes: 27 additions & 0 deletions changelog/0074-update-to-python-3.9.18.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
title: update-to-python-3.9.18
key: update-to-python-3.9.18
date: 2023-09-12
optional_per_env: no
# (optional) Min version of HQ that MUST be deployed before this change can be rolled out (commit hash)
min_commcare_version:
# (optional) Max version of HQ that can be deployed before this change MUST be rolled out (commit hash)
max_commcare_version:
context: |
Installs python 3.9.18 and build a new virutalenv for CommCare HQ
details: |
Python 3.9.18 applies some security patches to the known CVEs in earlier versions of Python 3.9.
Performing these steps will ensure that that those CVEs don't affect your environment
update_steps: |
1. Update to the latest commcare-cloud
2. Install Python 3.9.18, update supervisor configs and sudoers file, and create
virtualenv for CommCare HQ:
```bash
cchq <env> deploy-stack --tags=python
```
It will do a dry run, after which you can review the changes and enter 'y' to apply.
3. Restart services:
```bash
cchq <env> service commcare restart
```
32 changes: 32 additions & 0 deletions docs/source/changelog/0074-update-to-python-3.9.18.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<!--THIS FILE IS AUTOGENERATED: DO NOT EDIT-->
<!--See https://github.com/dimagi/commcare-cloud/blob/master/changelog/README.md for instructions-->
# 74. update-to-python-3.9.18

**Date:** 2023-09-12

**Optional per env:** _required on all environments_


## CommCare Version Dependency
This change is not known to be dependent on any particular version of CommCare.


## Change Context
Installs python 3.9.18 and build a new virutalenv for CommCare HQ

## Details
Python 3.9.18 applies some security patches to the known CVEs in earlier versions of Python 3.9.
Performing these steps will ensure that that those CVEs don't affect your environment

## Steps to update
1. Update to the latest commcare-cloud
2. Install Python 3.9.18, update supervisor configs and sudoers file, and create
virtualenv for CommCare HQ:
```bash
cchq <env> deploy-stack --tags=python
```
It will do a dry run, after which you can review the changes and enter 'y' to apply.
3. Restart services:
```bash
cchq <env> service commcare restart
```
5 changes: 5 additions & 0 deletions docs/source/changelog/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ need to be applied on your environment to keep it up to date.

### Changelog

#### **2023-09-12** [update-to-python-3.9.18](0074-update-to-python-3.9.18.md)
Installs python 3.9.18 and build a new virutalenv for CommCare HQ


---
#### **2023-06-14** [Prepare project spaces for Case List Explorer report release](0073-prepare-for-cle-ga.md)
We have created a management command to assist with syncing data to the Case Search Index so that legacy projects
may access data in a new report that will be made generally available.
Expand Down
1 change: 1 addition & 0 deletions environments/production/public.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ filebeat_inputs:
tags: nginx-error

formplayer_java_version: "{{ java_17_bin_path }}/java"
formplayer_archive_time_spec: '3d'
formplayer_purge_time_spec: '10d'
formplayer_sensitive_data_logging: true
formplayer_forward_ip_proxy: true
Expand Down
32 changes: 16 additions & 16 deletions environments/swiss/public.yml
Original file line number Diff line number Diff line change
Expand Up @@ -125,23 +125,23 @@ localsettings:
ENABLE_SOFT_ASSERT_EMAILS: True
ELASTICSEARCH_MAJOR_VERSION: 2
# Index Multiplexer Settings
ES_APPS_INDEX_MULTIPLEXED: False
ES_CASE_SEARCH_INDEX_MULTIPLEXED: False
ES_CASES_INDEX_MULTIPLEXED: False
ES_DOMAINS_INDEX_MULTIPLEXED: False
ES_FORMS_INDEX_MULTIPLEXED: False
ES_GROUPS_INDEX_MULTIPLEXED: False
ES_SMS_INDEX_MULTIPLEXED: False
ES_USERS_INDEX_MULTIPLEXED: False
ES_APPS_INDEX_MULTIPLEXED: False # Swapped
ES_CASE_SEARCH_INDEX_MULTIPLEXED: False # Swapped
ES_CASES_INDEX_MULTIPLEXED: False # Swapped
ES_DOMAINS_INDEX_MULTIPLEXED: False # Swapped
ES_FORMS_INDEX_MULTIPLEXED: False # Swapped
ES_GROUPS_INDEX_MULTIPLEXED: False # Swapped
ES_SMS_INDEX_MULTIPLEXED: False # Swapped
ES_USERS_INDEX_MULTIPLEXED: False # Swapped
# Index Swap Settings
ES_APPS_INDEX_SWAPPED: False
ES_CASE_SEARCH_INDEX_SWAPPED: False
ES_CASES_INDEX_SWAPPED: False
ES_DOMAINS_INDEX_SWAPPED: False
ES_FORMS_INDEX_SWAPPED: False
ES_GROUPS_INDEX_SWAPPED: False
ES_SMS_INDEX_SWAPPED: False
ES_USERS_INDEX_SWAPPED: False
ES_APPS_INDEX_SWAPPED: True
ES_CASE_SEARCH_INDEX_SWAPPED: True
ES_CASES_INDEX_SWAPPED: True
ES_DOMAINS_INDEX_SWAPPED: True
ES_FORMS_INDEX_SWAPPED: True
ES_GROUPS_INDEX_SWAPPED: True
ES_SMS_INDEX_SWAPPED: True
ES_USERS_INDEX_SWAPPED: True
# Index Settings End
IS_DIMAGI_ENVIRONMENT: True
EULA_COMPLIANCE: True
Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ clint==0.5.1
# via commcare-cloud (setup.py)
couchdb-cluster-admin==0.7.2
# via commcare-cloud (setup.py)
cryptography==41.0.3
cryptography==41.0.4
# via
# ansible-core
# commcare-cloud (setup.py)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
become: yes
apt:
name:
- python{{ python_version }}
- python{{ python_version }}={{ python_version }}.*
- python{{ python_version }}-dev
tags:
- python
Expand Down
2 changes: 2 additions & 0 deletions src/commcare_cloud/ansible/roles/elasticsearch/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@
daemon-reload: yes
name: elasticsearch
when: copy_elasticsearch_systemd_result.changed
tags:
- es_conf

- debug: msg="If you're just updating the ES conf then you sill need to restart the cluster. Use the 'es_rolling_restart' playbook."
when: copy_elasticsearch_conf_result.changed
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,6 @@ indices.fielddata.cache.size: {{ elasticsearch_fielddata_cache_size }}
# somewhat of a security risk but required by pact custom reports
script.engine.groovy.inline.aggs: true
script.engine.groovy.inline.search: true
script.engine.groovy.inline.update: true
{% endif %}
{% endif %}

Expand Down
16 changes: 9 additions & 7 deletions src/commcare_cloud/commands/ansible/ansible_playbook.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from commcare_cloud.alias import commcare_cloud
from commcare_cloud.cli_utils import ask, has_arg, check_branch, print_command, has_local_connection_arg
from commcare_cloud.user_utils import get_dev_username
from commcare_cloud.colors import color_error, color_notice, color_code
from commcare_cloud.colors import color_error, color_warning, color_notice, color_code
from commcare_cloud.commands import shared_args
from commcare_cloud.commands.ansible.helpers import (
AnsibleContext, DEPRECATED_ANSIBLE_ARGS,
Expand Down Expand Up @@ -141,12 +141,9 @@ def ansible_playbook(playbook, *cmd_args):
env_vars = ansible_context.build_env()
cmd_parts += get_user_arg(public_vars, unknown_args, use_factory_auth)

if has_arg(unknown_args, '-D', '--diff') or has_arg(unknown_args, '-C', '--check'):
puts(color_error("Options --diff and --check not allowed. "
"Please remove -D, --diff, -C, --check."))
puts(color_error("These ansible-playbook options are managed automatically "
"by commcare-cloud and cannot be set manually."))
return 2 # exit code
if has_arg(unknown_args, '-D', '--diff'):
puts(color_warning("WARNING: Redundant --diff option."))
puts(color_warning("This ansible-playbook option is managed automatically by commcare-cloud."))

cmd_parts += environment.secrets_backend.get_extra_ansible_args()

Expand All @@ -167,6 +164,11 @@ def run_check():
def run_apply():
return ansible_playbook(playbook, *unknown_args)

if has_arg(unknown_args, '-C', '--check'):
# run once with --check if that arg was specified explicitly
with ansible_context.environment.secrets_backend.suppress_datadog_event():
return ansible_playbook(playbook, *unknown_args)

return run_action_with_check_mode(run_check, run_apply, skip_check, quiet, always_skip_check)


Expand Down
23 changes: 19 additions & 4 deletions src/commcare_cloud/commands/terraform/aws.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
import json
import os
import subprocess
import textwrap
from datetime import datetime
from dateutil import parser
import pytz
Expand Down Expand Up @@ -40,6 +39,20 @@ def check_output(cmd_parts, env, silent=False):
return subprocess.check_output(cmd_parts, env=env_vars)


def run(cmd_parts, env, silent=False):

env_vars = os.environ.copy()
env_vars.update(env)
if not silent:
cmd = ' '.join(shlex_quote(arg) for arg in cmd_parts)
print_command('{} {}'.format(
' '.join('{}={}'.format(key, value) for key, value in env.items()),
cmd,
))
# check=True to raise error if results in non-zero exit status
return subprocess.run(cmd_parts, env=env_vars, check=True)


def aws_cli(environment, cmd_parts):

return json.loads(
Expand Down Expand Up @@ -90,7 +103,6 @@ def get_aws_resources(environment):
"--region", config.region,
])]


nlb_endpoints = aws_cli(environment, [
'aws', 'elbv2', 'describe-load-balancers',
'--query', "LoadBalancers[?Type=='network'].[LoadBalancerName,DNSName]",
Expand Down Expand Up @@ -414,7 +426,10 @@ def _aws_sign_in_with_sso(environment):
aws_session_profile = '{}:session'.format(environment.terraform_config.aws_profile)
# todo: add `... or if _date_modified(AWS_CONFIG_PATH) > _date_modified(AWS_CREDENTIALS_PATH)`
if not _has_profile_for_sso(aws_session_profile):
puts(color_notice("Configuring SSO. To further customize, run `aws configure sso --profile {}`".format(aws_session_profile)))
puts(color_notice(
"Configuring SSO. To further customize, run `aws configure sso "
"--profile {}`".format(
aws_session_profile)))
_write_profile_for_sso(
aws_session_profile,
sso_start_url=environment.aws_config.sso_config.sso_start_url,
Expand Down Expand Up @@ -627,7 +642,7 @@ def _has_valid_session_credentials_for_sso():


def _refresh_sso_credentials(aws_session_profile):
check_output(['aws', 'sso', 'login'], env={'AWS_PROFILE': aws_session_profile})
run(['aws', 'sso', 'login'], env={'AWS_PROFILE': aws_session_profile})


def _has_valid_v1_session_credentials(aws_profile):
Expand Down

0 comments on commit b2b4e37

Please sign in to comment.