Add support for persistent data volume

[hellais]

• add basic readme file
• also fix deploy checks when a branch lands on master, since it fails because the comment script doesn't have in it the reference to the open PR number

[github-actions]

Terraform Run Output 🤖

Format and Style 🖌success

Initialization ⚙️success

Validation 🤖success

Validation Output

$ terraform validate
Success! The configuration is valid.

Plan 📖success

• Plan: 0 to add, 1 to change, 0 to destroy.

Show Plan

$ terraform plan
Acquiring state lock. This may take a few moments...
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Reading...
data.aws_availability_zones.available: Reading...
aws_cloudwatch_log_group.app: Refreshing state... [id=tf-ecs-group/app-dataapi]
data.aws_ami.debian_ami: Reading...
module.terraform_state_backend.data.aws_region.current: Reading...
aws_acm_certificate.dataapi: Refreshing state... [id=arn:aws:acm:eu-central-1:082866812839:certificate/27b7c5e3-ef70-41e8-a6cf-ac0d58a99613]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=ooni-production-terraform-state-lock]
aws_iam_role.app_instance: Refreshing state... [id=tf-ecs-ooni-instance-role]
aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:cluster/ooni-ecs-cluster]
module.terraform_state_backend.aws_s3_bucket.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.data.aws_region.current: Read complete after 0s [id=eu-central-1]
aws_cloudwatch_log_group.ecs: Refreshing state... [id=tf-ecs-group/ecs-agent]
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Read complete after 0s [id=2050088263]
data.aws_ssm_parameter.ecs_optimized_ami: Reading...
aws_vpc.main: Refreshing state... [id=vpc-08b101405472a0b46]
data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 0s [id=/aws/service/ecs/optimized-ami/amazon-linux-2/recommended]
data.aws_availability_zones.available: Read complete after 0s [id=eu-central-1]
aws_iam_role.ecs_service: Refreshing state... [id=ooni_ecs_role]
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Read complete after 0s [id=2050088263]
aws_iam_instance_profile.app: Refreshing state... [id=tf-ecs-instprofile]
data.aws_ami.debian_ami: Read complete after 0s [id=ami-0e626c31414223120]
aws_iam_role_policy.instance: Refreshing state... [id=tf-ecs-ooni-instance-role:TfEcsOONIInstanceRole]
aws_ecs_task_definition.dataapi: Refreshing state... [id=ooni-dataapi-production-td]
aws_iam_role_policy.ecs_service: Refreshing state... [id=ooni_ecs_role:ooni_ecs_policy]
aws_route53_record.dataapi_cert_validation["dataapi.prod.ooni.io"]: Refreshing state... [id=Z02418652BOD91LFA5S9X__74a0d0e91f58697b6838bed5a2fb4939.dataapi.prod.ooni.io._CNAME]
aws_acm_certificate_validation.dataapi: Refreshing state... [id=2024-02-05 11:11:01.598 +0000 UTC]
aws_internet_gateway.gw: Refreshing state... [id=igw-05319878d828e0a53]
aws_subnet.main[0]: Refreshing state... [id=subnet-06d47d18b015109d4]
aws_subnet.main[1]: Refreshing state... [id=subnet-02241f60bd951358d]
aws_alb_target_group.dataapi: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:targetgroup/ooni-ecs-dataapi/c16001d24f858f20]
aws_security_group.clickhouse_sg: Refreshing state... [id=sg-0c0ffdfe535bededa]
aws_security_group.lb_sg: Refreshing state... [id=sg-090ff83a71e7ad235]
aws_route_table.r: Refreshing state... [id=rtb-06814accd93aca9ce]
aws_instance.clickhouse_server_prod_tier1: Refreshing state... [id=i-0aaa354af74a5e988]
module.terraform_state_backend.aws_s3_bucket_server_side_encryption_configuration.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_public_access_block.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_versioning.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_route_table_association.a[0]: Refreshing state... [id=rtbassoc-049f8c5e84cb32e55]
aws_route_table_association.a[1]: Refreshing state... [id=rtbassoc-034c4831c1754e284]
aws_alb.main: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:loadbalancer/app/ooni-alb-ecs/4cf8bd00b53f8961]
aws_security_group.instance_sg: Refreshing state... [id=sg-0f39450a61732d69b]
module.terraform_state_backend.aws_s3_bucket_policy.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_launch_template.app: Refreshing state... [id=lt-01b923563af0432b5]
module.terraform_state_backend.time_sleep.wait_for_aws_s3_bucket_settings[0]: Refreshing state... [id=2024-02-02T21:32:23Z]
module.terraform_state_backend.aws_s3_bucket_ownership_controls.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_autoscaling_group.app: Refreshing state... [id=ooni-tier1-production-backend-asg20240119184843559900000002]
aws_route53_record.alb_dns: Refreshing state... [id=Z02418652BOD91LFA5S9X_dataapi.prod.ooni.io_A]
aws_alb_listener.front_end: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/1cd9240a34a13129]
aws_alb_listener.front_end_https: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/b7ec77ceadac94c8]
aws_ecs_service.dataapi: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production]
data.aws_ebs_volume.clickhouse_data_volume: Reading...
aws_eip.clickhouse_ip: Refreshing state... [id=eipalloc-0f0a20bba4a1d858c]
aws_route53_record.clickhouse_dns: Refreshing state... [id=Z035992527R8VEIX2UVO0_clickhouse.tier1.prod.ooni.nu_A]
data.aws_ebs_volume.clickhouse_data_volume: Read complete after 0s [id=vol-0753909d310ed6d8d]
aws_volume_attachment.clickhouse_data_volume_attachment: Refreshing state... [id=vai-2291175638]
local_file.ansible_inventory: Refreshing state... [id=e7b3f71fb68cc29f026b251ba92a9bfe9a6e4fd6]
null_resource.ansible_update_known_hosts: Refreshing state... [id=4418919173081010018]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_ecs_service.dataapi will be updated in-place
  ~ resource "aws_ecs_service" "dataapi" {
        id                                 = "arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production"
        name                               = "ooni-ecs-dataapi-production"
        tags                               = {
            "Name"       = "ooni-tier1-production"
            "Repository" = "https://github.com/ooni/devops"
        }
      ~ triggers                           = {
          ~ "redeployment" = "2024-02-14T13:54:23Z" -> "2024-02-14T13:58:48Z"
        }
        # (15 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Pusher	@hellais
Action	pull_request
Working Directory	./tf/environments/production
Workflow	.github/workflows/check_deploy.yml
Last updated	Wed, 14 Feb 2024 13:58:54 GMT

[github-actions]

Ansible Run Output 🤖

Ansible Playbook Recap 🔍

clickhouse.tier1.prod.ooni.nu : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Ansible playbook output 📖success

Show Execution

$ ansible-playbook playbook.yml --check --diff -i inventory.ini

PLAY [ClickHouse servers] ******************************************************

TASK [Gathering Facts] *********************************************************
fatal: [clickhouse.tier1.prod.ooni.nu]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:wuGEgIlSHvtKTYutVHFude7FVWDt4KkdropiF+iHCew.
Please contact your system administrator.
Add correct host key in known_hosts to get rid of this message.
Offending ED25519 key in known_hosts:5
  remove with:
  ssh-keygen -f "known_hosts" -R "clickhouse.tier1.prod.ooni.nu"
Host key for clickhouse.tier1.prod.ooni.nu has changed and you have requested strict checking.
Host key verification failed.", "unreachable": true}

PLAY RECAP *********************************************************************
clickhouse.tier1.prod.ooni.nu : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   

Pusher	@hellais
Action	pull_request
Working Directory	./tf/environments/production
Workflow	.github/workflows/check_deploy.yml
Last updated	Wed, 14 Feb 2024 14:00:04 GMT

[hellais]

Something is funky in the plan. Need to troubleshoot that.

[hellais]

Ok I have fixed it.

What happened, was that debian updated their base OS image and so that was triggering a redeploy of the container. It's a good idea to figure out a pattern for a non destructive upgrade of the base OS, so I did that.

The basic idea is that we want to manually or semi-manually create the data volume and then pass in the tag for it as a data attribute to terraform. That way we don't destroy and recreate the data volume thereby loosing data.

I have added a comment about that.

[bassosimone]

🐳