helm chart for cht sync #90

witash · 2024-04-19T11:42:05Z

this branch contains

a helm chart for deploying cht sync on kubernetes.
including logstash, postgrest, and dbt containers for all deployments
optionally a postgres db and service
and experimentally a redis instance and the redis worker to copy data from it
change to the github task which was building and pushing images with the :latest tag on every push on every branch, which causes problems for deploying on kubernetes; for now, it just tags images with the branch name, we can make additional changes in its own branch after this is merged.
creates the couchdb database in the dbt worker.

Closes #75

github-advanced-security · 2024-04-23T09:18:04Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

dianabarsan · 2024-05-21T06:46:29Z

deploy/cht_sync/templates/postgrest.yaml

+          # Add environment variables and volume mounts as needed
+          env:
+            - name: PGRST_DB_URI
+              value: "postgres://postgres:postgres@postgres:5432/data"


It seems that sonar really wants to not share dummy passwords. Can we try something like:

postgres://<pguser>:<pgpassword>@postgres:5432/data

deploy/cht_sync/templates/postgrest.yaml

+        inner.service: postgrest
+    spec:
+      containers:
+        - name: postgrest


deploy/cht_sync/templates/postgrest.yaml

+        inner.service: postgrest
+    spec:
+      containers:
+        - name: postgrest


deploy/cht_sync/templates/postgrest.yaml

+      labels:
+        app: cht-sync
+        inner.service: postgrest
+    spec:


deploy/cht_sync/templates/logstash.yaml

+        app: cht-sync
+    spec:
+      containers:
+        - name: logstash


deploy/cht_sync/templates/logstash.yaml

+        app: cht-sync
+    spec:
+      containers:
+        - name: logstash


deploy/cht_sync/templates/logstash.yaml

+    metadata:
+      labels:
+        app: cht-sync
+    spec:


deploy/cht_sync/templates/dbt.yaml

+    metadata:
+      labels:
+        app: cht-sync
+    spec:


deploy/cht_sync/templates/dbt.yaml

+        app: cht-sync
+    spec:
+      containers:
+        - name: dbt


deploy/cht_sync/templates/dbt.yaml

+        app: cht-sync
+    spec:
+      containers:
+        - name: dbt


witash · 2024-05-03T13:20:22Z

this is not totally final because there isn't a good way to access the postgres db in the cluster (see #87). but I want to merge this to main since #79 depends on it and the github build task in main keeps overwriting the latest images with feature branches.
in the meantime, you can access the database for testing and development if you have kubectl access kubectl exec -it -tty postgres-xxxx -- psql

dianabarsan

I'm leaving the ymls to be reviewed by the experts, I added some small comments about the values template.

deploy/cht_sync/values.yaml.template

andrablaj · 2024-05-16T10:48:47Z

@witash, is this PR ready for re-review? If yes, re-request review from @dianabarsan or any other people who should have a look at this.

njuguna-n · 2024-05-20T14:48:32Z

deploy/cht_sync/templates/postgrest.yaml

+              memory: {{ (.Values.postgrest).memory_limit | default "500Mi" }}
+          env:
+            - name: PGRST_DB_URI
+              value: |


This causes the error below. Setting the value to {{ printf "postgres://%s:%s@%s:5432/%s" .Values.postgres.user .Values.postgres.password .Values.postgres.host .Values.postgres.db }} works though.

{"code":"PGRST000","details":"connection to server at \"postgres\" (172.20.202.161), port 5432 failed: FATAL: database \"data\n\" does not exist\n","hint":null,"message":"Database connection error. Retrying the connection."}

dianabarsan

Cool, I have an overarching comment about dbt-run.py, this is not entirely related to this work of adding helm charts.

For the helm chart work, I believe we should add an e2e test that deploys cht-sync over k3d to validate these charts.

dianabarsan · 2024-05-21T06:48:30Z

dbt/dbt-run.py

-            {os.getenv('POSTGRES_SCHEMA')}
+            {os.getenv('POSTGRES_SCHEMA')};
+
+            CREATE TABLE IF NOT EXISTS {os.getenv('POSTGRES_SCHEMA')}.{os.getenv('POSTGRES_TABLE')} (


I think we should totally aim not not create tables and such in code. I believe the whole allure of dbt is that you can have these versioned clean schema files, but here we're just inlining the table create in python?
Is there a way we can use dbt to create these and keep the schema in its own file?

Also, how is this connected to the helm chart?

its not related to the the helm chart and was already merged to main separately.
dbt mainly creates tables by wrapping select from existing tables in create table statements, the assumption is that there's some source db that dbt is not managing.

but it is possible to just run raw sql, including ddl and yea i agree in this case probably makes more sense to do that instead of creating the table here. Also then could just have one "root" table instead of two.

related to medic/cht-pipeline#84

dianabarsan

Cool stuff!
I left a comment about how couchdb instances affect values in config,

dianabarsan · 2024-07-18T05:00:02Z

deploy/cht_sync/templates/couch2pg.yaml

+            - name: POSTGRES_SCHEMA
+              value: {{ $.Values.postgres.schema }}
+            - name: COUCHDB_USER
+              value: {{ $.Values.couchdb.user }}


The user can vary between instances

dianabarsan · 2024-07-18T05:00:28Z

deploy/cht_sync/templates/couch2pg.yaml

+            - name: COUCHDB_HOST
+              value: {{ $service.host }}
+            - name: COUCHDB_DBS
+              value: {{ $.Values.couchdb.dbs }}


We could also vary between which databases we sync from every instance.

dianabarsan · 2024-07-18T05:00:58Z

deploy/cht_sync/templates/couch2pg.yaml

+            - name: COUCHDB_DBS
+              value: {{ $.Values.couchdb.dbs }}
+            - name: COUCHDB_PORT
+              value: {{ $.Values.couchdb.port | quote }}


Same for port and secure.
We can't assume all instances will be the same.

dianabarsan · 2024-07-18T13:02:01Z

deploy/cht_sync/values.yaml.template

+# values for each couchdb instance
+# host and password are required
+# other values can be ommitted if common to all couchdb instances and specified above
+couchdbs: # values for in


This doesn't really exemplify of how I would add multiple couchdb servers to sync from.

can you elaborate on this? what specifically does not exemplify how would you add multiple couchdb servers to sync from?

As a reader I'm not imagining how this would look like.

Is it like:

couchdbs: # values for in - host: "host1" password: "" user: "" dbs: "medic medic-sentinel" port: "5984" secure: "true" - host: "host2" password: "" user: "" dbs: "medic medic-users" port: "5984" secure: "true"

I think it would be helpful if we add an example, because these iterator types are weird in yml

yes, that's what its supposed to be.
There is a section couchdb for shared values, and couchdbs for a list of values which are not shared.
which makes it a little more complicated, but otherwise when user,dbs,port and secure are NOT different, like for MoH Kenya, have to copy them 47 times, and don't like copy/pasting that much.

added an example to values.yaml.template

couchdbs: - host: "host1" # required for all couchdb instances password: "" # required for all couchdb instances - host: "host2" password: "" - host: "host3" password: "" user: "user2" # required if different than above dbs: "medic medic_sentinel" # required if different than above port: "5984" # required if different than above secure: "true" # required if different than above

thanks a lot, I think this example is super helpful.

dianabarsan

Love it! Thanks for all the updates!

* add helm chart * fix typos * add volumeClaim and remove serviceName * changing postgres service to load balancer for access * add redis and redis-worker * fix sonar issues and resource limits * fix defaults and allow configurable limts * tag images with branch and allow use in templates * initialize table in dbt container * remove external load balancer * remove defaults in values template * change redis port * remove newline from env var * remove old templates and add couch2pg * multiple instances for couch2pg * allow user,dbs,port, and secure to vary * adding multi instance example

medic-ci · 2024-09-10T14:27:29Z

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

witash added 3 commits April 11, 2024 13:46

add helm chart

e9c077c

fix typos

abe9981

add volumeClaim and remove serviceName

ed79362

witash mentioned this pull request Apr 19, 2024

Create Postgres instance using EKS #87

Open

njuguna-n mentioned this pull request Apr 19, 2024

Use Redis for sequence token #4

Closed

changing postgres service to load balancer for access

7c3c398

github-advanced-security bot found potential problems Apr 23, 2024

View reviewed changes

witash added 7 commits April 23, 2024 12:26

Merge branch 'main' into helm

6e432da

add redis and redis-worker

07063c4

fix sonar issues and resource limits

a44b933

fix defaults and allow configurable limts

bb43d08

tag images with branch and allow use in templates

3118482

initialize table in dbt container

34ff7c8

remove external load balancer

c1bf1ff

witash marked this pull request as ready for review May 3, 2024 13:12

witash changed the title ~~helm chart~~ helm chart for cht sync May 3, 2024

witash requested review from Hareet and dianabarsan May 3, 2024 14:31

witash requested review from nydr and removed request for Hareet May 13, 2024 12:30

dianabarsan reviewed May 13, 2024

View reviewed changes

deploy/cht_sync/values.yaml.template Outdated Show resolved Hide resolved

deploy/cht_sync/values.yaml.template Outdated Show resolved Hide resolved

deploy/cht_sync/values.yaml.template Show resolved Hide resolved

witash added 3 commits May 14, 2024 09:50

remove defaults in values template

328375a

Merge branch 'main' into helm

9b24f2a

change redis port

afdca7b

njuguna-n mentioned this pull request May 16, 2024

Use http output with json batch #102

Closed

andrablaj requested a review from dianabarsan May 20, 2024 13:48

njuguna-n reviewed May 20, 2024

View reviewed changes

dianabarsan requested changes May 21, 2024

View reviewed changes

witash added 4 commits May 21, 2024 10:18

remove newline from env var

b86a1ad

Merge branch 'main' into helm

35462d4

Merge branch 'main' into helm

0e23dce

remove old templates and add couch2pg

6fbb592

andrablaj removed the request for review from nydr June 17, 2024 09:33

witash added 2 commits July 15, 2024 13:43

multiple instances for couch2pg

5e4f34f

Merge branch 'main' into helm

4325b6a

witash requested a review from dianabarsan July 16, 2024 08:15

dianabarsan requested changes Jul 18, 2024

View reviewed changes

allow user,dbs,port, and secure to vary

cd03d4b

witash requested a review from dianabarsan July 18, 2024 10:39

dianabarsan reviewed Jul 18, 2024

View reviewed changes

adding multi instance example

458e9c9

dianabarsan approved these changes Jul 18, 2024

View reviewed changes

witash merged commit 63cc236 into main Jul 19, 2024
5 checks passed

witash deleted the helm branch July 19, 2024 07:57

njuguna-n mentioned this pull request Jul 23, 2024

Deploy cht-sync on MoH Kenya servers #127

Closed

medic-ci added the released label Sep 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

helm chart for cht sync #90

helm chart for cht sync #90

witash commented Apr 19, 2024 •

edited by andrablaj

Loading

github-advanced-security bot commented Apr 23, 2024

dianabarsan May 21, 2024

witash commented May 3, 2024 •

edited

Loading

dianabarsan left a comment

andrablaj commented May 16, 2024

njuguna-n May 20, 2024

dianabarsan left a comment

dianabarsan May 21, 2024

dianabarsan May 21, 2024

witash May 21, 2024

dianabarsan left a comment

dianabarsan Jul 18, 2024

dianabarsan Jul 18, 2024

dianabarsan Jul 18, 2024

dianabarsan Jul 18, 2024

witash Jul 18, 2024

dianabarsan Jul 18, 2024

witash Jul 18, 2024

dianabarsan Jul 18, 2024

dianabarsan left a comment

medic-ci commented Sep 10, 2024

helm chart for cht sync #90

helm chart for cht sync #90

Conversation

witash commented Apr 19, 2024 • edited by andrablaj Loading

github-advanced-security bot commented Apr 23, 2024

Choose a reason for hiding this comment

witash commented May 3, 2024 • edited Loading

dianabarsan left a comment

Choose a reason for hiding this comment

andrablaj commented May 16, 2024

Choose a reason for hiding this comment

dianabarsan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dianabarsan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dianabarsan left a comment

Choose a reason for hiding this comment

medic-ci commented Sep 10, 2024

witash commented Apr 19, 2024 •

edited by andrablaj

Loading

witash commented May 3, 2024 •

edited

Loading