✨ Add: Manage grafana content/assets via terraform #913

mrnicegyu11 · 2024-12-16T09:38:12Z

What do these changes do?

Add: Conveniant access to CI_ENV_FILE to common.Makefile (necessary for handling S3 tf-state in grafana-tf)
Remove: grafana-export Makefile target, old python grafana import/export scripts
Add: terraform for grafana dashboard management, associated Makefile, jinja2 templating for terraform-grafana to make local deployment with local file provider for terraform state work

Related issue/s

#778

Related PR/s

https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1170

Checklist

I tested and it works

Merge remote-tracking branch 'upstream/main'

mrnicegyu11 · 2024-12-16T09:39:20Z

services/monitoring/grafana/terraform/.terraform-version

@@ -0,0 +1 @@
+1.5.1


is this the right place for the .terraform-version file? Should we rather have only one tf version for the whole repo, at the repo base dir, and maybe even have a pre-commit hook that enforces only one single tf version? not sure

mrnicegyu11 · 2024-12-16T09:40:30Z

services/monitoring/grafana/terraform/datasources.tf

+  url                = var.prometheus_catchall_url
+  basic_auth_enabled = false
+  is_default         = false
+  uid                = "RmZEr52nz"


This "could" have been a password (the academic proper term is "high entropy string" I guess). I think we should have some sort of test (pre-commit?) that asserts that no high-entropy strings are commited in this repo, as it is public on github, and we might leak secrets, and rotating secrets is a lot of annoying work, so I'd like that not to happen :D

…n osparc-config)

YuryHrytsuk

Thanks.

I wonder what can be done with ci.env file

YuryHrytsuk · 2024-12-18T13:08:35Z

scripts/common.Makefile

+# Extract DEPLOYMENT_FQDN using Make functions
+DEPLOYMENT_FQDN := $(notdir $(patsubst %/,%, $(dir $(REPO_CONFIG_LOCATION))))
+# Construct CI_ENV_FILE using Make functions
+CI_ENV_FILE := $(realpath $(dir $(REPO_CONFIG_LOCATION))/../../.gitlab/pipelines/1_configurations/$(DEPLOYMENT_FQDN)/ci.env)


I am not sure if hardcoding assumptions about the other repo here is sustainable.

We can brainstorm together what can be done and if we can avoid using ci.env file here

YuryHrytsuk · 2024-12-18T13:09:14Z

scripts/common.Makefile

+DEPLOYMENT_FQDN := $(notdir $(patsubst %/,%, $(dir $(REPO_CONFIG_LOCATION))))
+# Construct CI_ENV_FILE using Make functions
+CI_ENV_FILE := $(realpath $(dir $(REPO_CONFIG_LOCATION))/../../.gitlab/pipelines/1_configurations/$(DEPLOYMENT_FQDN)/ci.env)
+$(if $(CI_ENV_FILE),,$(error The location of the repo.config file given in .config.location is invalid. Cannot find the ci.env file. Aborting))


This is not going to work in CI where we do not have ci.env file

YuryHrytsuk · 2024-12-18T13:58:35Z

scripts/common.Makefile

@@ -253,13 +260,13 @@ venv: $(REPO_BASE_DIR)/.venv/bin/activate ## Creates a python virtual environmen
 ifeq ($(shell test -f j2cli_customization.py && echo -n yes),yes)

 define jinja
-	$(REPO_BASE_DIR)/.venv/bin/j2 --format=env $(1) .env -o $(2) --customize j2cli_customization.py
+	$(REPO_BASE_DIR)/.venv/bin/j2 --format=env $(1) $(2) -o $(3) --customize j2cli_customization.py


I am not sure if for a single corner case we need to repeat .env call in all places except the corner case

Ideas:

use default value .env if only 2 arguments are passed

define special / different jinja function for grafana terraform

Suggested change

$(REPO_BASE_DIR)/.venv/bin/j2 --format=env $(1) $(2) -o $(3) --customize j2cli_customization.py

$(REPO_BASE_DIR)/.venv/bin/j2 --format=env $(1) $(if $(strip $(3)),$(2),.env) -o $(if $(strip $(3)),$(3),$(2)) --customize j2cli_customization.py

YuryHrytsuk · 2024-12-18T14:05:50Z

services/monitoring/grafana/Makefile

+	set +o allexport; \
+	url=$${TF_VAR_GRAFANA_URL}; \
+	echo "Waiting for grafana at $$url to become reachable..."; \
+	while true; do \


Shall we add a timeout here to avoid waiting forever?

YuryHrytsuk · 2024-12-18T14:12:23Z

services/monitoring/grafana/terraform/dashboards.tf

@@ -0,0 +1,35 @@
+// Import subfolders using an external script
+data "external" "subfolders" {
+  program = ["bash", "${path.module}/../../../../scripts/tf_helper_list_subfolders.sh", "${path.module}/../assets/shared/dashboards"]


Q

since grafana dashboards are now controlled by terraform, shall we store them directly in terraform? Thus we can also avoid any relative ../ imports

YuryHrytsuk · 2024-12-18T14:14:42Z

services/monitoring/grafana/terraform/dashboards.tf

+data "external" "dashboard_files" {
+  for_each = local.folder_map
+
+  program = ["bash", "${path.module}/../../../../scripts/tf_helper_list_json_files_in_folder.sh", "${path.module}/../assets/shared/dashboards/${each.key}"]


I wonder if we can keep this scripts inside terraform folder to avoid having ../../../../

YuryHrytsuk · 2024-12-18T14:18:21Z

services/monitoring/grafana/Makefile

+terraform/main.tf: terraform/main.tf.j2 .venv $(CI_ENV_FILE)
+	# generate $@
+	@$(call jinja, $<, $(CI_ENV_FILE), $@)
+	# validate and format $@


is it a redundant line?

YuryHrytsuk · 2024-12-18T14:43:16Z

services/monitoring/grafana/terraform/main.tf.j2

@@ -0,0 +1,31 @@
+terraform {
+  required_version = "~> 1.5.1"
+  {% if TF_STATE_BACKEND_TYPE != "s3" %}


stricter check? there are many terraform backend types

YuryHrytsuk · 2024-12-18T14:48:36Z

services/monitoring/grafana/terraform/.gitignore

@@ -0,0 +1,6 @@
+main.tf
+.terraform/** */


Q: this */ is to ignore all files but it does not embrace hidden files, correct?

mrnicegyu11 and others added 14 commits October 15, 2024 16:18

wip

f0d8cf0

Merge remote-tracking branch 'upstream/main' into main

e906b41

Merge remote-tracking branch 'upstream/main' into main

14c751d

Add csi-s3 and have portainer use it

293f63c

Change request @Hrytsuk 1GB max portainer volume size

f7f72ec

t push

94cfb76

Merge remote-tracking branch 'upstream/main'

Merge remote-tracking branch 'upstream/main'

509c717

Merge remote-tracking branch 'upstream/main'

1a65ecf

Merge remote-tracking branch 'upstream/main'

77ee45e

Arch Linux Certificates Customization

c9c70d6

Merge remote-tracking branch 'upstream/main'

7b8be53

Merge remote-tracking branch 'upstream/main'

bcd61cd

Merge remote-tracking branch 'upstream/main'

58e1030

Add grafana terrform tooling

5b1c3fb

mrnicegyu11 self-assigned this Dec 16, 2024

mrnicegyu11 commented Dec 16, 2024

View reviewed changes

mrnicegyu11 added 9 commits December 16, 2024 10:50

Make osparc-config dotenv-precommit pass: Use all caps env-vars

00bc0c7

Refactoring: jinja2 takes .env file path as explicit argument (like i…

2e009bc

…n osparc-config)

Make CI_ENV_FILE vailable in makefile

8903007

Refactor makefile targets

f520e7c

Add grafana terraform gitignore

764bb22

Rename envvar: TF_STATE_S3_GRAFANAKEY

ead277a

Remove old scripts, makefile targets

13f92ba

Remove unused files

ebe66c0

undue arch style commit

fcab094

mrnicegyu11 mentioned this pull request Dec 18, 2024

Fix: Varia, Cleanup #911

Open

1 task

Remove references to Tempo

2ba8070

mrnicegyu11 changed the title ~~2024/add/grafana terraform~~ ✨ Add: Manage grafana content/assets via terraform Dec 18, 2024

mrnicegyu11 marked this pull request as ready for review December 18, 2024 09:06

mrnicegyu11 requested a review from YuryHrytsuk as a code owner December 18, 2024 09:06

YuryHrytsuk reviewed Dec 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add: Manage grafana content/assets via terraform #913

✨ Add: Manage grafana content/assets via terraform #913

mrnicegyu11 commented Dec 16, 2024 •

edited

Loading

mrnicegyu11 Dec 16, 2024

mrnicegyu11 Dec 16, 2024 •

edited

Loading

YuryHrytsuk left a comment

YuryHrytsuk Dec 18, 2024

YuryHrytsuk Dec 18, 2024

YuryHrytsuk Dec 18, 2024

YuryHrytsuk Dec 18, 2024

YuryHrytsuk Dec 18, 2024 •

edited

Loading

YuryHrytsuk Dec 18, 2024

YuryHrytsuk Dec 18, 2024

YuryHrytsuk Dec 18, 2024

YuryHrytsuk Dec 18, 2024

	$(REPO_BASE_DIR)/.venv/bin/j2 --format=env $(1) $(2) -o $(3) --customize j2cli_customization.py
	$(REPO_BASE_DIR)/.venv/bin/j2 --format=env $(1) $(if $(strip $(3)),$(2),.env) -o $(if $(strip $(3)),$(3),$(2)) --customize j2cli_customization.py

		@@ -0,0 +1 @@
		1.5.1

		@@ -0,0 +1,6 @@
		main.tf
		.terraform/** */

✨ Add: Manage grafana content/assets via terraform #913

Are you sure you want to change the base?

✨ Add: Manage grafana content/assets via terraform #913

Conversation

mrnicegyu11 commented Dec 16, 2024 • edited Loading

What do these changes do?

Related issue/s

Related PR/s

Checklist

Choose a reason for hiding this comment

mrnicegyu11 Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

YuryHrytsuk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YuryHrytsuk Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrnicegyu11 commented Dec 16, 2024 •

edited

Loading

mrnicegyu11 Dec 16, 2024 •

edited

Loading

YuryHrytsuk Dec 18, 2024 •

edited

Loading