feature: support GCP cloud storage

[lmello]

This changes leverage the contribution by cklingspor, which added a storage factory and azure support to add support for gcp.

This is a work in progress and need more testing.

[lmello]

I need help to test this in gcp. please let me know if you have access to it.

I will have access to a gcp environment in a project that will use this code in a few weeks, I would appreciate if anyone could help me test it out before that.

[pvlkov]

Apart from my comment above, the only thing I noticed while testing this in a GCP Kubernetes cluster was, that the Dockerfile does not produce a working container. You are not copying the src/storage directory into the final runtime-image container and the storage_factory.py file is also missing, which makes the imports fail.

opencost-parquet-exporter/Dockerfile

Line 12 in fa0df7b

COPY src/opencost_parquet_exporter.py /app/opencost_parquet_exporter.py

I would suggest replacing the copy directives to a single COPY src/ /app/. Also you might consider moving the storage_factory into the storage module as well.

With these changes I managed to get a running container in a GCP cluster which was able to query the Opencost endpoint and upload the result to a GCS bucket using a Kubernetes service account with workload identity.

I have not tested specifying OPENCOST_PARQUET_GCP_CREDENTIALS, neither have I tested using the default GOOGLE_APPLICATION_CREDENTIALS environment. I might get around to doing that at some later point but I cannot promise that.

[pvlkov]

Update: Tested from a non-GCP cluster with OPENCOST_PARQUET_CREDENTIALS_JSON and it also works.

[lmello]

Apart from my comment above, the only thing I noticed while testing this in a GCP Kubernetes cluster was, that the Dockerfile does not produce a working container. You are not copying the src/storage directory into the final runtime-image container and the storage_factory.py file is also missing, which makes the imports fail.

opencost-parquet-exporter/Dockerfile

Line 12 in fa0df7b

COPY src/opencost_parquet_exporter.py /app/opencost_parquet_exporter.py

I would suggest replacing the copy directives to a single COPY src/ /app/. Also you might consider moving the storage_factory into the storage module as well.

With these changes I managed to get a running container in a GCP cluster which was able to query the Opencost endpoint and upload the result to a GCS bucket using a Kubernetes service account with workload identity.

I have not tested specifying OPENCOST_PARQUET_GCP_CREDENTIALS, neither have I tested using the default GOOGLE_APPLICATION_CREDENTIALS environment. I might get around to doing that at some later point but I cannot promise that.

@pvlkov thanks for testing and providing feedback. we fixed the docker error on the main branch. I am going to rebase this changes on it so the docker would be fixed.

[lmello]

r with OPENCOST_PARQUET_CREDENTIALS_JSON and it also works.

Thank you.