WIP: Add GCP-Prefect recipe

[nicholasjng]

Content TBD, after more clear definition of what needs to be implemented and how.

I wanted to hack away at this in the coming week, since the time is now opportune. Because I am a complete terraform n00b, I might need some handholding during the implementation.

I touched base with @htahir1 on Slack about this some time ago, he thinks adding this is worthwhile (I hope he still does :^D).

Describe changes

My goal is to implement an mlstacks recipe for prefect on GCP.

The original Slack message:

Hey guys, I am looking to collaborate on a prefect-gcp recipe for mlstacks (https://github.com/zenml-io/mlstacks).

This is for a personal use-case that I am working on right now, but I think it is general enough that it might benefit others in the ZenML community as well.

Judging from the docs, the code and my experience on Prefect so far, I think the following might be a good solution to achieve this:

• Making a VM in GCP for the Prefect Server instance, with access rules, ports, security etc. set up.
• Connecting it to a Cloud SQL Postgres instance (which should also be created if not already there), for improving durability and flexibility by decoupling server and database.
• (optional) Setting up a Cloud logging bucket for the server logs - so far I only used it for client/agent logs, but might be useful.

If you are interested, let me know, of course I am happy for feedback and general discussions on the proposal.
-Nico

Link to the Slack fred: https://zenml.slack.com/archives/C01FWQ5D0TT/p1693740985687089

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation
  accordingly.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting
  develop. If your branch wasn't based on develop read
  Contribution guide on rebasing branch to develop.

Types of changes

• New feature (non-breaking change which adds functionality)

[strickvl]

Thank you for this addition / contribution! Could you rebase + set the base branch for this PR to develop? It's explained in the instructions on the PR description. Thank you!

[nicholasjng]

Rebased on develop.

EDIT: Great timing 👉😎👉

[strickvl]

Thanks, and look forward to seeing where this one goes! Let us know here or on Slack how we can support!

[nicholasjng]

Thanks, and look forward to seeing where this one goes! Let us know here or on Slack how we can support!

I linked the existing Slack thread in the PR comment - let's start by creating the VM. I think you got a TF module for that already, can I just borrow the config from an existing recipe?

The next issue is how to run a script in the VM that sets up Python and the prefect server on that VM (raw? in Docker?).

[strickvl]

@nicholasjng our TF modules (as it stands right now) are for creating a remote state bucket, and that happens independent of your modular prefect recipe, so you can ignore that. Also I think you're probably best off just working in terraform independently of mlstacks to start with (i.e. just work from within the gcp-modular recipe and use terraform there). Depending on how much needs changing, you might want to create a whole separate gcp-modular-vm folder / base within the src/mlstacks/terraform etc, but in any case I'd start working just to develop everything out the way you want, and then to fit it into mlstacks.

Note, for example, that our gcp-modular recipe is K8S based, and while we (theoretically) allow non-K8S deployments, we don't actually have any yet, so we'll probably want a way of distinguishing that within the recipes themselves.

[nicholasjng]

@strickvl Okay, that's a start. I checked the link to the TF registry from the Slack thread, seems like a good lead.

For clarification: I intended to make this usable standalone without k8s, hope that's not an issue. Will check back in when I have a small Compute Engine VM poc.

[strickvl]

@nicholasjng no problem at all! it's going to be a focus of ours going forward to provide more non-K8S options, so this is a welcome contribution in that sense!

[coderabbitai]

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

---

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit-tests for this file.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit tests for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository from git and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit tests.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• The JSON schema for the configuration file is available here.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

[strickvl]

Hi @nicholasjng. Just wanted to check in. Any way we can help support your work on this?

[nicholasjng]

Hey, I got my hands full with another project until at least mid next week. Happy to give it another go after that! (Or worst case I'll hand it off to my colleagues who are deeper into mlstacks than I currently am)