Merge branch 'develop' into feature/vertex-ai-deployer-model-registry

.gitbook.yaml

@@ -18,6 +18,11 @@ redirects:
   how-to/setting-up-a-project-repository/best-practices: how-to/project-setup-and-management/setting-up-a-project-repository/set-up-repository.md
   getting-started/zenml-pro/system-architectures: getting-started/system-architectures.md
   how-to/build-pipelines/name-your-pipeline-and-runs: how-to/pipeline-development/build-pipelines/name-your-pipeline-runs.md
+  how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-steps: how-to/model-management-metrics/track-metrics-metadata/attach-metadata-to-a-step.md
+
+  # ZenML Pro
+  getting-started/zenml-pro/user-management: getting-started/zenml-pro/core-concepts.md
+  getting-started/zenml-pro/zenml-cloud: getting-started/zenml-pro/README.md
 
   # Project Setup redirects
   how-to/setting-up-a-project-repository/: how-to/project-setup-and-management/setting-up-a-project-repository/README.md
@@ -47,7 +52,8 @@ redirects:
   how-to/build-pipelines/schedule-a-pipeline: how-to/pipeline-development/build-pipelines/schedule-a-pipeline.md
   how-to/build-pipelines/delete-a-pipeline: how-to/pipeline-development/build-pipelines/delete-a-pipeline.md
   how-to/build-pipelines/compose-pipelines: how-to/pipeline-development/build-pipelines/compose-pipelines.md
-  how-to/build-pipelines/dynamically-assign-artifact-names: how-to/pipeline-development/build-pipelines/dynamically-assign-artifact-names.md
+  how-to/build-pipelines/dynamically-assign-artifact-names: how-to/data-artifact-management/handle-data-artifacts/artifacts-naming.md
+  how-to/pipeline-development/build-pipelines/dynamically-assign-artifact-names: how-to/data-artifact-management/handle-data-artifacts/artifacts-naming.md
   how-to/build-pipelines/retry-steps: how-to/pipeline-development/build-pipelines/retry-steps.md
   how-to/build-pipelines/run-pipelines-asynchronously: how-to/pipeline-development/build-pipelines/run-pipelines-asynchronously.md
   how-to/build-pipelines/control-execution-order-of-steps: how-to/pipeline-development/build-pipelines/control-execution-order-of-steps.md
@@ -172,4 +178,27 @@ redirects:
   how-to/control-logging/enable-or-disable-logs-storing: how-to/advanced-topics/control-logging/enable-or-disable-logs-storing.md
   how-to/control-logging/set-logging-verbosity: how-to/advanced-topics/control-logging/set-logging-verbosity.md
   how-to/control-logging/disable-rich-traceback: how-to/advanced-topics/control-logging/disable-rich-traceback.md
-  how-to/control-logging/disable-colorful-logging: how-to/advanced-topics/control-logging/disable-colorful-logging.md
+  how-to/control-logging/disable-colorful-logging: how-to/advanced-topics/control-logging/disable-colorful-logging.md
+
+  # Redirects for cases that have `docs` in the path
+  docs/getting-started/zenml-pro/core-concepts/tenants: getting-started/zenml-pro/tenants.md
+  docs/introduction: introduction.md
+  docs/getting-started/installation: getting-started/installation.md
+  docs/getting-started/core-concepts: getting-started/core-concepts.md
+  docs/getting-started/system-architectures: getting-started/system-architectures.md
+  docs/getting-started/deploying-zenml/: getting-started/deploying-zenml/README.md
+  docs/getting-started/zenml-pro/: getting-started/zenml-pro/README.md
+  docs/user-guide/starter-guide/: user-guide/starter-guide/README.md
+  docs/user-guide/production-guide/: user-guide/production-guide/README.md
+  docs/user-guide/llmops-guide/: user-guide/llmops-guide/README.md
+  docs/how-to/interact-with-secrets: how-to/interact-with-secrets.md
+  docs/how-to/debug-and-solve-issues: how-to/debug-and-solve-issues.md
+  docs/how-to/contribute-to-zenml/: how-to/contribute-to-zenml/README.md
+  docs/component-guide/: component-guide/README.md
+  docs/reference/python-client: reference/python-client.md
+  docs/reference/global-settings: reference/global-settings.md
+  docs/reference/environment-variables: reference/environment-variables.md
+  docs/reference/api-reference: reference/api-reference.md
+  docs/reference/how-do-i: reference/how-do-i.md
+  docs/reference/community-and-content: reference/community-and-content.md
+  docs/reference/faq: reference/faq.md

.github/pull_request_template.md

@@ -4,10 +4,13 @@ I implemented/fixed _ to achieve _.
 ## Pre-requisites
 Please ensure you have done the following:
 - [ ] I have read the **CONTRIBUTING.md** document.
-- [ ] If my change requires a change to docs, I have updated the documentation accordingly.
 - [ ] I have added tests to cover my changes.
 - [ ] I have based my new branch on `develop` and the open PR is targeting `develop`. If your branch wasn't based on develop read [Contribution guide on rebasing branch to develop](https://github.com/zenml-io/zenml/blob/main/CONTRIBUTING.md#-pull-requests-rebase-your-branch-on-develop).
-- [ ] If my changes require changes to the dashboard, these changes are communicated/requested.
+- [ ] **IMPORTANT**: I made sure that my changes are reflected properly in the following resources:
+  - [ ] [ZenML Docs](https://docs.zenml.io)
+  - [ ] Dashboard: Needs to be communicated to the frontend team.
+  - [ ] Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
+  - [ ] [Projects](https://github.com/zenml-io/zenml-projects): Depending on the version dependencies, different projects might get affected.
 
 ## Types of changes
 <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->

.github/workflows/ci-slow.yml

@@ -243,12 +243,6 @@ jobs:
       matrix:
         os: [macos-latest]
         python-version: ['3.9', '3.10', '3.11', '3.12']
-        # Python 3.9 is on macos-13 but not macos-latest (macos-14-arm64)
-        # https://github.com/actions/setup-python/issues/696#issuecomment-1637587760
-        exclude:
-          - {python-version: '3.9', os: macos-latest}
-        include:
-          - {python-version: '3.9', os: macos-13}
       fail-fast: false
     uses: ./.github/workflows/unit-test.yml
     with:
@@ -275,14 +269,8 @@ jobs:
     needs: [run-slow-ci-label-is-set, macos-unit-test]
     strategy:
       matrix:
-        os: [macos-13]
+        os: [macos-latest]
         python-version: ['3.9', '3.10', '3.11', '3.12']
-        # Python 3.9 is on macos-13 but not macos-latest (macos-14-arm64)
-        # https://github.com/actions/setup-python/issues/696#issuecomment-1637587760
-        exclude:
-          - {python-version: '3.9', os: macos-latest}
-        include:
-          - {python-version: '3.9', os: macos-13}
         test_environment: [default]
       fail-fast: false
     uses: ./.github/workflows/integration-test-slow.yml

.github/workflows/integration-test-slow.yml

@@ -95,8 +95,7 @@ jobs:
       GCP_US_EAST4_SERVER_USERNAME: ${{ secrets.GCP_US_EAST4_SERVER_USERNAME }}
       GCP_US_EAST4_SERVER_PASSWORD: ${{ secrets.GCP_US_EAST4_SERVER_PASSWORD }}
     # TODO: add Windows testing for Python 3.11 and 3.12 back in
-    # TODO: add macos testing back in
-    if: ${{ ! startsWith(github.event.head_commit.message, 'GitBook:') && ! (inputs.os == 'windows-latest' && inputs.python-version == '3.11') && ! (inputs.os == 'windows-latest' && inputs.python-version == '3.12') && ! (inputs.os == 'macos-13' || inputs.os == 'macos-latest')  }}
+    if: ${{ ! startsWith(github.event.head_commit.message, 'GitBook:') && ! (inputs.os == 'windows-latest' && inputs.python-version == '3.11') && ! (inputs.os == 'windows-latest' && inputs.python-version == '3.12') }}
     defaults:
       run:
         shell: bash
@@ -162,37 +161,6 @@ jobs:
       - name: Install MacOS System Dependencies
         if: runner.os=='macOS'
         run: brew install libomp
-      - name: Unbreak Python in GHA for 3.9-3.10
-        if: runner.os=='macOS' && inputs.python-version != '3.11'
-        # github actions overwrites brew's python. Force it to reassert itself, by
-        # running in a separate step.
-        # Workaround GitHub Actions Python issues
-        # see https://github.com/Homebrew/homebrew-core/issues/165793#issuecomment-1989441193
-        run: |
-          find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
-          sudo rm -rf /Library/Frameworks/Python.framework/
-          brew install --force python3 && brew unlink python3 && brew unlink python3 && brew link --overwrite python3
-      - name: Unbreak Python in GHA for 3.11
-        if: runner.os=='macOS' && inputs.python-version == '3.11'
-        run: |
-          # Unlink and re-link to prevent errors when github mac runner images
-          # https://github.com/actions/setup-python/issues/577
-          brew list -1 | grep python | while read formula; do brew unlink $formula; brew link --overwrite $formula; done
-      - name: Install Docker and Colima on MacOS
-        if: runner.os=='macOS'
-        run: |
-          export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=1
-          brew update
-          brew install docker colima
-          brew reinstall --force qemu
-
-          # We need to mount the /private/tmp/zenml-test/ folder because
-          # this folder is also mounted in the Docker containers that are
-          # started by local ZenML orchestrators.
-          colima start --mount /private/tmp/zenml-test/:w
-
-          # This is required for the Docker Python SDK to work
-          sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock
       - name: Install kubectl on Linux
         run: |
           curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

.github/workflows/update-templates-to-examples.yml

@@ -46,7 +46,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
           stack-name: local
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.10.30  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout
@@ -118,7 +118,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
           stack-name: local
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.10.30  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout
@@ -189,7 +189,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
           stack-name: local
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.10.30  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout
@@ -261,7 +261,7 @@ jobs:
         with:
           python-version: ${{ inputs.python-version }}
           ref-zenml: ${{ github.ref }}
-          ref-template: 2024.11.08  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
+          ref-template: 2024.11.28  # Make sure it is aligned with ZENML_PROJECT_TEMPLATES from src/zenml/cli/base.py
       - name: Clean-up
         run: |
           rm -rf ./local_checkout

docs/book/component-guide/README.md

@@ -70,9 +70,13 @@ You can run `zenml integration --help` to see a full list of CLI commands that Z
 
 Note, that you can also install your dependencies directly, but please note that there is no guarantee that ZenML internals with work with any arbitrary version of any external library.
 
-#### Experimental: Use `uv` for package installation
+#### Use `uv` for package installation
 
-You can use [`uv`](https://github.com/astral-sh/uv) as a package manager if you want. Simply pass the `--uv` flag to the `zenml integration ...` command and it'll use `uv` for installation, upgrades and uninstallations. Note that `uv` must be installed for this to work. This is an experimental option that we've added for users wishing to use `uv` but given that it is relatively new as an option there might be certain packages that don't work well with `uv`. We will monitor how this performs and update as `uv` becomes more stable.
+You can use [`uv`](https://github.com/astral-sh/uv) as a package manager if you want. Simply pass the `--uv` flag to the `zenml integration ...` command and it'll use `uv` for installation, upgrades and uninstalls. Note that `uv` must be installed for this to work. This is an experimental option that we've added for users wishing to use `uv` but given that it is relatively new as an option there might be certain packages that don't work well with `uv`.
+
+Full documentation for how it works with PyTorch can be found on Astral's docs
+website [here](https://docs.astral.sh/uv/guides/integration/pytorch/). It covers
+some of the particular gotchas and details you might need to know.
 
 ### Upgrade ZenML integrations