From 54fd52f9d9ea4631d085df33a7dccb2eb41d47d6 Mon Sep 17 00:00:00 2001 From: Maxime Armstrong Date: Mon, 5 Aug 2024 13:19:55 -0400 Subject: [PATCH] Update dbt source post review --- .../3-representing-the-dbt-project-in-dagster.md | 8 +++++--- .../5-loading-dbt-models-into-dagster-as-assets.md | 14 +++++--------- .../2-speeding-up-the-development-cycle.md | 6 +++--- .../4-creating-the-manifest-during-deployment.md | 2 +- .../lesson-7/5-preparing-for-a-successful-run.md | 4 ++-- 5 files changed, 16 insertions(+), 18 deletions(-) diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md b/docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md index f08044d22f673..3cd80ce31fba1 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md @@ -8,7 +8,7 @@ lesson: '3' As you’ll frequently point your Dagster code to the `target/manifest.json` file and your dbt project in this course, it’ll be helpful to keep a reusable representation of the dbt project. This can be easily done using the `DbtProject` class. -Create a new file in the `assets` directory called `dbt.py`. Open that file and add the following import at the top: +In the `assets` directory, create a new `dbt.py` file and add the following imports: ```python from pathlib import Path @@ -16,7 +16,9 @@ from pathlib import Path from dagster_dbt import DbtProject ``` -The `Path` class from the `pathlib` standard library will help us create an accurate pointer to where our dbt project is. The `DbtProject` class is imported from the `dagster_dbt` package that we installed earlier. After the import, add the following code: +The `Path` class from the `pathlib` standard library will help us create an accurate pointer to where our dbt project is. The `DbtProject` class is imported from the `dagster_dbt` package that we installed earlier. + +After the import, add the following code: ```python dbt_project = DbtProject( @@ -26,7 +28,7 @@ dbt_project = DbtProject( This code creates a representation of the dbt project called `dbt_project`. The code defining the location of the project directory might look a little complicated, so let’s break it down: -- It uses the location of the `dbt.py` file (via `__file__`) as a point of reference for finding the dbt project +- The location of the `dbt.py` file (via `__file__`) is used as a point of reference for finding the dbt project - The arguments in `joinpath` point us towards our dbt project by appending the following to the current path: - Three directory levels up (`"..", "..", ".."`) - A directory named `analytics`, which is the directory containing our dbt project diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-3/5-loading-dbt-models-into-dagster-as-assets.md b/docs/dagster-university/pages/dagster-dbt/lesson-3/5-loading-dbt-models-into-dagster-as-assets.md index fd9f571e69551..bfb1f7bff20c7 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-3/5-loading-dbt-models-into-dagster-as-assets.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-3/5-loading-dbt-models-into-dagster-as-assets.md @@ -28,7 +28,7 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson, ## Loading the models as assets -1. Navigate to the `assets/dbt.py` created earlier and open it. +1. Open the `assets/dbt.py` file. 2. Add the following imports to the top of the file: @@ -37,13 +37,7 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson, from dagster_dbt import dbt_assets, DbtCliResource ``` -3. The `@dbt_assets` decorator requires a path to the project’s manifest file, which is within our `dbt_project` representation created earlier. The manifest path can be easily accessed with the code below: - - ```python - dbt_project.manifest_path - ``` - -4. Now, use the `@dbt_assets` decorator to create a new asset function and provide it with a reference to the manifest: +3. Next, we'll use the `@dbt_assets` decorator to create a new asset function and provide it with a reference to the project's manifest file: ```python @dbt_assets( @@ -52,7 +46,9 @@ We’ll only create one `@dbt_assets` definition for now, but in a later lesson, def dbt_analytics(context: AssetExecutionContext, dbt: DbtCliResource): ``` -5. Finally, add the following to the body of `dbt_analytics` function: + Here, we used `dbt_project.manifest_path` to provide the reference to the project's manifest file. This is possible because the `dbt_project` representation we created earlier contains the manifest path, accessible by using the `manifest_path` attribute. + +4. Finally, add the following to the body of `dbt_analytics` function: ```python yield from dbt.cli(["run"], context=context).stream() diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-4/2-speeding-up-the-development-cycle.md b/docs/dagster-university/pages/dagster-dbt/lesson-4/2-speeding-up-the-development-cycle.md index dd9d6cdd8f303..035ec6f26f5ff 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-4/2-speeding-up-the-development-cycle.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-4/2-speeding-up-the-development-cycle.md @@ -8,7 +8,7 @@ lesson: '4' By now, you’ve had to run `dbt parse` to create the manifest file and reload your code location quite frequently, which doesn’t feel like the cleanest developer experience. -Before we move on, we’ll reduce the number of steps in the feedback loop. We'll automate the creation the manifest file by taking advantage of the `DbtProject` that we wrote earlier. +Before we move on, we’ll reduce the number of steps in the feedback loop. We'll automate the creation of the manifest file by taking advantage of the `dbt_project` representation that we wrote earlier. --- @@ -22,7 +22,7 @@ In `dbt.py`, after the code initializing `dbt_project`, add the following code: dbt_project.prepare_if_dev() ``` -If you look at the dbt project’s `/target` directory, you’ll see it stores the artifacts. When reloading your code in local development, when using `dagster_dev`, you'll see that a new manifest file is generated. +If you look at the dbt project’s `/target` directory, you’ll see it stores the artifacts. When you use `dagster dev` in local development and you reload your code, you'll see that a new manifest file is generated. Reload your code location in the Dagster UI, and you’ll see that everything should still work: the dbt models are still shown as assets and you can manually materialize any of the models. The key difference is that you no longer have to manually run `dbt parse` anymore! @@ -30,6 +30,6 @@ Reload your code location in the Dagster UI, and you’ll see that everything sh ## Creating the manifest for production -This is great, however, it only handles the preparation of a new manifest file in local development. In production, where a dbt project is stable, we may want to prepare a new manifest file only at build time, during the deployment process. This can be done using the command line interface (CLI) available in the `dagster_dbt` package installed earlier. +This is great, however, it only handles the preparation of a new manifest file in local development. In production, where a dbt project is stable, we may want to prepare a new manifest file only at build time, during the deployment process. This can be done using the command line interface (CLI) available in the `dagster_dbt` package. Don't worry about the details for now! In Lesson 7, we’ll discuss the details on how to create a manifest file programmatically during deployment using the `dagster_dbt` CLI. diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md b/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md index 4db9709dde474..60378031a1051 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md @@ -12,7 +12,7 @@ Building your manifest for your production deployment will be needed for both op Since your CI/CD will be running in a fresh environment, you'll need to install dbt and other dependencies before building your manifest. -To get our deployment working, we need to add a step to our GitHub Actions workflow that runs the commands required to generate the `manifest.json`. Specifically, we need to run the `dbt project prepare-and-package` command, available in the `dagster_dbt` package discussed earlier. +To get our deployment working, we need to add a step to our GitHub Actions workflow that runs the commands required to generate the `manifest.json`. Specifically, we need to run the `dbt project prepare-and-package` command, available in the `dagster_dbt` package. 1. In your Dagster project, locate the `.github/workflows` directory. 2. Open the `deploy.yml` file. diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-7/5-preparing-for-a-successful-run.md b/docs/dagster-university/pages/dagster-dbt/lesson-7/5-preparing-for-a-successful-run.md index 886e617ea839b..b544eef9bee74 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-7/5-preparing-for-a-successful-run.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-7/5-preparing-for-a-successful-run.md @@ -90,9 +90,9 @@ dbt_project = DbtProject( ## Adding a prod target to deploy.yml -After that, we need to update the dbt commands in the `.github/workflows/deploy.yml` file to target the new `prod` profile. This will ensure that dbt uses the correct connection details when the GitHub Action runs as part of our Dagster+ deployment. +Next, we need to update the dbt commands in the `.github/workflows/deploy.yml` file to target the new `prod` profile. This will ensure that dbt uses the correct connection details when the GitHub Action runs as part of our Dagster+ deployment. -Open the file, scroll to the environment variable sections, and set an environment variable named `DBT_TARGET` to `prod`. This command should be on or around line 12: +Open the file, scroll to the environment variable section, and set an environment variable named `DBT_TARGET` to `prod`. This should be on or around line 12: ```bash env: