diff --git a/docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md b/docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md index cc59bd13364a9..6b88b8104a216 100644 --- a/docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md +++ b/docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md @@ -1,6 +1,393 @@ --- title: Managing multiple projects and teams -unlisted: true --- -{/* TODO move from https://docs.dagster.io/dagster-plus/best-practices/managing-multiple-projects-and-teams */} \ No newline at end of file +In this guide, we'll cover some strategies for managing multiple projects/code bases and teams in a Dagster+ account. + +## Separating code bases + +:::note +In this section, repository refers to a version control system, such as Git or Mercurial. +::: + +If you want to manage complexity or divide your work into areas of responsibility, consider isolating your code bases into multiple projects with: + +- Multiple directories in a single repository, or +- Multiple repositories + +Refer to the following table for more information, including the pros and cons of each approach. + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Approach + + Multiple directories in a single repository + Multiple repositories
+ How it works + + You can use a single repository to manage multiple projects by placing + each project in a separate directory. Depending on your VCS, you may be + able to set{" "} + + code owners + {" "} + to restrict who can modify each project. + + For stronger isolation, you can use multiple repositories to manage + multiple projects. +
+ Pros + +
    +
  • + Simple to implement +
    • +
  • Facilitates code sharing between projects
    • +
+ 		+
    +
  • + Stronger isolation between projects and teams +
    • +
  • + Each project has its own CI/CD pipeline and be deployed + independently +
    • +
  • Dependencies between projects can be managed independently
    • +
+
+ Cons + +
    +
  • + All projects share the same CI/CD pipeline and cannot be deployed + independently +
    • +
  • + Shared dependencies between projects may cause conflicts and require + coordination between teams +
    • +
+ 		+ Code sharing between projects require additional coordination to publish + and reuse packages between projects +
+ +### Deployment configuration + +{/* Whether you use a single repository or multiple, you can use a [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml) to define the code locations to deploy. For each repository, follow the [steps appropriate to your CI/CD provider](/dagster-plus/getting-started#step-4-configure-cicd-for-your-project) and include only the code locations that are relevant to the repository in your CI/CD workflow. */} +Whether you use a single repository or multiple, you can use a [`dagster_cloud.yaml` file](/todo) to define the code locations to deploy. For each repository, follow the [steps appropriate to your CI/CD provider](/todo) and include only the code locations that are relevant to the repository in your CI/CD workflow. + +#### Example with GitHub CI/CD on Hybrid deployment + +1. **For each repository**, use the CI/CD workflow provided in [Dagster+ Hybrid quickstart repository](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/main/.github/workflows/dagster-cloud-deploy.yml). + +{/* 2. **For each project in the repository**, configure a code location in the [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml): */} +2. **For each project in the repository**, configure a code location in the [`dagster_cloud.yaml` file](/todo): + + ```yaml + # dagster_cloud.yml + + locations: + - location_name: project_a + code_source: + package_name: project_a + build: + # ... + - location_name: project_b + code_source: + package_name: project_b + build: + # ... + ``` + +3. In the repository's `dagster-cloud-deploy.yml` file, modify the CI/CD workflow to deploy all code locations for the repository: + + ```yaml + # .github/workflows/dagster-cloud-deploy.yml + + jobs: + dagster-cloud-deploy: + # ... + steps: + - name: Update build session with image tag for "project_a" code location + id: ci-set-build-output-project-a + if: steps.prerun.outputs.result != 'skip' + uses: dagster-io/dagster-cloud-action/actions/utils/dagster-cloud-cli@v0.1 + with: + command: "ci set-build-output --location-name=project_a --image-tag=$IMAGE_TAG" + + - name: Update build session with image tag for "project_b" code location + id: ci-set-build-output-project-b + if: steps.prerun.outputs.result != 'skip' + uses: dagster-io/dagster-cloud-action/actions/utils/dagster-cloud-cli@v0.1 + with: + command: "ci set-build-output --location-name=project_b --image-tag=$IMAGE_TAG" + # ... + ``` + +--- + +## Isolating execution context between projects + +Separating execution context between projects can have several motivations: + +- Facilitating separation of duty between teams to prevent access to sensitive data +- Differing compute environments and requirements, such as different architecture, cloud provider, etc. +- Reducing impact on other projects. For example, a project with a large number of runs can impact the performance of other projects. + +In order from least to most isolated, there are three levels of isolation: + +- [Code location](#code-location-isolation) +- [Agent](#agent-isolation) +- [Deployment](#deployment-isolation) + +### Code location isolation + +If you have no specific requirements for isolation beyond the ability to deploy and run multiple projects, you can use a single agent and deployment to manage all your projects as individual code locations. + +![Diagram of isolation at the code location level](/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png) + + + + + + + + + + + + + + +
+ Pros + Cons
+
    +
  • + Simplest and most cost-effective solution +
    • +
  • User access control can be set at the code location level
    • +
  • Single glass pane to view all assets
    • +
+ 		+
    +
  • + No isolation between execution environments +
    • +
+
+ +### Agent isolation + +:::note +Agent queues are a Dagster+ Pro feature available on hybrid deployment. +::: + +{/* Using the [agent routing feature](/dagster-plus/deployment/agents/running-multiple-agents#routing-requests-to-specific-agents), you can effectively isolate execution environments between projects by using a separate agent for each project. */} +Using the [agent routing feature](/todo), you can effectively isolate execution environments between projects by using a separate agent for each project. + +Motivations for utilizing this approach could include: + +- Different compute requirements, such as different cloud providers or architectures +- Optimizing for locality or access, such as running the data processing closer or in environment with access to the storage locations + +![Diagram of isolation at the agent level](/images/dagster-cloud/managing-deployments/isolation-level-agents.png) + + + + + + + + + + + + + + +
+ Pros + Cons
+
    +
  • + Isolation between execution environments +
    • +
  • User access control can be set at the code location level
    • +
  • Single glass pane to view all assets
    • +
+ 		Extra work to set up additional agents and agent queues
+ +### Deployment isolation + +:::note +Multiple deployments are only available in Dagster+ Pro. +::: + +Of the approaches outlined in this guide, multiple deployments are the most isolated solution. The typical motivation for this isolation level is to separate production and non-production environments. It may be considered to satisfy other organization specific requirements. + +![Diagram of isolation at the Dagster+ deployment level](/images/dagster-cloud/managing-deployments/isolation-level-deployments.png) + + + + + + + + + + + + + + +
+ Pros + Cons
+
    +
  • + Isolation between assets and execution environments +
    • +
  • + User access control can be set at the code location and deployment + level +
    • +
+ 		+ No single glass pane to view all assets (requires switching between + multiple deployments in the UI) +
+ + +{/* +## Related + +- [Dagster+ Hybrid deployments](/dagster-plus/deployment/hybrid) +- [Dagster+ Hybrid agents](/dagster-plus/deployment/agents) +- [Managing deployments in Dagster+](/dagster-plus/managing-deployments) +(/* - [Running multiple Dagster+ Hybrid agents](/dagster-plus/deployment/agents/running-multiple-agents#routing-requests-to-specific-agents) */} +- [Running multiple Dagster+ Hybrid agents](/todo) +{/* - [dagster_cloud.yaml](/dagster-plus/managing-deployments/dagster-cloud-yaml) */} +- [dagster_cloud.yaml](/todo) diff --git a/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-agents.png b/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-agents.png new file mode 100644 index 0000000000000..2fd662c04df8a Binary files /dev/null and b/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-agents.png differ diff --git a/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png b/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png new file mode 100644 index 0000000000000..558b74df3d244 Binary files /dev/null and b/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png differ diff --git a/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-deployments.png b/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-deployments.png new file mode 100644 index 0000000000000..27cc815b9bf44 Binary files /dev/null and b/docs/docs-beta/static/images/dagster-cloud/managing-deployments/isolation-level-deployments.png differ