Skip to content

Commit

Permalink
[docs-beta] migrate - managing multiple projects and teams doc (#26501)
Browse files Browse the repository at this point in the history 
## Summary & Motivation

Broken links marked `/todo`

```
  - Broken link on source page path = /dagster-plus/deployment/management/managing-multiple-projects-and-teams:
     -> linking to /dagster-plus/managing-deployments/dagster-cloud-yaml
     -> linking to /dagster-plus/deployment/agents/running-multiple-agents#routing-requests-to-specific-agents
```

## How I Tested These Changes

## Changelog

NOCHANGELOG
  • Loading branch information
@cmpadden
cmpadden authored Dec 19, 2024
1 parent 68fb4d8 commit 5a4dc5d
Show file tree
Hide file tree
Showing 4 changed files with 389 additions and 2 deletions.
391 changes: 389 additions & 2 deletions ...docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,393 @@
---
title: Managing multiple projects and teams
unlisted: true
---

{/* TODO move from https://docs.dagster.io/dagster-plus/best-practices/managing-multiple-projects-and-teams */}
In this guide, we'll cover some strategies for managing multiple projects/code bases and teams in a Dagster+ account.

## Separating code bases

:::note
In this section, repository refers to a version control system, such as Git or Mercurial.
:::

If you want to manage complexity or divide your work into areas of responsibility, consider isolating your code bases into multiple projects with:

- Multiple directories in a single repository, or
- Multiple repositories

Refer to the following table for more information, including the pros and cons of each approach.

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "14%",
}}
>
Approach
</th>
<th
style={{
width: "43%",
}}
>
Multiple directories in a single repository
</th>
<th>Multiple repositories</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<strong>How it works</strong>
</td>
<td>
You can use a single repository to manage multiple projects by placing
each project in a separate directory. Depending on your VCS, you may be
able to set{" "}
<a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners">

Check warning on line 54 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md

View workflow job for this annotation

GitHub Actions / deploy 

Do not use an `<a>` element to navigate. Use the `<Link />` component from `@docusaurus/Link` instead. See: https://docusaurus.io/docs/docusaurus-core#link
code owners
</a>{" "}
to restrict who can modify each project.
</td>
<td>
For stronger isolation, you can use multiple repositories to manage
multiple projects.
</td>
</tr>
<tr>
<td>
<strong>Pros</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Simple to implement
</li>
<li>Facilitates code sharing between projects</li>
</ul>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Stronger isolation between projects and teams
</li>
<li>
Each project has its own CI/CD pipeline and be deployed
independently
</li>
<li>Dependencies between projects can be managed independently</li>
</ul>
</td>
</tr>
<tr>
<td>
<strong>Cons</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
All projects share the same CI/CD pipeline and cannot be deployed
independently
</li>
<li>
Shared dependencies between projects may cause conflicts and require
coordination between teams
</li>
</ul>
</td>
<td>
Code sharing between projects require additional coordination to publish
and reuse packages between projects
</td>
</tr>
</tbody>
</table>

### Deployment configuration

{/* Whether you use a single repository or multiple, you can use a [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml) to define the code locations to deploy. For each repository, follow the [steps appropriate to your CI/CD provider](/dagster-plus/getting-started#step-4-configure-cicd-for-your-project) and include only the code locations that are relevant to the repository in your CI/CD workflow. */}
Whether you use a single repository or multiple, you can use a [`dagster_cloud.yaml` file](/todo) to define the code locations to deploy. For each repository, follow the [steps appropriate to your CI/CD provider](/todo) and include only the code locations that are relevant to the repository in your CI/CD workflow.

#### Example with GitHub CI/CD on Hybrid deployment

1. **For each repository**, use the CI/CD workflow provided in [Dagster+ Hybrid quickstart repository](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/main/.github/workflows/dagster-cloud-deploy.yml).

{/* 2. **For each project in the repository**, configure a code location in the [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml): */}
2. **For each project in the repository**, configure a code location in the [`dagster_cloud.yaml` file](/todo):

```yaml
# dagster_cloud.yml

locations:
- location_name: project_a
code_source:
package_name: project_a
build:
# ...
- location_name: project_b
code_source:
package_name: project_b
build:
# ...
```

3. In the repository's `dagster-cloud-deploy.yml` file, modify the CI/CD workflow to deploy all code locations for the repository:

```yaml
# .github/workflows/dagster-cloud-deploy.yml

jobs:
dagster-cloud-deploy:
# ...
steps:
- name: Update build session with image tag for "project_a" code location
id: ci-set-build-output-project-a
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci set-build-output --location-name=project_a --image-tag=$IMAGE_TAG"

- name: Update build session with image tag for "project_b" code location
id: ci-set-build-output-project-b
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci set-build-output --location-name=project_b --image-tag=$IMAGE_TAG"
# ...
```

---

## Isolating execution context between projects

Separating execution context between projects can have several motivations:

- Facilitating separation of duty between teams to prevent access to sensitive data
- Differing compute environments and requirements, such as different architecture, cloud provider, etc.
- Reducing impact on other projects. For example, a project with a large number of runs can impact the performance of other projects.

In order from least to most isolated, there are three levels of isolation:

- [Code location](#code-location-isolation)
- [Agent](#agent-isolation)
- [Deployment](#deployment-isolation)

### Code location isolation

If you have no specific requirements for isolation beyond the ability to deploy and run multiple projects, you can use a single agent and deployment to manage all your projects as individual code locations.

![Diagram of isolation at the code location level](/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Simplest and most cost-effective solution
</li>
<li>User access control can be set at the code location level</li>
<li>Single glass pane to view all assets</li>
</ul>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
No isolation between execution environments
</li>
</ul>
</td>
</tr>
</tbody>
</table>

### Agent isolation

:::note
Agent queues are a Dagster+ Pro feature available on hybrid deployment.
:::

{/* Using the [agent routing feature](/dagster-plus/deployment/agents/running-multiple-agents#routing-requests-to-specific-agents), you can effectively isolate execution environments between projects by using a separate agent for each project. */}
Using the [agent routing feature](/todo), you can effectively isolate execution environments between projects by using a separate agent for each project.

Motivations for utilizing this approach could include:

- Different compute requirements, such as different cloud providers or architectures
- Optimizing for locality or access, such as running the data processing closer or in environment with access to the storage locations

![Diagram of isolation at the agent level](/images/dagster-cloud/managing-deployments/isolation-level-agents.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Isolation between execution environments
</li>
<li>User access control can be set at the code location level</li>
<li>Single glass pane to view all assets</li>
</ul>
</td>
<td>Extra work to set up additional agents and agent queues</td>
</tr>
</tbody>
</table>

### Deployment isolation

:::note
Multiple deployments are only available in Dagster+ Pro.
:::

Of the approaches outlined in this guide, multiple deployments are the most isolated solution. The typical motivation for this isolation level is to separate production and non-production environments. It may be considered to satisfy other organization specific requirements.

![Diagram of isolation at the Dagster+ deployment level](/images/dagster-cloud/managing-deployments/isolation-level-deployments.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Isolation between assets and execution environments
</li>
<li>
User access control can be set at the code location and deployment
level
</li>
</ul>
</td>
<td>
No single glass pane to view all assets (requires switching between
multiple deployments in the UI)
</td>
</tr>
</tbody>
</table>


{/*
## Related

- [Dagster+ Hybrid deployments](/dagster-plus/deployment/hybrid)
- [Dagster+ Hybrid agents](/dagster-plus/deployment/agents)
- [Managing deployments in Dagster+](/dagster-plus/managing-deployments)
(/* - [Running multiple Dagster+ Hybrid agents](/dagster-plus/deployment/agents/running-multiple-agents#routing-requests-to-specific-agents) */}
- [Running multiple Dagster+ Hybrid agents](/todo)
{/* - [dagster_cloud.yaml](/dagster-plus/managing-deployments/dagster-cloud-yaml) */}
- [dagster_cloud.yaml](/todo)
Binary file added ...eta/static/images/dagster-cloud/managing-deployments/isolation-level-agents.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ...ic/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ...tatic/images/dagster-cloud/managing-deployments/isolation-level-deployments.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

1 comment on commit 5a4dc5d

@github-actions
Copy link

@github-actions github-actions bot commented on 5a4dc5d Dec 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs-beta ready!

✅ Preview
https://dagster-docs-beta-bp08lwt0c-elementl.vercel.app

Built with commit 5a4dc5d.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.