Skip to content

Commit

Permalink
[1.9] Remove Python 3.8 Support (#25523)
Browse files Browse the repository at this point in the history
## Summary & Motivation

Remove support for Python 3.8

## How I Tested These Changes

## Changelog

Dagster no longer supports Python 3.8, which hit EOL on 2024-10-07.
  • Loading branch information
OwenKephart authored Oct 28, 2024
1 parent 9365bb0 commit 84305e0
Show file tree
Hide file tree
Showing 115 changed files with 92 additions and 200 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@

class AvailablePythonVersion(str, Enum):
# Ordering is important here, because some steps will take the highest/lowest available version.
V3_8 = "3.8"
V3_9 = "3.9"
V3_10 = "3.10"
V3_11 = "3.11"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ def build_helm_steps() -> List[BuildkiteStep]:
os.path.join("helm", "dagster", "schema"),
unsupported_python_versions=[
# run helm schema tests only once, on the latest python version
AvailablePythonVersion.V3_8,
AvailablePythonVersion.V3_9,
AvailablePythonVersion.V3_10,
AvailablePythonVersion.V3_11,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,6 @@ def k8s_extra_cmds(version: str, _) -> List[str]:
PackageSpec(
"examples/with_airflow",
unsupported_python_versions=[
AvailablePythonVersion.V3_9,
AvailablePythonVersion.V3_10,
AvailablePythonVersion.V3_11,
AvailablePythonVersion.V3_12,
Expand Down Expand Up @@ -637,9 +636,6 @@ def tox_factors_for_folder(tests_folder_name: str) -> List[str]:
),
PackageSpec(
"python_modules/libraries/dagster-ge",
unsupported_python_versions=[
AvailablePythonVersion.V3_8,
],
),
PackageSpec(
"python_modules/libraries/dagster-k8s",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ def build_test_project_steps() -> List[GroupStep]:
)
.on_python_image(
# py version can be bumped when rebuilt
f"buildkite-build-test-project-image:py{AvailablePythonVersion.V3_8}-{BUILDKITE_BUILD_TEST_PROJECT_IMAGE_IMAGE_VERSION}",
f"buildkite-build-test-project-image:py{AvailablePythonVersion.V3_11}-{BUILDKITE_BUILD_TEST_PROJECT_IMAGE_IMAGE_VERSION}",
[
"AIRFLOW_HOME",
"AWS_ACCOUNT_ID",
Expand Down
1 change: 0 additions & 1 deletion .buildkite/dagster-buildkite/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
description="Tools for buildkite automation",
url="https://github.com/dagster-io/dagster/tree/master/.buildkite/dagster-buildkite",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
Expand Down
2 changes: 1 addition & 1 deletion docs/content/concepts/dagster-pipes/aws-glue.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The [dagster-aws](/\_apidocs/libraries/dagster-aws) integration library provides
- **In AWS**:

- An existing AWS account
- An AWS Glue job with a Python 3.8+ runtime environment
- An AWS Glue job with a Python 3.9+ runtime environment

---

Expand Down
4 changes: 2 additions & 2 deletions docs/content/dagster-plus/deployment/serverless.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ if __name__ == "__main__":

### Using a different Python version

The default version of Python for Serverless deployments is Python 3.8. Versions 3.9 through 3.12 are also supported. You can specify the version you want by updating your GitHub workflow or using the `--python-version` command line argument:
The default version of Python for Serverless deployments is Python 3.9. Versions 3.10 through 3.12 are also supported. You can specify the version you want by updating your GitHub workflow or using the `--python-version` command line argument:

- **With GitHub**: Change the `python_version` parameter for the `build_deploy_python_executable` job in your `.github/workflows` files. For example:

Expand All @@ -190,7 +190,7 @@ The default version of Python for Serverless deployments is Python 3.8. Versions

Dagster+ runs your code on a Docker image that we build as follows:

1. The standard Python "slim" [Docker image](https://hub.docker.com/\_/python), such as `python:3.8-slim` is used as the base.
1. The standard Python "slim" [Docker image](https://hub.docker.com/\_/python), such as `python:3.9-slim` is used as the base.
2. The `dagster-cloud[serverless]` module installed in the image.

As far as possible, add all dependencies by including the corresponding native Python bindings in your `setup.py`. When that is not possible, you can build and upload a custom base image that will be used to run your Python code.
Expand Down
1 change: 0 additions & 1 deletion docs/dagster-ui-screenshot/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
description="Utility for taking automated screenshots from the Dagster UI",
classifiers=[
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ env:
DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_ORGANIZATION }}
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
ENABLE_FAST_DEPLOYS: 'true'
PYTHON_VERSION: '3.8'
PYTHON_VERSION: '3.12'
DAGSTER_CLOUD_FILE: 'dagster_cloud.yaml'
DBT_TARGET: 'prod'
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ lesson: '2'

To install Dagster, you’ll need:

- **To install Python**. Dagster supports Python 3.8 through 3.12.
- **To install Python**. Dagster supports Python 3.9 through 3.12.
- **A package manager like pip or poetry**. If you need to install a package manager, refer to the following installation guides:
- [pip](https://pip.pypa.io/en/stable/installation/)
- [Poetry](https://python-poetry.org/docs/)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ In the following image, each dark purple box is a code location. By separating c
Code locations can be used to separate code by categories, such as:

- Teams (ex., marketing or product)
- Python version (ex., legacy code using Python 3.8 and newer Python 3.11 code locations)
- Python version (ex., legacy code using Python 3.9 and newer Python 3.11 code locations)
- Dependency versions (ex., one model using PyTorch v1 and another using PyTorch v2)

These code locations are all maintained in one single Dagster deployment, and changes made to one code location won’t lead to downtime in another one. This allows you to silo packages and versions, but still create connections between data assets as needed. For example, an asset in one code location can depend on an asset in another code location.
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Dagster uses [PEX](https://docs.pex-tool.org/) to package your code and deploy t


## Use a different Python version \{#python-version}
The default Python version for Dagster+ Serverless is Python 3.8. Python versions 3.9 through 3.12 are also supported. You can specify the Python version you want to use in your GitHub or GitLab workflow, or by using the `dagster-cloud` CLI.
The default Python version for Dagster+ Serverless is Python 3.9. Python versions 3.10 through 3.12 are also supported. You can specify the Python version you want to use in your GitHub or GitLab workflow, or by using the `dagster-cloud` CLI.

<Tabs groupId="method">
<TabItem value="GitHub" label="GitHub">
Expand Down
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sidebar_label: Installation

To follow the steps in this guide, you'll need:

- To install Python 3.8 or higher. **Python 3.11 is recommended**.
- To install Python 3.9 or higher. **Python 3.12 is recommended**.
- To install pip, a Python package installer

## Setting up a virtual environment
Expand Down
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Welcome to Dagster! In this guide, you'll use Dagster to create a basic pipeline
To follow the steps in this guide, you'll need:

- Basic Python knowledge
- Python 3.8+ installed on your system. Refer to the [Installation guide](/getting-started/installation) for information.
- Python 3.9+ installed on your system. Refer to the [Installation guide](/getting-started/installation) for information.
</details>

## Step 1: Set up the Dagster environment
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Dagster supports Python 3.8 through 3.12.
Dagster supports Python 3.9 through 3.12.
2 changes: 1 addition & 1 deletion docs/next/markdoc/partials/DagsterVersion.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Dagster supports Python 3.8 through 3.12.
Dagster supports Python 3.9 through 3.12.
1 change: 0 additions & 1 deletion docs/sphinx/_ext/dagster-sphinx/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
description="Dagster-specific sphinx extension.",
url="https://github.com/dagster-io/dagster",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
Expand Down
2 changes: 0 additions & 2 deletions docs/sphinx/_ext/sphinx-mdx-builder/tox.ini
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
[tox]
minversion = 4
envlist =
python3.8-sphinx{7,8}
python3.9-sphinx{7,8}
python3.10-sphinx{7,8}
python3.11-sphinx{7,8}
Expand All @@ -10,7 +9,6 @@ envlist =

[testenv]
basepython =
python3.8: python3.8
python3.9: python3.9
python3.10: python3.10
python3.11: python3.11
Expand Down
1 change: 0 additions & 1 deletion examples/development_to_production/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
description="Dagster example of local development and production deployment.",
url="https://github.com/dagster-io/dagster/tree/master/examples/development_to_production",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ variables:
# highlight-end
DAGSTER_CLOUD_URL: $DAGSTER_CLOUD_URL
DAGSTER_CLOUD_API_TOKEN: $DAGSTER_CLOUD_API_TOKEN
PYTHON_VERSION: '3.8'
PYTHON_VERSION: '3.9'
1 change: 0 additions & 1 deletion examples/docs_beta_snippets/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
license="Apache-2.0",
url="https://github.com/dagster-io/dagster/tree/master/examples/docs_beta_snippets",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
Expand Down
1 change: 0 additions & 1 deletion examples/docs_snippets/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
license="Apache-2.0",
url="https://github.com/dagster-io/dagster/tree/master/examples/docs_snippets",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
Expand Down
1 change: 0 additions & 1 deletion examples/experimental/assets_yaml_dsl/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
description="Dagster example of yaml dsl for building asset graphs",
url="https://github.com/dagster-io/dagster/tree/master/examples/assets_yaml_dsl",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
Expand Down
1 change: 0 additions & 1 deletion examples/experimental/dagster-airlift/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ def get_version() -> str:
"dagster-airlift"
),
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@ def test_single_file_many_blueprints() -> None:
def test_single_file_many_blueprints_builtin_list() -> None:
defs = load_defs_from_yaml(
path=Path(__file__).parent / "yaml_files" / "list_of_blueprints.yaml",
per_file_blueprint_type=list[SimpleAssetBlueprint], # type: ignore
per_file_blueprint_type=list[SimpleAssetBlueprint],
)
assert set(defs.get_asset_graph().get_all_asset_keys()) == {
AssetKey("asset1"),
Expand Down
1 change: 0 additions & 1 deletion examples/experimental/dagster-blueprints/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ def get_version() -> str:
"dagster-blueprints"
),
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
1 change: 0 additions & 1 deletion examples/feature_graph_backed_assets/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
description="Dagster example of op and graph-backed assets.",
url="https://github.com/dagster-io/dagster/tree/master/examples/feature_graph_backed_assets",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
Expand Down
1 change: 0 additions & 1 deletion examples/project_fully_featured/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
author="Dagster Labs",
author_email="[email protected]",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Operating System :: OS Independent",
Expand Down
1 change: 0 additions & 1 deletion helm/dagster/schema/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
description="Tools for Dagster Helm schema",
url="https://github.com/dagster-io/dagster/tree/master/helm/dagster/schema",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
description="A Dagster integration for k8s-test-infra",
url="https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-k8s-test-infra",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.8-slim
FROM python:3.10-slim

ARG USER_CODE_VERSION
ARG USER_CODE_LIBRARY_VERSION
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.8-slim
FROM python:3.10-slim

ARG USER_CODE_DEFINITIONS_FILE

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.8-slim
FROM python:3.10-slim

ARG WEBSERVER_VERSION
ARG WEBSERVER_LIBRARY_VERSION
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.8-slim
FROM python:3.10-slim

COPY ./python_modules/ /tmp/python_modules/

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ analyzeUnannotatedFunctions = false

# Minimum version of Python on which code must run. This determines the standard library stubs used by
# pyright.
pythonVersion = "3.8"
pythonVersion = "3.9"

# Use "basic" ruleset. This differs from strict in several ways, but most
# importantly it does not flag untyped code as an error.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ RUN apt-get update && apt-get install -y \

RUN curl -sSL https://get.docker.com/ > install_docker.sh

RUN pip install pex==2.1.12
RUN pip install pex

ADD requirements.txt .

Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
"3.8": 2024-01-19T144058
"3.11": 2024-10-28T164833
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"3.8":
"3.11":
docker_args:
BASE_IMAGE: python:3.8-slim
BASE_IMAGE: python:3.11-slim
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"3.8":
"3.9":
docker_args:
BASE_IMAGE: python:3.8-windowsservercore-1809
BASE_IMAGE: python:3.9-windowsservercore-1809
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
"3.8":
docker_args:
BASE_IMAGE: python:3.8-slim
"3.9":
docker_args:
BASE_IMAGE: python:3.9-slim
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
"3.8":
docker_args:
BASE_IMAGE: python:3.8-slim
"3.9":
docker_args:
BASE_IMAGE: python:3.9-slim
Expand Down
1 change: 0 additions & 1 deletion python_modules/automation/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
description="Tools for infrastructure automation",
url="https://github.com/dagster-io/dagster/tree/master/python_modules/automation",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"License :: OSI Approved :: Apache Software License",
Expand Down
3 changes: 1 addition & 2 deletions python_modules/dagit/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ def get_version():
long_description_content_type="text/markdown",
url="https://github.com/dagster-io/dagster",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand All @@ -42,7 +41,7 @@ def get_version():
],
packages=find_packages(exclude=["dagit_tests*"]),
include_package_data=True,
python_requires=">=3.8,<3.13",
python_requires=">=3.9,<3.13",
install_requires=[
f"dagster-webserver{pin}",
],
Expand Down
3 changes: 1 addition & 2 deletions python_modules/dagster-graphql/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ def get_version() -> str:
description="The GraphQL frontend to python dagster.",
url="https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-graphql",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand All @@ -34,7 +33,7 @@ def get_version() -> str:
],
packages=find_packages(exclude=["dagster_graphql_tests*"]),
include_package_data=True,
python_requires=">=3.8,<3.13",
python_requires=">=3.9,<3.13",
install_requires=[
f"dagster{pin}",
"graphene>=3,<4",
Expand Down
3 changes: 1 addition & 2 deletions python_modules/dagster-pipes/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ def get_version() -> str:
long_description=desc,
url="https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-pipes",
classifiers=[
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand All @@ -36,6 +35,6 @@ def get_version() -> str:
],
packages=find_packages(exclude=["dagster_pipes_tests*"]),
include_package_data=True,
python_requires=">=3.8,<3.13",
python_requires=">=3.9,<3.13",
zip_safe=False,
)
Loading

3 comments on commit 84305e0

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-university ready!

✅ Preview
https://dagster-university-b417ut56w-elementl.vercel.app

Built with commit 84305e0.
This pull request is being automatically deployed with vercel-action

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-qk2akzs7l-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 84305e0.
This pull request is being automatically deployed with vercel-action

@github-actions
Copy link

@github-actions github-actions bot commented on 84305e0 Oct 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs-beta ready!

✅ Preview
https://dagster-docs-beta-jj92rvagq-elementl.vercel.app

Built with commit 84305e0.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.