diff --git a/content/sessions/2024/107-hello-quality-building-cis-to-run-providers-packages-system-tests.md b/content/sessions/2024/104-hello-quality-building-cis-to-run-providers-packages-system-tests.md similarity index 51% rename from content/sessions/2024/107-hello-quality-building-cis-to-run-providers-packages-system-tests.md rename to content/sessions/2024/104-hello-quality-building-cis-to-run-providers-packages-system-tests.md index 0f1ac91e..d4b3061e 100644 --- a/content/sessions/2024/107-hello-quality-building-cis-to-run-providers-packages-system-tests.md +++ b/content/sessions/2024/104-hello-quality-building-cis-to-run-providers-packages-system-tests.md @@ -5,19 +5,17 @@ speakers: - Freddy Demiane - Rahul Vats - Dennis Ferruzzi -time_start: 2024-09-12 15:45:00 -time_end: 2024-09-12 16:45:00 +time_start: 2024-09-12 15:10:00 +time_end: 2024-09-12 16:10:00 room: California West track: Community day: 20243 -timeslot: 107 -gridarea: "14/3/16/4" +timeslot: 104 +gridarea: "13/3/15/4" images: - /images/sessions/2024/hello-quality.jpg --- Airflow operators are a core feature of Apache Airflow and it’s extremely important that we maintain high quality of operators, prevent regressions and on the other hand we help developers with automated tests results to double check if introduced changes don’t cause regressions or backward incompatible changes and we provide Airflow release managers with information whether a given version of a provider should be released or not yet. - - - Recently a new approach to assuring production quality was implemented for AWS, Google and Astronomer-provided operators - standalone Continuous Integration processes were configured for them and test results dashboards show the results of the last test runs. What has been working well for these operator providers might be a pattern to follow for others - during this presentation, AWS, Google and Astronomer engineers are going to share the information about the internals of Test Dashboards implemented for AWS, Google and Astronomer-provided operators. This approach might be a a ‘blueprint’ to follow for other providers. \ No newline at end of file +Recently a new approach to assuring production quality was implemented for AWS, Google and Astronomer-provided operators - standalone Continuous Integration processes were configured for them and test results dashboards show the results of the last test runs. What has been working well for these operator providers might be a pattern to follow for others - during this presentation, AWS, Google and Astronomer engineers are going to share the information about the internals of Test Dashboards implemented for AWS, Google and Astronomer-provided operators. This approach might be a a ‘blueprint’ to follow for other providers. \ No newline at end of file diff --git a/content/sessions/2024/104-lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities.md b/content/sessions/2024/104-lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities.md deleted file mode 100644 index 2c3d05d0..00000000 --- a/content/sessions/2024/104-lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: "Lessons from the Ecosystem: What can Airflow Learn from Other Open-source Communities?" -slug: lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities -speakers: - - Michael Robinson -time_start: 2024-09-12 15:10:00 -time_end: 2024-09-12 15:35:00 -room: California West -track: Community -day: 20243 -timeslot: 104 -gridarea: "13/3/14/4" -images: - - /images/sessions/2024/lessons-ecosystem.jpg ---- - -The Apache Airflow community is so large and active that it’s tempting to take the view that “if it ain’t broke don’t fix it.” - - - - In a community as in a codebase, however, improvement and attention are essential to sustaining growth. And bugs are just as inevitable in community management as they are in software development. If only the fixes were, too! - - - - Airflow is large and growing because users love Airflow and our community. But what steps could be taken to enhance the typical user’s and developer’s experience of the community? - - - - This talk will provide an overview of potential learnings for Airflow community management efforts, such as project governance and analytics, derived from the speaker's experience managing the OpenLineage and Marquez open-source communities. - - - - The talk will answer questions such as: What can we learn from other open-source communities when it comes to supporting users and developers and learning from them? For example, what options exist for getting historical data out of Slack despite the limitations of the free tier? What tools can be used to make adoption metrics more reliable? What are some effective supplements to asynchronous governance? \ No newline at end of file diff --git a/content/sessions/2024/92-lessons-learned-airflow-open-source.md b/content/sessions/2024/92-lessons-learned-airflow-open-source.md deleted file mode 100644 index 03ecc3b2..00000000 --- a/content/sessions/2024/92-lessons-learned-airflow-open-source.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: "Lessons Learned While Using Airflow as Open-Source Software" -slug: lessons-learned-airflow-open-source -speakers: - - Xiaodong Deng -time_start: 2024-09-12 12:00:00 -time_end: 2024-09-12 12:25:00 -room: California West -track: Community -day: 20243 -timeslot: 92 -gridarea: "7/3/8/4" -images: - - /images/sessions/2024/lessons-learned-os.jpg -draft: true ---- - -Apache Airflow is an essential piece of the data infrastructure for many organizations and has been largely adopted by data engineers across domains for orchestration. Due to its open-source nature, there are varied strategies to operate Airflow, resulting in different challenges. In this talk, we will explore common challenges when Airflow users operate it as an open source software, and the lessons learned. Such lessons should be applicable for operating other open source softwares as well. - diff --git a/content/sessions/2024/93-lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities.md b/content/sessions/2024/93-lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities.md new file mode 100644 index 00000000..4d91c897 --- /dev/null +++ b/content/sessions/2024/93-lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities.md @@ -0,0 +1,27 @@ +--- +title: "Lessons from the Ecosystem: What can Airflow Learn from Other Open-source Communities?" +slug: lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities +speakers: + - Michael Robinson + +time_start: 2024-09-12 12:00:00 +time_end: 2024-09-12 12:25:00 +room: California West +track: Community +day: 20243 +timeslot: 93 +gridarea: "7/3/8/4" + +images: + - /images/sessions/2024/lessons-ecosystem.jpg +--- + +The Apache Airflow community is so large and active that it’s tempting to take the view that “if it ain’t broke don’t fix it.” + +In a community as in a codebase, however, improvement and attention are essential to sustaining growth. And bugs are just as inevitable in community management as they are in software development. If only the fixes were, too! + +Airflow is large and growing because users love Airflow and our community. But what steps could be taken to enhance the typical user’s and developer’s experience of the community? + +This talk will provide an overview of potential learnings for Airflow community management efforts, such as project governance and analytics, derived from the speaker's experience managing the OpenLineage and Marquez open-source communities. + +The talk will answer questions such as: What can we learn from other open-source communities when it comes to supporting users and developers and learning from them? For example, what options exist for getting historical data out of Slack despite the limitations of the free tier? What tools can be used to make adoption metrics more reliable? What are some effective supplements to asynchronous governance? \ No newline at end of file diff --git a/content/sessions/2024/93-adaptive-memory-scaling-for-robust-airflow-pipelines.md b/content/sessions/2024/94-adaptive-memory-scaling-for-robust-airflow-pipelines.md similarity index 99% rename from content/sessions/2024/93-adaptive-memory-scaling-for-robust-airflow-pipelines.md rename to content/sessions/2024/94-adaptive-memory-scaling-for-robust-airflow-pipelines.md index dedafd2b..32dd5b68 100644 --- a/content/sessions/2024/93-adaptive-memory-scaling-for-robust-airflow-pipelines.md +++ b/content/sessions/2024/94-adaptive-memory-scaling-for-robust-airflow-pipelines.md @@ -10,7 +10,7 @@ time_end: 2024-09-12 13:15:00 room: California East track: Use cases day: 20243 -timeslot: 93 +timeslot: 94 gridarea: "8/2/10/3" images: - /images/sessions/2024/adaptive-memory-scaling.jpg diff --git a/content/sessions/2024/94-the-silent-symphony-keeping-airflow-s-ci-cd-and-dev-tools-in-tune.md b/content/sessions/2024/95-the-silent-symphony-keeping-airflow-s-ci-cd-and-dev-tools-in-tune.md similarity index 98% rename from content/sessions/2024/94-the-silent-symphony-keeping-airflow-s-ci-cd-and-dev-tools-in-tune.md rename to content/sessions/2024/95-the-silent-symphony-keeping-airflow-s-ci-cd-and-dev-tools-in-tune.md index f67c7c33..f8f94988 100644 --- a/content/sessions/2024/94-the-silent-symphony-keeping-airflow-s-ci-cd-and-dev-tools-in-tune.md +++ b/content/sessions/2024/95-the-silent-symphony-keeping-airflow-s-ci-cd-and-dev-tools-in-tune.md @@ -8,7 +8,7 @@ time_end: 2024-09-12 13:15:00 room: California West track: Community day: 20243 -timeslot: 94 +timeslot: 95 gridarea: "8/3/10/4" images: - /images/sessions/2024/silent-symphony.jpg diff --git a/content/sessions/2024/95-seeing-clearly-with-airflow-the-shift-to-data-aware-orchestration.md b/content/sessions/2024/96-seeing-clearly-with-airflow-the-shift-to-data-aware-orchestration.md similarity index 98% rename from content/sessions/2024/95-seeing-clearly-with-airflow-the-shift-to-data-aware-orchestration.md rename to content/sessions/2024/96-seeing-clearly-with-airflow-the-shift-to-data-aware-orchestration.md index f5c77d42..9189b98d 100644 --- a/content/sessions/2024/95-seeing-clearly-with-airflow-the-shift-to-data-aware-orchestration.md +++ b/content/sessions/2024/96-seeing-clearly-with-airflow-the-shift-to-data-aware-orchestration.md @@ -9,7 +9,7 @@ time_end: 2024-09-12 13:15:00 room: Elizabethan A+B track: New features day: 20243 -timeslot: 95 +timeslot: 96 gridarea: "8/4/10/5" images: - /images/sessions/2024/data-centric.jpg diff --git a/content/sessions/2024/96-astronomer.md b/content/sessions/2024/97-astronomer.md similarity index 100% rename from content/sessions/2024/96-astronomer.md rename to content/sessions/2024/97-astronomer.md