Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for elementary to utilize partitions in Bigquery #1715

Open
fredrik-ic opened this issue Oct 3, 2024 · 0 comments
Open

Allow for elementary to utilize partitions in Bigquery #1715

fredrik-ic opened this issue Oct 3, 2024 · 0 comments

Comments

@fredrik-ic
Copy link

fredrik-ic commented Oct 3, 2024

Is your feature request related to a problem? Please describe.
My team has noticed high query costs for elementary on recurring queries to the table elementary test_result_rows. This is because:

  • The table isn't created partitioned
  • Even if we partition the table on detected_at, the where-clause in the query isn't designed to use the partitions. (It will look like this: where timestamp_diff(current_timestamp, cast(detected_at as TIMESTAMP), day) < 7. The casting of the column prevents bigquery from using the partitioned column.)

Issue is also described in this slack-thread

Describe the solution you'd like
I'd like to see elementary output a query that utilizes a partitioned table. (A bonus would also be if the table was automatically created partitioned.)

Describe alternatives you've considered
I've considered adjusting the where-clause in the macro elementary/monitor/dbt_project/macros/get_result_rows_agate.sql
to something that doesn't transform detected_at, for example by using edr_timeadd, but I'm a bit unsure how it would work on the different cloud solutions.

Haven't checked out how to create the table partitioned.

Additional context

Would you be willing to contribute this feature?
Sure, but would need some guidance to not break the other clouds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant