Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Calcite to preserve LATERALs in LATERAL UNNEST calls #542

Merged
merged 6 commits into from
Nov 14, 2024

Conversation

KevinGe00
Copy link
Contributor

@KevinGe00 KevinGe00 commented Oct 30, 2024

What changes are proposed in this pull request, and why are they necessary?

RelNode type derivation on the Coral IR -> Spark SQL path was introduced in #507 in ExtractUnionFunctionTransformer here. The type derivation requires calling Calcite's validate function which was incorrectly dropping the LATERAL keyword, this was previously only fixed for ... LATERAL (SELECT ..... case but has since been fixed for .. LATERAL (UNNEST ... here.

This PR bumps the Calcite version to include the fix and adds a new unit to ensure LATERALs are not dropped for views with the Coral IR -> Spark SQL type derivation (through ExtractUnionFunctionTransformer).

How was this patch tested?

  • ./gradlew clean build
  • new unit test passing when using newest Calcite with fix
  • no regressions when testing on production views
    • since certain views that were already failing due to this issue, their translation wouldn't be tested in the regression pipeline (since the pipeline is designed is test regressions, we take only the views with valid translations on the coral master branch and run translations on that subset of views only with the feature coral branch), ran adhoc regression tests with those failing views and they now succeed with these changes.

@KevinGe00 KevinGe00 marked this pull request as ready for review November 12, 2024 21:03
@KevinGe00 KevinGe00 changed the title [WIP DO NOT MERGE] Bump Calcite to preserve LATERALs in LATERAL UNNEST calls Bump Calcite to preserve LATERALs in LATERAL UNNEST calls Nov 12, 2024
@KevinGe00 KevinGe00 merged commit 15fc504 into linkedin:master Nov 14, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants