Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][SPARK-50361][SQL] Fix bugs of using Lateral Column Alias in ORDER BY #48903

Draft
wants to merge 17 commits into
base: master
Choose a base branch
from

Conversation

yaooqinn
Copy link
Member

@yaooqinn yaooqinn commented Nov 20, 2024

What changes were proposed in this pull request?

  1. Aggreges using Lateral Column Alias in ORDER BY should throw LATERAL_COLUMN_ALIAS_IN_AGGREGATE_FUNC
  2. AMBIGUOUS_LATERAL_COLUMN_ALIAS if there are 2 or more matches of lateral column aliases

Why are the changes needed?

For 1,
An invalid case, such as SELECT MAX(id) AS id2 FROM range(1) ORDER BY AVG(id2), is considered valid in versions before 3.2.x. Since version 3.2, a refactoring pull request accidentally 'fixes' this case; however, the error message is not straightforward.

[MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved attribute(s) "id2" missing from "id" in operator !Aggregate [max(id#130L) AS id2#129L, avg(id2#129L) AS avg(id2#129L)#133].  SQLSTATE: XX000; line 1 pos 36;
Project [id2#129L]
+- Sort [avg(id2#129L)#133 ASC NULLS FIRST], true
   +- !Aggregate [max(id#130L) AS id2#129L, avg(id2#129L) AS avg(id2#129L)#133]
      +- Range (0, 1, step=1)

This error message somewhat confuses users when debugging/upgrading.

For 2, it's bugfix, we shall not ignore the AMBIGUOUS_LATERAL_COLUMN_ALIAS but go further resolution

In this pull request, we explicitly address this issue as we did for standard aggregates.

Does this PR introduce any user-facing change?

Yes, Aggregates using Lateral Column Alias in ORDER BY throws LATERAL_COLUMN_ALIAS_IN_AGG instead of RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT.

How was this patch tested?

new tests

Was this patch authored or co-authored using generative AI tooling?

NO

@yaooqinn yaooqinn marked this pull request as draft November 20, 2024 09:16
@github-actions github-actions bot added the SQL label Nov 20, 2024
@yaooqinn yaooqinn changed the title [WIP][SPARK-50361][SQL] Aggregates using Lateral Column Alias in ORDER BY should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC [WIP][SPARK-50361][SQL] Fix Lateral Column Alias in ORDER BY Bugs Nov 21, 2024
@yaooqinn yaooqinn changed the title [WIP][SPARK-50361][SQL] Fix Lateral Column Alias in ORDER BY Bugs [WIP][SPARK-50361][SQL] Fix bugs of using Lateral Column Alias in ORDER BY Nov 25, 2024
@yaooqinn yaooqinn closed this Nov 25, 2024
@yaooqinn yaooqinn reopened this Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant