Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary sorts before joins in ztests #4784

Merged
merged 2 commits into from
Sep 28, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions compiler/ztests/from-pass.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ inputs:
- name: join.zed
data: |
* | from (
pass => sort x
file right.zson => sort y
pass
file right.zson
) | inner join on x=y matched:=true
- name: left.zson
data: |
Expand All @@ -15,6 +15,7 @@ inputs:
- name: right.zson
data: |
{y:2,y:"y-two"}

outputs:
- name: stdout
data: |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ inputs:
- name: join.zed
data: |
from (
pool PoolOne => put color:=upper(color) | put namelen:=len(key) | sort key
pool PoolTwo => put price:=price+0.5 | put tag:="mytag" | sort key
pool PoolOne => put color:=upper(color) | put namelen:=len(key)
pool PoolTwo => put price:=price+0.5 | put tag:="mytag"
) | join on key=key priceinfo:=this | drop priceinfo.key

outputs:
Expand Down
4 changes: 2 additions & 2 deletions lake/ztests/cross-pool-join.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ inputs:
- name: join.zed
data: |
from (
pool fruit => cut f:=this | sort f.flavor
pool person => cut p:=this | sort p.likes
pool fruit => cut f:=this
pool person => cut p:=this
)
| join on f.flavor=p.likes p:=p
| summarize likes:=union(f.name) by name:=p.name | sort name
Expand Down
4 changes: 2 additions & 2 deletions lake/ztests/index/apply.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ inputs:
- name: query.zed
data: |
from (
pool :index_rules => sort id
pool test@main:indexes => sort rule.id | cut o:=this
pool :index_rules
pool test@main:indexes => cut o:=this
)
| left join on id = o.rule.id o
| count(o) by name, fields
Expand Down
4 changes: 2 additions & 2 deletions lake/ztests/index/update.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ inputs:
- name: query.zed
data: |
from (
pool :index_rules => sort id
pool test@main:indexes => sort rule.id | cut o:=this
pool :index_rules
pool test@main:indexes => cut o:=this
)
| left join on id = o.rule.id o
| count(o) by name,fields
Expand Down
4 changes: 2 additions & 2 deletions runtime/op/join/ztests/cut-like.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ inputs:
- name: test.zed
data: |
from (
file people.ndjson => sort likes
file fruit.ndjson => sort flavor
file people.ndjson
file fruit.ndjson
) | left join on likes=flavor fruit:=name,note:=quiet(note)
- name: people.ndjson
data: |
Expand Down
8 changes: 4 additions & 4 deletions runtime/op/join/ztests/kinds.yaml
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
script: |
echo === ANTI ===
zq -z 'anti join (file B.zson) on a=b | sort a' A.zson
zq -z 'anti join (file B.zson) on a=b' A.zson
echo === LEFT ===
zq -z 'left join (file B.zson) on a=b hit:=sb | sort a' A.zson
zq -z 'left join (file B.zson) on a=b hit:=sb' A.zson
echo === INNER ===
zq -z 'inner join (file B.zson) on a=b hit:=sb | sort a' A.zson
zq -z 'inner join (file B.zson) on a=b hit:=sb' A.zson
echo === RIGHT ===
zq -z 'right join (file C.zson) on b=c hit:=sb | sort c' B.zson
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While reviewing, I happened to notice that the original actually looks broken here, since the | sort are being tacked on after the join. It looks like it only worked consistently because the inputs happened to be pre-sorted by the respective keys.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well spotted! I'll back out the change to this file since these sorts aren't before the join.

zq -z 'right join (file C.zson) on b=c hit:=sb' B.zson

inputs:
- name: A.zson
Expand Down
4 changes: 2 additions & 2 deletions service/ztests/index/apply.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ inputs:
- name: query.zed
data: |
from (
pool :index_rules => sort id
pool test@main:indexes => sort rule.id | cut o:=this
pool :index_rules
pool test@main:indexes => cut o:=this
)
| left join on id = o.rule.id o
| count(o) by name, fields
Expand Down
4 changes: 2 additions & 2 deletions service/ztests/index/update.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ inputs:
- name: query.zed
data: |
from (
pool :index_rules => sort id
pool test@main:indexes => sort rule.id | cut o:=this
pool :index_rules
pool test@main:indexes => cut o:=this
)
| left join on id = o.rule.id o
| count(o) by name,fields
Expand Down
8 changes: 2 additions & 6 deletions zfmt/ztests/decls.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ inputs:
op stamp(assignee): ( yield {...this, assignee, ts: now()} )
op nop(foo): ( pass )
op joinTest(left_file, right_file, left_key, right_key, left_dest, right_source): (
file left_file | sort left_key
file left_file
| inner join (
file right_file | sort right_key
file right_file
) on left_key = right_key left_dest := right_source
)
joinTest("fruit.ndjson", "people.ndjson", flavor, likes, eater, name)
Expand All @@ -42,12 +42,10 @@ outputs:
from (
file left_file
)
| sort left_key
| join (
from (
file right_file
)
| sort right_key
) on left_key=right_key left_dest:=right_source
)
joinTest("fruit.ndjson", "people.ndjson", flavor, likes, eater, name)
Expand All @@ -64,13 +62,11 @@ outputs:
)

file fruit.ndjson
| sort flavor
| fork (
=>
pass
=>
file people.ndjson
| sort likes
)
| join on flavor=likes eater:=name
| yield {...this,assignee:"bob",ts:now()}
Expand Down