improvements to "from" operator #5378
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mccanne
changed the title
This commit improves the from operator and brings it into closer alignment with SQL. We also deprecated pool, get, and file though they can still be used but appear as "from" when running super compile. Targets of the from operator are now self describing and names are interpreted as files when running locally (super -c) and pools when running on a lake (super db). URLs continue to work in both cases. Const expressions continue to work but now need to be enclosed in brackets, e.g., "from [poolConst]". This makes easy to distinguish a file reference e.g., "from foo.json", from a record deref. e.g., "from [this.pool]" Currently these expressions must evaluate at compile time, but this sets us up to configure "from" to have a parent and make this all dynamic. Note you can also now say 'from ["a.json", "b.json"]', which will do a merge of the scans (not sequential eval). The semantic pass now does existence checks of files and pools so "super compile" tests needed to be updated to "touch" files etc. We will update docs to reflect these changes after we add the parent concept to "from" and have a bit of experience with the new syntax and semantics.
nwt
reviewed
Oct 28, 2024
| Kind string `json:"kind" unpack:""` | ||
| Path string `json:"path"` | ||
| Format string `json:"format"` | ||
| SortKeys order.SortKeys `json:"sort_keys"` |
There was a problem hiding this comment.
I don't think it's reason enough to keep this (we can just add it back if we ever want it) but Parquet has support for sorted data: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L757-L770
There was a problem hiding this comment.
I agree that the system should be able to use sort keys from file objects but this doesn't belong in the query language. Instead this should be part of the interface between the compiler and the file objects as it is with the pool adaptor.
compiler/semantic/op.go
Outdated
| names := make([]string, 0, len(vals)) | ||
| for _, val := range vals { | ||
| if super.TypeUnder(val.Type()) != super.TypeString { | ||
| a.error(entity.Expr, fmt.Errorf("from expression requires a string but encountered: %s", zson.String(val))) |
There was a problem hiding this comment.
Error messages read poorly with this colon.
Suggested change
| a.error(entity.Expr, fmt.Errorf("from expression requires a string but encountered: %s", zson.String(val))) | |
| a.error(entity.Expr, fmt.Errorf("from expression requires a string but encountered %s", zson.String(val))) |
nwt
reviewed
Oct 28, 2024
nwt
reviewed
Oct 28, 2024
nwt
reviewed
Oct 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit improves the from operator and brings it into closer alignment with SQL. We also deprecated pool, get, and file though they can still be used but appear as "from" when running super compile.
Targets of the from operator are now self describing and names are interpreted as files when running locally (super -c) and pools when running on a lake (super db). URLs continue to work in both cases.
Const expressions continue to work but now need to be enclosed in brackets, e.g., "from [poolConst]". This makes easy to distinguish a file reference e.g., "from foo.json", from a record deref. e.g., "from [this.pool]" Currently these expressions must evaluate at compile time, but this sets us up to configure "from" to have a parent and make this all dynamic. Note you can also now say 'from ["a.json", "b.json"]', which will do a merge of the scans (not sequential eval).
The semantic pass now does existence checks of files and pools so "super compile" tests needed to be updated to "touch" files etc.
We will update docs to reflect these changes after we add the parent concept to "from" and have a bit of experience with the new syntax and semantics.