Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support dynamic paths for put and cut op #4795

Merged
merged 1 commit into from
Oct 25, 2023
Merged

Support dynamic paths for put and cut op #4795

merged 1 commit into from
Oct 25, 2023

Conversation

mattnibs
Copy link
Collaborator

@mattnibs mattnibs commented Oct 4, 2023

Fixes #4555

@philrz
Copy link
Contributor

philrz commented Oct 5, 2023

As a sign that we're on the right track, here's verification that this branch at commit 271bb18 seems to be addressing the use case laid out in #4555. As shown there, creating a new record field with a name derived from a string value required me to use a record literal + unflatten() idiom, which I've never found super intuitive.

$ zq -Z 'over this | ports:=collect(ports[0]) by ip | unflatten([{key:ip,value:ports}])' scan.json | tee 1
{
    "192.168.5.53": [
        {
            port: 515,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        },
        {
            port: 21,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        }
    ]
}
{
    "192.168.5.49": [
        {
            port: 49152,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        },
        {
            port: 3401,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        }
    ]
}

Running this branch at commit 271bb18, I can now obtain the same output using the syntax below, which previously triggered the illegal left-hand side of assignment error.

$ zq -version
Version: v1.10.0-5-g271bb189

$ zq -Z 'over this | ports:=collect(ports[0]) by ip | put this[ip]:=ports | drop ip,ports' scan.json | tee 2
{
    "192.168.5.53": [
        {
            port: 515,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        },
        {
            port: 21,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        }
    ]
}
{
    "192.168.5.49": [
        {
            port: 49152,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        },
        {
            port: 3401,
            proto: "tcp",
            status: "open",
            reason: "syn-ack",
            ttl: 64
        }
    ]
}

$ diff 1 2
[no output]

@mattnibs
Copy link
Collaborator Author

mattnibs commented Oct 5, 2023

Thanks @philrz.

Btw, I didn't realize this until working on the pr but

over this | ports:=collect(ports[0]) by ip | put this[ip]:=ports | drop ip,ports

can be shortened to

over this | ports:=collect(ports[0]) by ip | cut this[ip]:=ports

compiler/ast/dag/expr.go Outdated Show resolved Hide resolved
compiler/ast/dag/expr.go Outdated Show resolved Hide resolved
compiler/ast/dag/expr.go Outdated Show resolved Hide resolved
compiler/ast/dag/unpack.go Outdated Show resolved Hide resolved
@mattnibs mattnibs force-pushed the dynamic-paths branch 5 times, most recently from 10b823c to d2e3ddc Compare October 11, 2023 21:56
compiler/ast/dag/op.go Outdated Show resolved Hide resolved
@mattnibs mattnibs force-pushed the dynamic-paths branch 6 times, most recently from ab23f99 to 2208eab Compare October 12, 2023 18:45
@mattnibs mattnibs requested a review from nwt October 12, 2023 18:47
compiler/kernel/expr.go Outdated Show resolved Hide resolved
compiler/kernel/expr.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/kernel/expr.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/kernel/groupby.go Outdated Show resolved Hide resolved
@mattnibs mattnibs requested a review from nwt October 13, 2023 21:22
@mattnibs mattnibs force-pushed the dynamic-paths branch 2 times, most recently from 4a84927 to 95e35ea Compare October 16, 2023 19:03
compiler/kernel/expr.go Outdated Show resolved Hide resolved
compiler/kernel/groupby.go Outdated Show resolved Hide resolved
compiler/optimizer/op.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/kernel/op.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
compiler/semantic/expr.go Outdated Show resolved Hide resolved
@mattnibs mattnibs force-pushed the dynamic-paths branch 2 times, most recently from d23bf6d to 34a1db7 Compare October 16, 2023 20:24
compiler/semantic/op.go Outdated Show resolved Hide resolved
runtime/expr/eval.go Outdated Show resolved Hide resolved
compiler/semantic/op.go Outdated Show resolved Hide resolved
runtime/expr/putter.go Outdated Show resolved Hide resolved
@mattnibs mattnibs force-pushed the dynamic-paths branch 3 times, most recently from 3993399 to d4c282c Compare October 16, 2023 22:55
@mattnibs mattnibs requested a review from nwt October 16, 2023 22:56
@mattnibs mattnibs force-pushed the dynamic-paths branch 2 times, most recently from 99ee1fa to c39dd64 Compare October 17, 2023 00:21
compiler/kernel/groupby.go Outdated Show resolved Hide resolved
compiler/semantic/op.go Outdated Show resolved Hide resolved
compiler/semantic/op.go Outdated Show resolved Hide resolved
compiler/semantic/op.go Outdated Show resolved Hide resolved
compiler/semantic/op.go Outdated Show resolved Hide resolved
compiler/semantic/op.go Outdated Show resolved Hide resolved
compiler/ztests/dynamic-field-cut.yaml Outdated Show resolved Hide resolved
compiler/ztests/summarize-lhs-error.yaml Show resolved Hide resolved
runtime/expr/lval.go Outdated Show resolved Hide resolved
runtime/op/ztests/cut-dynamic-field.yaml Outdated Show resolved Hide resolved
runtime/op/ztests/put-dynamic-field.yaml Outdated Show resolved Hide resolved
runtime/expr/lval.go Outdated Show resolved Hide resolved
runtime/expr/cutter.go Outdated Show resolved Hide resolved
runtime/expr/cutter.go Outdated Show resolved Hide resolved
runtime/expr/cutter.go Outdated Show resolved Hide resolved
runtime/expr/cutter.go Outdated Show resolved Hide resolved
runtime/expr/cutter.go Show resolved Hide resolved
runtime/expr/cutter.go Outdated Show resolved Hide resolved
runtime/expr/putter.go Outdated Show resolved Hide resolved
runtime/expr/putter.go Outdated Show resolved Hide resolved
runtime/expr/putter.go Outdated Show resolved Hide resolved
runtime/expr/putter.go Outdated Show resolved Hide resolved
runtime/expr/putter.go Outdated Show resolved Hide resolved
@mattnibs mattnibs force-pushed the dynamic-paths branch 2 times, most recently from 93c8f92 to ec310b9 Compare October 23, 2023 22:15
Copy link
Member

@nwt nwt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:!!!

index/keys.go Outdated Show resolved Hide resolved
runtime/op/join/join.go Outdated Show resolved Hide resolved
@mattnibs mattnibs merged commit ef9695f into main Oct 25, 2023
4 checks passed
@mattnibs mattnibs deleted the dynamic-paths branch October 25, 2023 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Zed shorthand to create a record field name from string
3 participants