feat(chunks-inspect): support structured metadata #11506
Conversation
|
Trivy scan found the following vulnerabilities:
|
cmd/chunks-inspect/loki.go
Outdated
| c, _ := chunkenc.NewByteChunk(data, 0, 0) | ||
| bs := c.Blocks(from, through) | ||
| fmt.Println("Blocks: ", len(bs)) | ||
|
|
||
| pipeline := log.NewNoopPipeline() | ||
| for idx, block := range bs { | ||
| fmt.Println("Block : ", idx) | ||
| fmt.Println("MinTime: ", time.Unix(0, block.MinTime()).UTC()) | ||
| fmt.Println("MaxTime: ", time.Unix(0, block.MaxTime()).UTC()) | ||
| fmt.Println("Offset : ", block.Offset()) | ||
| iter := block.Iterator(context.Background(), pipeline.ForStream(nil)) | ||
| for iter.Next() { | ||
| e := iter.Entry() | ||
| fmt.Println(e.Timestamp.UTC(), " ", e.Line) | ||
| for _, meta := range e.StructuredMetadata { | ||
| fmt.Println("\t", meta.Name, "=", meta.Value) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| err := c.Close() | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
There was a problem hiding this comment.
this seems like a reasonable change, but looking at the structure of the rest of the code I think printing of all these details would be done in main.go, it looks to me like process just builds the details of the chunk into something that main.go prints
I haven't been keeping up with structured metadata. If it's separate from a chunk then imo we should either have a separate function to gather and return the structured metadata to main.go or an additional return value in parseLokiChunk for the metadata, perhaps with a parameter to choose whether or not you want to receive info about the structured metadata.
There was a problem hiding this comment.
Thank you @cstyan for your feedback! I will restructure the code and go forward.
agebhar1
force-pushed
the
feature/gh10767-chunks-inspect
branch
2 times, most recently
from
79694b3 to
2423fba
Compare
agebhar1
force-pushed
the
feature/gh10767-chunks-inspect
branch
from
2423fba to
d803d44
Compare
|
Hi @cstyan! I've moved the output to the |
|
I can do another review sometime this week 👍 for now it looks like there's some linting failures |
|
Thanks a lot @cstyan. I've removed temporarily the option to store the binary (raw) block data. I suggest to add an option to save the original log data to file. |
cmd/chunks-inspect/loki.go
Outdated
| "encoding/binary" | ||
| "fmt" | ||
| "hash/crc32" | ||
| "github.com/grafana/loki/pkg/chunkenc" |
There was a problem hiding this comment.
I think we also have a lint rule that will say that this needs to be at the bottom of the import list with a newline in between it and the stdlib imports.
|
There's still some lint failures I think those are viewable if you click |
|
I'm sorry @cstyan. Will check and fix it hopefully in a couple of days. |
agebhar1
force-pushed
the
feature/gh10767-chunks-inspect
branch
2 times, most recently
from
4f6dae9 to
9006fc1
Compare
|
Hello @cstyan, I fixed the lint issues. The I also had a look into reading the header of the chunk with existing implementation but did not found the piece of code yet. I will try with the local storage client. |
agebhar1
force-pushed
the
feature/gh10767-chunks-inspect
branch
from
9006fc1 to
a471ec9
Compare
|
Sounds great @cstyan. I re-based the commit to latest main and updated the commit message. |
agebhar1
changed the title
|
sorry @agebhar1 it looks like there was a CI change merged yesterday afternoon NA time so I think we need one more rebase on top of main 🙇 |
agebhar1
force-pushed
the
feature/gh10767-chunks-inspect
branch
from
a471ec9 to
ad92220
Compare
|
No worries @cstyan. I did the rebase. |
|
Thanks for your patience @agebhar1, the build is green but one last thing, I don't think we should be removing the |
|
Hi @cstyan, if I try to add add the module github.com/grafana/loki/cmd/chunks-inspect
go 1.20and run $ go mod tidy
go: finding module for package github.com/grafana/loki/pkg/logql/log
go: finding module for package github.com/golang/snappy
go: finding module for package github.com/grafana/loki/pkg/chunkenc
go: found github.com/golang/snappy in github.com/golang/snappy v0.0.4
go: found github.com/grafana/loki/pkg/chunkenc in github.com/grafana/loki v1.6.1
go: finding module for package github.com/grafana/loki/pkg/logql/log
go: github.com/grafana/loki/cmd/chunks-inspect imports
github.com/grafana/loki/pkg/logql/log: module github.com/grafana/loki@latest found (v1.6.1), but does not contain package github.com/grafana/loki/pkg/logql/logAll other (loki, logcli, ...) usees the top level module description. |
agebhar1
force-pushed
the
feature/gh10767-chunks-inspect
branch
from
ad92220 to
dd0bb42
Compare
|
@cstyan will fix it |
agebhar1
force-pushed
the
feature/gh10767-chunks-inspect
branch
from
d65a7c4 to
d7ae1b9
Compare
|
I fixed the lint error @cstyan. |
|
sorry, missed this for a while, lets merge and make follow up improvements if needed 👍 |
|
That's awesome @cstyan 👍. I'm ready to help for future improvements. |
…'clean room' implementation able to decode chunks without importing any Loki code, this update undid that implementation and we would prefer to keep it as a "clean room" type application Signed-off-by: Edward Welch <[email protected]>
Signed-off-by: Edward Welch <[email protected]>
What this PR does / why we need it:
Support chunk format v4 (aka structured metadata) for
chunks-inspect.Which issue(s) this PR fixes:
Fixes #10767
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)CHANGELOG.mdupdatedadd-to-release-noteslabeldocs/sources/setup/upgrade/_index.mdproduction/helm/loki/Chart.yamland updateproduction/helm/loki/CHANGELOG.mdandproduction/helm/loki/README.md. Example PRdeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR