fix(loki.secretfilter): Fix partial masking for short secrets and support multiple allowlists per rule #2320

romain-gaillard · 2024-12-31T12:06:01Z

PR Description

Fixes the issue reported in Panic: runtime error utilizing secretfilter component #2288 where the component was crashing when the secret to mask was shorter than the partial_mask value.
Adds (partial) support for the new [[rules.allowlists]] format added in Gitleaks v8.21.0
Updates the documentation to be clearer on the fact that the component doesn't support all features of the Gitleaks configuration format.

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated
Config converters updated

github-actions · 2024-12-31T12:08:00Z

💻 Deploy preview deleted.

docs/sources/reference/components/loki/loki.secretfilter.md

mattdurham · 2025-01-06T15:01:48Z

internal/component/loki/secretfilter/secretfilter.go

 		Regexes     []string
 	}
 	Rules []struct {
 		ID          string
-		Description string
+		Description string // Not used


Are these unused variables still here so the format processes? What is the harm in removing them?

Yes, indeed they could be removed. I initially added them to keep track of features we don't currently support but might in the future.

In that case I would rather remote/comment out the whole line so they cant mistakenly be used.

👍 I removed it

mostafa

LGTM!

docs/sources/reference/components/loki/loki.secretfilter.md

Co-authored-by: Clayton Cornell <[email protected]>

romain-gaillard · 2025-01-10T16:03:39Z

@clayton-cornell @mattdurham
Thank you so much for your comments and suggestions, they should all be implemented.

mattdurham

LGTM

mattdurham · 2025-01-10T17:21:51Z

Will merge once clayton approves.

clayton-cornell

LGTM @mattdurham it's good to go

* update changelog for rc (#2360) * update changelog for rc * update changelog for rc * update changelog for rc (#2361) * update version (#2362) * update changelog for rc (#2360) * update changelog for rc * update changelog for rc Signed-off-by: matt durham <[email protected]> * fix conversion * Fix changelog main (#2364) * update version * Fix changelog * update the image version to work with the given example (#2358) * docs: fixed kafka config example (#2359) Example shows `loki.source.kafka "local"` pointing to `loki.relabel.kafka.receiver`. This leads to no new label being added. Correct example should have the kafka source pointing directly to `loki.write.local.receiver` * feat(helm): add the ability to deploy extra manifest files (#2347) * feat(helm): add the ability to deploy extra manifest files * docs(helm): run helm-docs * ci(helm): add tests * Update wal queue tls (#2363) * add tls to wal * add alloy config * update version * Add support for TLS doc. * Add changelog. * fix import order * add support and doc for round robin. * fix conversion * Update docs/sources/reference/components/prometheus/prometheus.write.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Add test * fix merge * Update internal/component/prometheus/write/queue/types.go Co-authored-by: William Dumont <[email protected]> --------- Co-authored-by: Clayton Cornell <[email protected]> Co-authored-by: William Dumont <[email protected]> * #229 Add OpenTelemetry Collector Server Auth Extensions to Receivers (#2203) * Work on adding auth so far * Cleanup * Made a ton of progress * Fix test fails? * Refactor * Add auth blocks to implementing extensions * Refactor to use feature flag * Comments * Cleanup * Spacing * Update docs * Update CHANGELOG * Last auth extension missing * We also need grpc auth * Fix opencensus docs * Fix extra comment * Update comment with findings * Properly fix merge conflict * Save file * Spelling error * That has been released now * Add auth support to influxdb receiver * Fix failing auth test/MAIL * Comment cleanup * MAIL for documentation * docs MAIL * MAIL * Move from Auth to Authentication * Update triton-go dependency to avoid embedded RSA key (#2380) * Fix examples for filter and transform processors (#2379) * fix examples filter and transform processors * remove unecessary docs about escaping strings and backticks * fix(loki.secretfilter): Fix partial masking for short secrets and support multiple allowlists per rule (#2320) * Fix partial masking bug and support new allowlist format * Add docs and changelog * Update docs * Add comments * Add comments * Minor docs update * Add more tests * Change criteria for partial redaction * Changes to partial masking rules * Fix comment location * Clarify usage of secret types * Clarify usage of secret types * Update docs/sources/reference/components/loki/loki.secretfilter.md Co-authored-by: Clayton Cornell <[email protected]> * Suggestions * Suggestions --------- Co-authored-by: Clayton Cornell <[email protected]> * Fix only run on fork guard (#2378) * Fix only run on fork guard The previous guard fails because `github.repository` resolves to the base repository on `pull_request` events. * Fix syntax * Fix relabel processed bug (#2394) * Fix issue where alloy_prometheus_relabel_metrics_processed was not being incremented. * Add unit tests * Update WAL to version that supports v2. (#2397) * Update WAL to version that supports v2. * Update WAL to version that supports v2. * Add samples check. * Clean up Alloy component docs (#2387) * First pass at cleanup, pretty tables, sort lists * Sort content, add badge * Fix link * Set link URL correctly * Still fxing link targets * One more tidy pass * database_observability: report health of component and collectors (#2392) Report unhealthy in case of errors when starting up the collectors or of any collector is stopped during operations. * update for rc.1 (#2401) * Update version. * fix version * fix version --------- Signed-off-by: matt durham <[email protected]> Co-authored-by: Adam ABICHOU <[email protected]> Co-authored-by: Jay Clifford <[email protected]> Co-authored-by: dbluxo <[email protected]> Co-authored-by: Clayton Cornell <[email protected]> Co-authored-by: William Dumont <[email protected]> Co-authored-by: Aidan Leuck <[email protected]> Co-authored-by: Sam DeHaan <[email protected]> Co-authored-by: Romain Gaillard <[email protected]> Co-authored-by: Jack Baldry <[email protected]> Co-authored-by: Cristian Greco <[email protected]>

romain-gaillard added 2 commits December 31, 2024 12:55

Fix partial masking bug and support new allowlist format

edce003

Add docs and changelog

3b0ca15

romain-gaillard requested a review from a team December 31, 2024 12:07

romain-gaillard added 4 commits December 31, 2024 13:20

Update docs

427d537

Add comments

b57bde5

Add comments

ba96cb3

Minor docs update

1cb0a46

romain-gaillard self-assigned this Dec 31, 2024

romain-gaillard added 7 commits December 31, 2024 16:35

Add more tests

d8070a9

Change criteria for partial redaction

c27fb75

Changes to partial masking rules

7fb2246

Fix comment location

30a23a9

Clarify usage of secret types

73358f8

Clarify usage of secret types

fed5146

Merge branch 'main' into secretfilter-fix-partialmasking

1ed6011

romain-gaillard marked this pull request as ready for review January 6, 2025 12:00

romain-gaillard requested review from clayton-cornell and a team as code owners January 6, 2025 12:00

mattdurham reviewed Jan 6, 2025

View reviewed changes

docs/sources/reference/components/loki/loki.secretfilter.md Outdated Show resolved Hide resolved

mattdurham reviewed Jan 6, 2025

View reviewed changes

docs/sources/reference/components/loki/loki.secretfilter.md Outdated Show resolved Hide resolved

mattdurham reviewed Jan 6, 2025

View reviewed changes

mostafa reviewed Jan 6, 2025

View reviewed changes

docs/sources/reference/components/loki/loki.secretfilter.md Outdated Show resolved Hide resolved

clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Jan 6, 2025

clayton-cornell reviewed Jan 6, 2025

View reviewed changes

docs/sources/reference/components/loki/loki.secretfilter.md Outdated Show resolved Hide resolved

docs/sources/reference/components/loki/loki.secretfilter.md Outdated Show resolved Hide resolved

docs/sources/reference/components/loki/loki.secretfilter.md Outdated Show resolved Hide resolved

romain-gaillard and others added 3 commits January 6, 2025 17:20

Update docs/sources/reference/components/loki/loki.secretfilter.md

fc34160

Co-authored-by: Clayton Cornell <[email protected]>

Suggestions

5499774

Suggestions

83a0ae9

romain-gaillard requested review from clayton-cornell and mattdurham January 10, 2025 15:40

romain-gaillard requested a review from mostafa January 10, 2025 15:40

mostafa approved these changes Jan 10, 2025

View reviewed changes

Merge branch 'main' into secretfilter-fix-partialmasking

10f8d09

mattdurham approved these changes Jan 10, 2025

View reviewed changes

clayton-cornell approved these changes Jan 10, 2025

View reviewed changes

dehaansa merged commit 3a4be44 into main Jan 10, 2025
20 checks passed

dehaansa deleted the secretfilter-fix-partialmasking branch January 10, 2025 20:26

romain-gaillard mentioned this pull request Jan 13, 2025

Panic: runtime error utilizing secretfilter component #2288

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(loki.secretfilter): Fix partial masking for short secrets and support multiple allowlists per rule #2320

fix(loki.secretfilter): Fix partial masking for short secrets and support multiple allowlists per rule #2320

romain-gaillard commented Dec 31, 2024

github-actions bot commented Dec 31, 2024 •

edited

Loading

mattdurham Jan 6, 2025

romain-gaillard Jan 6, 2025

mattdurham Jan 6, 2025

romain-gaillard Jan 6, 2025

mostafa left a comment

romain-gaillard commented Jan 10, 2025

mattdurham left a comment

mattdurham commented Jan 10, 2025

clayton-cornell left a comment

fix(loki.secretfilter): Fix partial masking for short secrets and support multiple allowlists per rule #2320

fix(loki.secretfilter): Fix partial masking for short secrets and support multiple allowlists per rule #2320

Conversation

romain-gaillard commented Dec 31, 2024

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

github-actions bot commented Dec 31, 2024 • edited Loading

mattdurham Jan 6, 2025

Choose a reason for hiding this comment

romain-gaillard Jan 6, 2025

Choose a reason for hiding this comment

mattdurham Jan 6, 2025

Choose a reason for hiding this comment

romain-gaillard Jan 6, 2025

Choose a reason for hiding this comment

mostafa left a comment

Choose a reason for hiding this comment

romain-gaillard commented Jan 10, 2025

mattdurham left a comment

Choose a reason for hiding this comment

mattdurham commented Jan 10, 2025

clayton-cornell left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 31, 2024 •

edited

Loading