Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruler: Disable x-scope-orgid header append in remote write #11819

Merged

Conversation

jburnham
Copy link
Contributor

@jburnham jburnham commented Jan 29, 2024

What this PR does / why we need it:
Adds option to disable addition of X-Scope-OrgID header in ruler remote write config. Needed to remote write to Amazon Managed Prometheus (Cortex).

Which issue(s) this PR fixes:
I suspect (but cannot prove) this will add a workaround for this existing open issue #9438

Special notes for your reviewer:
This is a similar PR from the Tempo project where it could not remote write to Amazon Managed Prometheus. We were having the same issues with the X-Scope-OrgID header that now works once this header is disabled.
Related issue from Tempo project

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jan 29, 2024
@jburnham jburnham changed the title ruler: disable x-scope-orgid header append in remote write Ruler: Disable x-scope-orgid header append in remote write Jan 29, 2024
@jburnham jburnham force-pushed the jburnham/ruler-disable-x-scope-orgid branch 2 times, most recently from a6bad93 to 89e4ac2 Compare January 29, 2024 22:01
@jburnham jburnham force-pushed the jburnham/ruler-disable-x-scope-orgid branch from 89e4ac2 to 91f91b1 Compare January 29, 2024 22:10
@jburnham jburnham marked this pull request as ready for review January 29, 2024 22:32
@jburnham jburnham requested a review from a team as a code owner January 29, 2024 22:32
@jburnham
Copy link
Contributor Author

jburnham commented Feb 2, 2024

Had an open ticket with AWS about this behavior and received the following reply.

The AMP team has confirmed this is expected behavior, and applications should remove this header from their remote write requests to AMP.

They then also linked to this PR.

Copy link
Contributor

@dannykopping dannykopping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with a minor nit.

CHANGELOG.md Outdated Show resolved Hide resolved
@jburnham
Copy link
Contributor Author

@dannykopping I ran make format to fix the Drone CI check.

@dannykopping dannykopping merged commit 1b4d23f into grafana:main Feb 14, 2024
9 checks passed
@dannykopping
Copy link
Contributor

Thanks for your contribution @jburnham 👍

@jburnham jburnham deleted the jburnham/ruler-disable-x-scope-orgid branch February 14, 2024 17:46
rhnasc pushed a commit to inloco/loki that referenced this pull request Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/M type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants