Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-1728002 get rid of reflection when enabling iceberg streaming #995

Merged
merged 7 commits into from
Nov 14, 2024
Merged

SNOW-1728002 get rid of reflection when enabling iceberg streaming #995

merged 7 commits into from
Nov 14, 2024

Conversation

sfc-gh-bzabek
Copy link
Contributor

@sfc-gh-bzabek sfc-gh-bzabek commented Nov 12, 2024 •
edited
Loading

Overview

related to SNOW-1728002
It's temporary approach. Finally we will set iceberg mode by invoking SnowflakeStreamingIngestClientFactory.builder.setIceberg(true) but the method is not yet public.

Pre-review checklist

  • This change should be part of a Behavior Change Release. See go/behavior-change.
  • This change has passed Merge gate tests
  • Snowpipe Changes
  • Snowpipe Streaming Changes
  • This change is TEST-ONLY
  • This change is README/Javadocs only
  • This change is protected by a config parameter <PARAMETER_NAME> eg snowflake.ingestion.method.
    • Yes - Added end to end and Unit Tests.
    • No - Suggest why it is not param protected
  • Is his change protected by parameter <PARAMETER_NAME> on the server side?
    • The parameter/feature is not yet active in production (partial rollout or PrPr, see Changes for Unreleased Features and Fixes).
    • If there is an issue, it can be safely mitigated by turning the parameter off. This is also verified by a test (See go/ppp).

@sfc-gh-bzabek sfc-gh-bzabek requested a review from a team as a code owner November 12, 2024 14:53
sfc-gh-bzabek
sfc-gh-bzabek commented Nov 12, 2024
View reviewed changes
src/main/java/com/snowflake/kafka/connector/internal/streaming/StreamingClientProperties.java
// TODO should be replaced by SnowflakeStreamingIngestClientFactory.builder.setIceberg(true)
// call in SNOW-1728002
this.parameterOverrides.put("enable_iceberg_streaming", "true");
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way the param is set in IcebergIngestionIT setUp.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no enum for "enable_iceberg_streaming" in ingest-sdk.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one. Please try: ParameterProvider.ENABLE_ICEBERG_STREAMING

Copy link
Contributor

@sfc-gh-mbobowski sfc-gh-mbobowski Nov 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally please remove the TODO comment and add SNOW-1728002 to the PR name. This is everything that needs to be done in scope of this JIRA.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one. Please try: ParameterProvider.ENABLE_ICEBERG_STREAMING

I had to update ingest-sdk for it.

Additionally please remove the TODO comment and add SNOW-1728002 to the PR name. This is everything that needs to be done in scope of this JIRA.

Done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I used variable from ParameterProvider it fails to compile, I think ingest-sdk 3.0.0 is not yet available

@sfc-gh-bzabek sfc-gh-bzabek changed the title in progress NO-SNOW set iceberg mode based on "snowflake.streaming.iceberg.enabled" param Nov 12, 2024
@sfc-gh-bzabek sfc-gh-bzabek force-pushed the bzabek-SNOW-1728002-read-iceberg-enabled-param branch from 27e973f to ff4f353 Compare November 13, 2024 12:54
@sfc-gh-bzabek sfc-gh-bzabek changed the title NO-SNOW set iceberg mode based on "snowflake.streaming.iceberg.enabled" param SNOW-1728002 get rid of reflection when enabling iceberg streaming Nov 13, 2024
@sfc-gh-mbobowski sfc-gh-mbobowski merged commit f7652f9 into master Nov 14, 2024
45 of 52 checks passed
@sfc-gh-mbobowski sfc-gh-mbobowski deleted the bzabek-SNOW-1728002-read-iceberg-enabled-param branch November 14, 2024 12:01
@sfc-gh-bzabek sfc-gh-bzabek restored the bzabek-SNOW-1728002-read-iceberg-enabled-param branch November 14, 2024 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sfc-gh-mbobowski sfc-gh-mbobowski sfc-gh-mbobowski approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sfc-gh-bzabek @sfc-gh-mbobowski