Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PROD-39429 Implement migrate sys func from new channel(Format V2) to old channel (V1) #750

Conversation

sfc-gh-japatel
Copy link
Collaborator

@sfc-gh-japatel sfc-gh-japatel commented Nov 16, 2023

This is a long term implementation for potential data duplication introduced because of a new channel name format.

  • It is about moving away for customers who might have onboarded to new channel format which was introduced in version 2.1.0 (Now de-listed)
  • We will stick to old format for new customers running this for the first time.
  • Renamed channelName to channelNameFormatV1 and introduced channelNameFormatV2

Added tests in TopicPartitionChannel and TopicPartitionChannel

Notes:

End to End tests:

  1. Use 2.1.0

  2. Use two connectors:

  3. Uses channel Name v2
    Screenshot 2023-11-20 at 4 54 59 PM

  4. Stop

  5. Replace jar (2.1.1) and restart.

  6. Does the migration and you can only see old channel format
    Screenshot 2023-11-20 at 5 17 59 PM

  7. Stop again and restart in 2.1.1
    Nothing happens and we do get a valid response saying newchannelFormatV2 doesnt exist
    Screenshot 2023-11-20 at 5 18 11 PM

@sfc-gh-japatel sfc-gh-japatel marked this pull request as ready for review November 16, 2023 22:13
@sfc-gh-japatel sfc-gh-japatel changed the base branch from master to release-tag-2-1-0 November 16, 2023 22:13
@sfc-gh-japatel sfc-gh-japatel changed the base branch from release-tag-2-1-0 to japatel-SNOW-971412-release-2-1-1-ready November 16, 2023 22:15
@sfc-gh-japatel sfc-gh-japatel force-pushed the japatel-PROD-39429-migrate-sys-func branch 2 times, most recently from c985826 to 45ee965 Compare November 16, 2023 22:28
Copy link
Contributor

@sfc-gh-tzhang sfc-gh-tzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change should be merged into master first and then port to this branch?

@sfc-gh-japatel
Copy link
Collaborator Author

The change should be merged into master first and then port to this branch?

Does the order matter? (I dont think so)
Lot of merge conflicts on main hence two different PRs. Also, the commit will be needed in release branch 2.1.1 anyways.

Since you have commented on other one, lets get the discussion their and I will make sure its cp'ed in this branch too.

@sfc-gh-japatel sfc-gh-japatel force-pushed the japatel-PROD-39429-migrate-sys-func branch from 45ee965 to fe7e528 Compare November 21, 2023 01:09
@sfc-gh-japatel sfc-gh-japatel force-pushed the japatel-PROD-39429-migrate-sys-func branch from f186c2f to 7b5e72d Compare November 22, 2023 05:00
@sfc-gh-japatel
Copy link
Collaborator Author

@sfc-gh-tzhang @sfc-gh-rcheng I am going to push this to the release branch japatel-SNOW-971412-release-2-1-1-ready

This branch will be used to release 2.1.1 (No need to update the release version)

@sfc-gh-japatel
Copy link
Collaborator Author

FYI, the tests were failing if I had schematization enabled and the first batch was missing on target table. Hence the new test I added had schematization off. I think this got fixed in later versions and is not a problem in main but we should be aware of this in this version.

CC: @sfc-gh-tzhang @sfc-gh-rcheng

@sfc-gh-japatel sfc-gh-japatel merged commit 39bf407 into japatel-SNOW-971412-release-2-1-1-ready Nov 22, 2023
30 checks passed
@sfc-gh-japatel sfc-gh-japatel deleted the japatel-PROD-39429-migrate-sys-func branch November 22, 2023 08:06
khsoneji pushed a commit to confluentinc/snowflake-kafka-connector that referenced this pull request Dec 6, 2023
khsoneji pushed a commit to confluentinc/snowflake-kafka-connector that referenced this pull request Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants