Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NO-SNOW Improve offset logging for Snowpipe #1027

Merged
merged 10 commits into from
Dec 11, 2024
Merged

NO-SNOW Improve offset logging for Snowpipe #1027

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0a4b399
NO-SNOW Improve logging
sfc-gh-mbobowski Dec 10, 2024
1d04590
NO-SNOW Get rid of shouldLogDebugOrTrace / introduce @ParameterizedTest
sfc-gh-mbobowski Dec 10, 2024
813a66f
NO-SNOW Get rid of fileType
sfc-gh-mbobowski Dec 10, 2024
a4dbded
NO-SNOW Rewritten to streams
sfc-gh-mbobowski Dec 10, 2024
b354f54
NO-SNOW Code polishing
sfc-gh-mbobowski Dec 10, 2024
30bc9f2
NO-SNOW Sorting
sfc-gh-mbobowski Dec 10, 2024
6d19b6b
List files on debug lvl
sfc-gh-wtrefon Dec 10, 2024
23c8585
One log to rule them all
sfc-gh-mbobowski Dec 11, 2024
090eed5
One more log fix
sfc-gh-mbobowski Dec 11, 2024
f8315ad
Update src/main/java/com/snowflake/kafka/connector/internal/Snowflake…
sfc-gh-mbobowski Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/main/java/com/snowflake/kafka/connector/internal/KCLogger.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,10 @@ public void error(String format, Object... vars) {
}
}

public boolean isDebugEnabled() {
return logger.isDebugEnabled();
}

private String getFormattedLogMessage(String format, Object... vars) {
if (prependMdcContext) {
String connCtx = MDC.get(MDC_CONN_CTX_KEY);
Expand Down
21 changes: 13 additions & 8 deletions src/main/java/com/snowflake/kafka/connector/internal/SnowflakeSinkServiceV1.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1102,14 +1102,19 @@ private void purge(List<String> files) {
private void moveToTableStage(List<String> failedFiles) {
if (!failedFiles.isEmpty()) {
OffsetContinuityRanges offsets = searchForMissingOffsets(failedFiles);
LOGGER.info(
"Moving failed files for pipe: {} to tableStage failedFileCount: {}, continuousOffsets:"
+ " {}, missingOffsets: {}",
pipeName,
failedFiles.size(),
offsets.getContinuousOffsets(),
offsets.getMissingOffsets());
LOGGER.debug("Moving failed files: {}", failedFiles);
String baseLog =
String.format(
"Moving failed files for pipe: %s to tableStage failedFileCount: %d,"
+ " continuousOffsets: %s, missingOffsets: %s",
pipeName,
failedFiles.size(),
offsets.getContinuousOffsets(),
offsets.getMissingOffsets());
if (LOGGER.isDebugEnabled()) {
LOGGER.info("{} - {}", baseLog, failedFiles);
sfc-gh-dseweryn marked this conversation as resolved.
Show resolved Hide resolved
} else {
LOGGER.info(baseLog);
}
conn.moveToTableStage(tableName, stageName, failedFiles);
}
}
Expand Down
Loading