Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Watermark Updates not reflected on a streaming query #2160

Closed
1 of 2 tasks
polyzos opened this issue Oct 21, 2023 · 1 comment
Closed
1 of 2 tasks

[Bug] Watermark Updates not reflected on a streaming query #2160

polyzos opened this issue Oct 21, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@polyzos
Copy link
Contributor

polyzos commented Oct 21, 2023

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

0.5.0

Compute Engine

Apache Flink

Minimal reproduce step

Create an append-only table and create a streaming query that calculates a Tumbling Window.
For example a one-hour window.
Assume we have a few records for the hour 13:00:00-14:00:00 and the current watermark is < 14:00:00.
If I run a streaming query, I have no results for 13:00:00-14:00:00 as the watermark hasn't advanced yet.
If I advance the watermark I don't see the result being emitted although a snapshot gets created.
example. If I kill the query and rerun it though the results are emitted correctly.
See example output below.

Screenshot 2023-10-20 at 10 59 55 AM

What doesn't meet your expectations?

The results should be reflected while the query was running

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@polyzos polyzos added the bug Something isn't working label Oct 21, 2023
@polyzos polyzos changed the title [Bug] Watermark Updates don't not reflected on a streaming query [Bug] Watermark Updates not reflected on a streaming query Oct 21, 2023
@polyzos
Copy link
Contributor Author

polyzos commented Oct 24, 2023

I rerun this with a different checkpoint interval.
This time it worked as expected.

Wondering if I missed something on previous runs with regard to checkpointing.
Closing this for now

@polyzos polyzos closed this as completed Oct 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant