Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Improve error message when error is raised from dataToWrite.foreach in JobOperator #967

Open
ykmr1224 opened this issue Dec 4, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@ykmr1224
Copy link
Collaborator

ykmr1224 commented Dec 4, 2024

Is your feature request related to a problem?
Currently, in case spark throws exception such as AnalysisException, it is handled by processQueryException to generate right error message for the error thrown.
If statementExecutionManager doesn't collect data from the Dataframe, the error is raised from dataToWrite.foreach in finally block, and the error won't be processed correctly and all the errors are reported as Failed to write to result index. ....

FlintREPL doesn't have this problem, since it calls queryResultWriter.processDataFrame within the try block, and raise error from there.

What solution would you like?
Make it throw the exceptions from the try block, or process the exceptions thrown from dataToWrite.foreach.

What alternatives have you considered?
n/a

Do you have any additional context?

@ykmr1224 ykmr1224 added enhancement New feature or request untriaged and removed untriaged labels Dec 4, 2024
@ykmr1224 ykmr1224 changed the title [FEATURE] Improve error message when Catalyst raised AnalysisException [FEATURE] Improve error message when error is raised from dataToWrite.foreach in JobOperator Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant