-
Notifications
You must be signed in to change notification settings - Fork 423
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TEZ-4549: Upgrade Hadoop Version to 3.4.1. #342
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We usually couple Hadoop upgrade on Tez & Hive together, they have a history of not working when not on same version, everytime we didn't, it creates classpath issues when running queries in hive, we had to quickly push the last Tez release to unblock hive, since hive with 3.3.6 wasn't compatible with Tez on hadoop-3.3.1, some Htrace stuff...
Hive doesn't compile as of now with Hadoop-3.4.0, so, I think we should hold this till Hive gets that sorted
Thank you for explanation! |
From changes POV, Hadoop 3.4.0 has zookeeper version 3.8.3 which uses logback. We have to explicitly exclude it from hadoop dependecies, otherwise it will lead to class loader issue. For example, in hive, If logback jar gets picked up first then hive-log4j2.properties won't be honored. |
@Aggarwal-Raghav We can trying to upgrading Hadoop version to 3.4.0 in Hive apache/hive#5500. |
Have added comments on hive PR apache/hive#5500 |
@slfan1989 Could you please change the Hadoop version to 3.4.1? Thx. |
This comment was marked as outdated.
This comment was marked as outdated.
I pushed a commit to this PR to bump the version to 3.4.1, if it passes we can merge this EDIT: Guess it requires a rebase :-) |
@zhangbutao Thank you for your message! and sorry I missed some information. @ayushtkn Thank you for updating this PR again! |
💔 -1 overall
This message was automatically generated. |
regarding latest precommit failures
|
@pjfanning / @slfan1989 by any chance you folks remember any change around jackson in 3.4.1 as compared to 3.4.0? any dependency removal/change or so? |
I don't know of anything. That JacksonFeature class is not used in Hadoop or Tez as far as I can see. The class was introduced in Jackson 2.12 and Hadoop has a dependency on Jackson 2.12 - we're stuck on that old version. |
hmm, Something from HDFS, some jackson-databind transitive dependency missing or so. The entire trace is like
Looking at the dependency tree now, Jackson 2.10 is getting pulled in by Avro
which earlier was coming from Jackson-databind defined in hadoop-common
Why it started happening, I am not very sure but there was some Avro related change in Hadoop Ideally I feel we should exclude Jackson coming from Avro in Hadoop itself |
💔 -1 overall
This message was automatically generated. |
seems like excluding avro did the trick
so this LGTM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
JIRA: TEZ-4549. Upgrade Hadoop Version to 3.4.0.
hadoop has been upgraded to 3.4.0, try to upgrade hadoop to 3.4.0.
Local compilation successful.