YARN-11580. YARN Router Web supports displaying information for Non-Federation. #6127

slfan1989 · 2023-09-29T04:03:43Z

Description of PR

JIRA: YARN-11580. YARN Router Web supports displaying information for Non-Federation.

Typically, YARN Router serves multiple YARN sub-clusters in what we refer to as YARN Federation. In this scenario, there are two or more YARN sub-clusters, and Router, based on queue configuration weights, forwards client requests to a specific sub-cluster.

However, YARN Router also offers the capability to directly forward client requests to a particular local cluster. This functionality has already been implemented, and you can refer to components such as DefaultRequestInterceptorREST, DefaultClientRequestInterceptor, and DefaultRMAdminRequestInterceptor.

In this JIRA ticket, we aim to enhance the functionality of YARN Router Web to display information about a specific local cluster.

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…ederation.

hadoop-yetus · 2023-09-29T06:22:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 29s		trunk passed
+1 💚	compile	0m 31s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 28s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 27s		trunk passed
+1 💚	mvnsite	0m 32s		trunk passed
+1 💚	javadoc	0m 34s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 27s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	0m 57s		trunk passed
+1 💚	shadedclient	37m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 24s		the patch passed
+1 💚	compile	0m 24s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 24s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	0m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 23s		the patch passed
-1 ❌	javadoc	0m 23s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
-1 ❌	javadoc	0m 20s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_382-8u382-ga-1~~20.04.1-b05 with JDK Private Build-1.8.0_382-8u382-ga-1~~20.04.1-b05 generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
+1 💚	spotbugs	0m 54s		the patch passed
+1 💚	shadedclient	38m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 27s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		137m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/1/artifact/out/Dockerfile
GITHUB PR	#6127
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 0fd16f816d79 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cfa1a9f`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/1/testReport/
Max. process+thread count	579 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-09-29T11:44:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 1s		Maven dependency ordering for branch
+1 💚	mvninstall	36m 50s		trunk passed
+1 💚	compile	2m 42s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	2m 19s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	1m 24s		trunk passed
+1 💚	mvnsite	1m 11s		trunk passed
+1 💚	javadoc	1m 11s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 2s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	2m 20s		trunk passed
+1 💚	shadedclient	40m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 1s		the patch passed
+1 💚	compile	2m 33s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	2m 33s		the patch passed
+1 💚	compile	2m 20s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	2m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 16s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	mvnsite	1m 0s		the patch passed
-1 ❌	javadoc	0m 24s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt	hadoop-yarn-server-router in the patch failed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt	hadoop-yarn-server-router in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.
+1 💚	spotbugs	2m 32s		the patch passed
+1 💚	shadedclient	38m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 15s		hadoop-yarn-server-common in the patch passed.
+1 💚	unit	0m 27s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		165m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/2/artifact/out/Dockerfile
GITHUB PR	#6127
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 2582a4ccc181 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `72a565c`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/2/testReport/
Max. process+thread count	537 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2023-09-30T00:30:18Z

@goiri Thank you very much for your help in reviewing the code! The following is a screenshot in Local mode.

Router

We don't need to make changes to Router's configuration. By configuring a specific YARN cluster in Router's yarn-site.xml, Router can automatically route client requests to the YARN cluster.

yarn-site.xml

<property>
  <name>yarn.resourcemanager.ha.enabled</name>
  <value>true</value>
</property>
<property>
  <name>yarn.resourcemanager.cluster-id</name>
  <value>cluster1</value>
</property>
<property>
  <name>yarn.resourcemanager.ha.rm-ids</name>
  <value>rm1,rm2</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm1</name>
  <value>master1</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname.rm2</name>
  <value>master2</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm1</name>
  <value>master1:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address.rm2</name>
  <value>master2:8088</value>
</property>
<property>
  <name>hadoop.zk.address</name>
  <value>zk1:2181,zk2:2181,zk3:2181</value>
</property>

About Page

LocalCluster Page

Nodes Page

NodeLabels Page

Applications Page

hadoop-yetus · 2023-09-30T03:10:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 43s		Maven dependency ordering for branch
+1 💚	mvninstall	36m 16s		trunk passed
+1 💚	compile	2m 37s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	compile	2m 19s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	1m 23s		trunk passed
+1 💚	mvnsite	1m 11s		trunk passed
+1 💚	javadoc	1m 13s		trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 2s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	2m 19s		trunk passed
+1 💚	shadedclient	38m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 54s		the patch passed
+1 💚	compile	2m 28s		the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
+1 💚	javac	2m 28s		the patch passed
+1 💚	compile	2m 15s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	2m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 13s		the patch passed
+1 💚	mvnsite	0m 58s		the patch passed
+1 💚	javadoc	0m 33s		hadoop-yarn-server-common in the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.
+1 💚	javadoc	0m 23s		hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	javadoc	0m 29s		hadoop-yarn-server-common in the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.
+1 💚	javadoc	0m 22s		hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router-jdkPrivateBuild-1.8.0_382-8u382-ga-1~~20.04.1-b05 with JDK Private Build-1.8.0_382-8u382-ga-1~~20.04.1-b05 generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	spotbugs	2m 30s		the patch passed
+1 💚	shadedclient	38m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	3m 16s		hadoop-yarn-server-common in the patch passed.
+1 💚	unit	0m 28s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		162m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/3/artifact/out/Dockerfile
GITHUB PR	#6127
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 5958fd9eed0c 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `86552bf`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/3/testReport/
Max. process+thread count	532 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6127/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2023-10-03T10:22:37Z

@goiri Thank you very much for your help in reviewing the code! I will continue to follow up on YARN-11583.

…ederation. (apache#6127) Contributed by Shilun Fan. Reviewed-by: Inigo Goiri <[email protected]> Signed-off-by: Shilun Fan <[email protected]>

YARN-11580. YARN Router Web supports displaying information for Non-F…

cfa1a9f

…ederation.

github-actions bot added YARN trunk labels Sep 29, 2023

slfan1989 marked this pull request as draft September 29, 2023 04:08

YARN-11580. Fix CheckStyle.

72a565c

slfan1989 marked this pull request as ready for review September 29, 2023 09:01

goiri approved these changes Sep 29, 2023

View reviewed changes

YARN-11580. Fix CheckStyle.

86552bf

goiri approved these changes Oct 3, 2023

View reviewed changes

slfan1989 merged commit fe3984a into apache:trunk Oct 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11580. YARN Router Web supports displaying information for Non-Federation. #6127

YARN-11580. YARN Router Web supports displaying information for Non-Federation. #6127

slfan1989 commented Sep 29, 2023 •

edited

Loading

hadoop-yetus commented Sep 29, 2023

hadoop-yetus commented Sep 29, 2023

slfan1989 commented Sep 30, 2023 •

edited

Loading

hadoop-yetus commented Sep 30, 2023

slfan1989 commented Oct 3, 2023

YARN-11580. YARN Router Web supports displaying information for Non-Federation. #6127

YARN-11580. YARN Router Web supports displaying information for Non-Federation. #6127

Conversation

slfan1989 commented Sep 29, 2023 • edited Loading

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Sep 29, 2023

hadoop-yetus commented Sep 29, 2023

slfan1989 commented Sep 30, 2023 • edited Loading

hadoop-yetus commented Sep 30, 2023

slfan1989 commented Oct 3, 2023

slfan1989 commented Sep 29, 2023 •

edited

Loading

slfan1989 commented Sep 30, 2023 •

edited

Loading