HDFS-17688: Add Option to limit Balancer aboveAvgUtilized nodes num in each iteration.

[huangzhaobo99]

Description of PR

With https://issues.apache.org/jira/browse/HDFS-17646 Similarly.

It is used to limit the number of nodes that exceed the cluster average but are below threshold.

During each round of balancer operation, sometimes does not focus on nodes that are greater than the average value(AboveAvgUtilized), but instead prioritizes processing nodes that exceed the threshold(OverUtilized), we can set 'limitAboveAvgUtilizedNum' value to 0.

How was this patch tested?

Add UT

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	6m 32s		Maven dependency ordering for branch
+1 💚	mvninstall	34m 32s		trunk passed
+1 💚	compile	18m 47s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	16m 57s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	checkstyle	4m 40s		trunk passed
+1 💚	mvnsite	3m 8s		trunk passed
+1 💚	javadoc	2m 39s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 42s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	5m 48s		trunk passed
+1 💚	shadedclient	40m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 0s		the patch passed
+1 💚	compile	17m 49s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	17m 49s		the patch passed
+1 💚	compile	16m 52s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	16m 52s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 34s		the patch passed
+1 💚	mvnsite	3m 6s		the patch passed
+1 💚	javadoc	2m 32s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 41s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	6m 9s		the patch passed
+1 💚	shadedclient	40m 2s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	21m 6s		hadoop-common in the patch passed.
+1 💚	unit	261m 14s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 12s		The patch does not generate ASF License warnings.
		517m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/1/artifact/out/Dockerfile
GITHUB PR	#7225
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 72598d2360e7 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 2976131
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/1/testReport/
Max. process+thread count	2577 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[huangzhaobo99]

Hi, @Hexiaoqiao if you have time, please help to take a look at this PR, thanks!

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	7m 21s		Maven dependency ordering for branch
+1 💚	mvninstall	35m 27s		trunk passed
+1 💚	compile	22m 53s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	20m 56s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	checkstyle	4m 51s		trunk passed
+1 💚	mvnsite	3m 16s		trunk passed
+1 💚	javadoc	2m 42s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 45s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	6m 21s		trunk passed
+1 💚	shadedclient	40m 29s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 7s		the patch passed
+1 💚	compile	21m 39s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	21m 39s		the patch passed
+1 💚	compile	19m 12s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	19m 12s		the patch passed
-1 ❌	blanks	0m 2s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	checkstyle	5m 23s		the patch passed
+1 💚	mvnsite	3m 2s		the patch passed
+1 💚	javadoc	2m 33s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 46s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	6m 49s		the patch passed
+1 💚	shadedclient	41m 9s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	20m 59s		hadoop-common in the patch passed.
-1 ❌	unit	290m 9s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 41s		The patch does not generate ASF License warnings.
		565m 28s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/2/artifact/out/Dockerfile
GITHUB PR	#7225
JIRA Issue	HDFS-17688
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 4679b90d11c9 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / a8db02e
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/2/testReport/
Max. process+thread count	3136 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	7m 0s		Maven dependency ordering for branch
+1 💚	mvninstall	34m 56s		trunk passed
+1 💚	compile	22m 32s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	21m 1s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	checkstyle	4m 46s		trunk passed
+1 💚	mvnsite	3m 18s		trunk passed
+1 💚	javadoc	2m 41s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 47s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	6m 21s		trunk passed
+1 💚	shadedclient	40m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 14s		the patch passed
+1 💚	compile	22m 7s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	22m 7s		the patch passed
+1 💚	compile	20m 45s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	20m 45s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	checkstyle	4m 41s		the patch passed
+1 💚	mvnsite	3m 12s		the patch passed
+1 💚	javadoc	2m 36s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 44s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	6m 37s		the patch passed
+1 💚	shadedclient	41m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	20m 47s		hadoop-common in the patch passed.
-1 ❌	unit	291m 5s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 20s		The patch does not generate ASF License warnings.
		566m 17s

Reason	Tests
Failed junit tests	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/3/artifact/out/Dockerfile
GITHUB PR	#7225
JIRA Issue	HDFS-17688
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 39d3b23adc21 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / a8db02e
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/3/testReport/
Max. process+thread count	3135 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	18m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	7m 12s		Maven dependency ordering for branch
+1 💚	mvninstall	35m 14s		trunk passed
+1 💚	compile	18m 40s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	17m 42s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	checkstyle	4m 36s		trunk passed
+1 💚	mvnsite	3m 9s		trunk passed
+1 💚	javadoc	2m 39s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 44s		trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	5m 52s		trunk passed
+1 💚	shadedclient	40m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 2s		the patch passed
+1 💚	compile	17m 51s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	17m 51s		the patch passed
+1 💚	compile	16m 52s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	javac	16m 52s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 39s		the patch passed
+1 💚	mvnsite	3m 8s		the patch passed
+1 💚	javadoc	2m 34s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	2m 44s		the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚	spotbugs	6m 11s		the patch passed
+1 💚	shadedclient	40m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	19m 43s		hadoop-common in the patch passed.
+1 💚	unit	262m 57s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 11s		The patch does not generate ASF License warnings.
		536m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/4/artifact/out/Dockerfile
GITHUB PR	#7225
JIRA Issue	HDFS-17688
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 1ce1be67f7dc 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 8b4494a
Default Java	Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/4/testReport/
Max. process+thread count	3135 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7225/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[huangzhaobo99]

Hi, @Hexiaoqiao if you have time, please help to take a look at this PR, thanks!