HDFS-17641 .Psane/backport hdfs 17641 to 3.3

[PrateekSane]

Description of PR

Backporting metrics for Low Redundancy blocks and additional tracked metrics for misreplicated blocks back to 3.3 branch.

Requires cherry picks of the following
#7123
275c478

How was this patch tested?

mvn clean install -DskipTests

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 13s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	buf	0m 1s		buf was not available.
+0 🆗	buf	0m 1s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	14m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	34m 12s		branch-3.3 passed
+1 💚	compile	17m 48s		branch-3.3 passed
+1 💚	checkstyle	2m 46s		branch-3.3 passed
+1 💚	mvnsite	5m 22s		branch-3.3 passed
+1 💚	javadoc	4m 44s		branch-3.3 passed
-1 ❌	spotbugs	2m 37s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 💚	shadedclient	34m 33s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 40s		the patch passed
+1 💚	compile	17m 1s		the patch passed
+1 💚	cc	17m 1s		the patch passed
+1 💚	javac	17m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 8s	/results-checkstyle-root.txt	root: The patch generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417)
+1 💚	mvnsite	5m 21s		the patch passed
+1 💚	javadoc	4m 43s		the patch passed
+1 💚	spotbugs	10m 42s		the patch passed
+1 💚	shadedclient	34m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 5s		hadoop-common in the patch passed.
+1 💚	unit	2m 34s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	202m 0s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	16m 44s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	1m 9s		The patch does not generate ASF License warnings.
		453m 8s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.TestFileCreation
	hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/1/artifact/out/Dockerfile
GITHUB PR	#7208
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle cc buflint bufcompat
uname	Linux 632c2fd9f2c8 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 60bc95a
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/1/testReport/
Max. process+thread count	3777 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	14m 9s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 35s		branch-3.3 passed
+1 💚	compile	10m 44s		branch-3.3 passed
+1 💚	checkstyle	1m 26s		branch-3.3 passed
+1 💚	mvnsite	3m 25s		branch-3.3 passed
+1 💚	javadoc	3m 10s		branch-3.3 passed
-1 ❌	spotbugs	1m 29s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 💚	shadedclient	20m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 17s		the patch passed
+1 💚	compile	10m 30s		the patch passed
+1 💚	cc	10m 30s		the patch passed
+1 💚	javac	10m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 25s	/results-checkstyle-root.txt	root: The patch generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417)
+1 💚	mvnsite	3m 19s		the patch passed
+1 💚	javadoc	3m 0s		the patch passed
+1 💚	spotbugs	6m 14s		the patch passed
+1 💚	shadedclient	20m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 10s		hadoop-common in the patch passed.
+1 💚	unit	1m 56s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	170m 41s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	14m 35s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		337m 23s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
	hadoop.hdfs.TestReconstructStripedFileWithValidator
	hadoop.hdfs.server.datanode.TestLargeBlockReport

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/2/artifact/out/Dockerfile
GITHUB PR	#7208
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle cc buflint bufcompat
uname	Linux be5a920b0d05 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / b7a3139
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/2/testReport/
Max. process+thread count	4714 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	15m 22s		Maven dependency ordering for branch
+1 💚	mvninstall	33m 58s		branch-3.3 passed
+1 💚	compile	17m 51s		branch-3.3 passed
+1 💚	checkstyle	2m 45s		branch-3.3 passed
+1 💚	mvnsite	5m 18s		branch-3.3 passed
+1 💚	javadoc	4m 41s		branch-3.3 passed
-1 ❌	spotbugs	2m 34s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in branch-3.3 has 2 extant spotbugs warnings.
+1 💚	shadedclient	35m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 38s		the patch passed
+1 💚	compile	17m 9s		the patch passed
+1 💚	cc	17m 9s		the patch passed
+1 💚	javac	17m 9s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 40s	/results-checkstyle-root.txt	root: The patch generated 1 new + 416 unchanged - 1 fixed = 417 total (was 417)
+1 💚	mvnsite	5m 22s		the patch passed
+1 💚	javadoc	4m 42s		the patch passed
+1 💚	spotbugs	10m 37s		the patch passed
+1 💚	shadedclient	34m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 3s		hadoop-common in the patch passed.
+1 💚	unit	2m 33s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	195m 56s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	unit	16m 50s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	1m 13s		The patch does not generate ASF License warnings.
		447m 41s

Reason	Tests
Failed junit tests	hadoop.hdfs.protocol.TestBlockListAsLongs
	hadoop.hdfs.server.datanode.TestLargeBlockReport
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/3/artifact/out/Dockerfile
GITHUB PR	#7208
Optional Tests	dupname asflicense mvnsite codespell detsecrets markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle cc buflint bufcompat
uname	Linux 8efe64e74005 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / e68c0ac
Default Java	Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/3/testReport/
Max. process+thread count	3810 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7208/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[ctrezzo]

@PrateekSane this seems to be a backwards incompatible change as we are adding a new required field to ClientNamenodeProtocol.proto. As such, I am not sure we can safely backport this to the 3.x line.