HDFS-17768. Observer namenode network delay causing empty block location for getBatchedListing

[dParikesit]

Description of PR

Jira: HDFS-17768

In our testing with the latest hdfs version (e8a64d0), we found a similar case to HDFS-16732 happening in getBatchedListing. During a getBatchedListing, if the block report of the observer nn is delayed, one or more of the listing results will return blocks without location.

Steps to reproduce this bug:

1. Start a cluster with 1 observer namenode
2. Create an empty file
3. Inject network delay between observer nn and active nn to delay block report (or add sleep to the BlockReportProcessingThread of the observer).
4. Append file to add block
5. Send a batchedListPaths request using client API
6. Check that the result has block without location

In HDFS-16732 and HDFS-13924, a check was added in getBlockLocations, getFileInfo, and getListing that checks whether the found blocks have valid locations. Missing locations indicate that the observer namenode is not up-to-date compared to the active namenode.

We propose to add the same check to getBatchedListing. If any of the sub-listing return blocks without location then it will throw ObserverRetryOnActiveException and exit the function early. The entire batchedListing request will be then retried on active namenode.

Your insights are very much appreciated. We will continue following up this issue until it is resolved.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	14m 24s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 6s		trunk passed
+1 💚	compile	1m 20s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	checkstyle	1m 14s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
+1 💚	javadoc	1m 11s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 40s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	spotbugs	3m 16s		trunk passed
+1 💚	shadedclient	39m 40s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 0s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 58s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 37s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚	spotbugs	3m 20s		the patch passed
+1 💚	shadedclient	38m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 36s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		153m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7593/1/artifact/out/Dockerfile
GITHUB PR	#7593
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ed4243b09a13 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / b15b473
Default Java	Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7593/1/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7593/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[jojochuang]

+1

[dParikesit]

Thanks for voting to approve my pr, @jojochuang! Feel free to let me know if there’s anything else I could do to help the review and merging process. Happy to help!