HADOOP-19737: ABFS: Add metrics to identify improvements with read and write aggressiveness #8056
+1,287
−100
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
manika137
reviewed
Nov 12, 2025
| */ | ||
| @Override | ||
| public AbfsWriteThreadPoolMetrics getAbfsWriteThreadPoolMetrics() { | ||
| return abfsWriteThreadPoolMetrics != null ? abfsWriteThreadPoolMetrics : null; |
Contributor
There was a problem hiding this comment.
we can have it as- return abfsWriteThreadPoolMetrics;
Same for getAbfsReadThreadPoolMetrics() below
manika137
reviewed
Nov 18, 2025
| if (lastTime == ZERO) { | ||
| lastCpuTime = cpuTime; | ||
| lastTime = now; | ||
| return 0.0; // first call has no previous data |
Contributor
There was a problem hiding this comment.
nit: we can use ZERO_D here too
manika137
reviewed
Nov 18, 2025
| * Closes this manager by shutting down executors and cleaning up resources. | ||
| * Removes the instance from the active manager map. | ||
| * | ||
| * @throws IOException if an error occurs during shutdown. |
Contributor
There was a problem hiding this comment.
we are throwing this exception right?
manika137
reviewed
Nov 18, 2025
| public String toString() { | ||
| return String.format( | ||
| "currentPoolSize=%d, maxPoolSize=%d, activeThreads=%d, jvmCpuUtilization=%.2f%%, cpuUtilization=%.2f%%, availableHeap=%dGB", | ||
| currentPoolSize, maxPoolSize, activeThreads, jvmCpuUtilization, cpuUtilization * HUNDRED, availableHeapGB); |
Contributor
There was a problem hiding this comment.
we dont need to multiply jvmCpuUtilization also by 100?
Also, should we be multiplying with HUNDRED_D instead?
manika137
reviewed
Nov 19, 2025
| configuration1.set(FS_AZURE_ENABLE_PREFETCH_REQUEST_PRIORITY, "true"); | ||
|
|
||
| Configuration configuration2 = new Configuration(getRawConfiguration()); | ||
| //use the default value for the config: false |
Contributor
There was a problem hiding this comment.
we can remove the comment as well. The default traffic priority value is true now
manika137
reviewed
Nov 19, 2025
| if (lastTime == ZERO) { | ||
| lastCpuTime = cpuTime; | ||
| lastTime = now; | ||
| return 0.0; // first call has no previous data |
manika137
reviewed
Nov 19, 2025
| @Test | ||
| public void verifyWriteRequestOfBufferSizeAndClose() throws Exception { | ||
|
|
||
| AbfsCounters abfsCounters = Mockito.spy(new AbfsCountersImpl(new URI("abcd"))); |
Contributor
There was a problem hiding this comment.
we can have a constant for URI "abcd"
Contributor
There was a problem hiding this comment.
if creating mock client handlers, client, abfsrestop, spying abfscounters is repeated- then we could have it as a helper method maybe
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces new performance metrics in the ABFS driver to monitor and evaluate the effectiveness of read and write aggressiveness tuning. These metrics help in understanding how thread pool behavior, CPU utilization, and heap availability impact overall I/O throughput and latency. By capturing detailed statistics such as active thread count, pool size, and system resource utilization, this enhancement enables data-driven analysis of optimizations made to improve ABFS read and write performance under varying workloads.