MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter #2500
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
aajisaka
requested changes
Aug 30, 2021
Member
aajisaka
left a comment
aajisaka
left a comment
There was a problem hiding this comment.
If the test case fails before L327, the statistics is not cleared. Would you create an @After method and clear the statistics there? That way the statistics is alway cleared.
Contributor
Author
@aajisaka Good point. Updated the fix accordingly. Thanks! |
|
🎊 +1 overall
This message was automatically generated. |
Member
|
Merged. Thank you @lzx404243 for stabilizing tests. |
aajisaka
pushed a commit
that referenced
this pull request
Sep 1, 2021
…ogressReporter (#2500) Co-authored-by: Zhengxi Li <[email protected]> (cherry picked from commit 6187f76)
aajisaka
pushed a commit
that referenced
this pull request
Sep 1, 2021
…ogressReporter (#2500) Co-authored-by: Zhengxi Li <[email protected]> (cherry picked from commit 6187f76)
kiran-maturi
pushed a commit
to kiran-maturi/hadoop
that referenced
this pull request
Nov 24, 2021
…ogressReporter (apache#2500) Co-authored-by: Zhengxi Li <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The test
org.apache.hadoop.mapred.TestTaskProgressReporter.testBytesWrittenRespectingLimitis not idempotent and fails if run twice in the same JVM, because it pollutes state shared among tests. It may be good to clean this state pollution so that some other tests do not fail in the future due to the shared state polluted by this test.Details
Running
TestTaskProgressReporter.testBytesWrittenRespectingLimittwice would result in the second run failing with the following assertion:The root cause for this is that when
testBytesWrittenRespectingLimitwrites some bytes on the local file system, some counters are being incremented. The problem is that, after the test is done, the counter is not reset. With this polluted shared state, assumptions are broken, resulting in test failure in the second run.The fix is to invoke
FileSystem.clearStatistics()to reset the counter(among others statistics) when the test finishes.With the proposed fix, the test does not pollute the shared state (and passes when run twice in the same JVM).
Jira link: MAPREDUCE-7311