Skip to content

Conversation

@dkhalanskyjb
Copy link
Collaborator

Not sure what the test checks for. Does it really need to make all these iterations, or just waiting for every type of exception to fire is enough?

@dkhalanskyjb dkhalanskyjb requested a review from qwwdfsad June 21, 2022 08:55
@dkhalanskyjb dkhalanskyjb changed the base branch from master to develop June 21, 2022 08:55
@dkhalanskyjb dkhalanskyjb changed the base branch from develop to master June 21, 2022 08:56
var loopsToMake = iterations

repeat(iterations) {
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0 || loopsToMake-- > 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rationale behind this test was the following: given enough parallelism (3 threads) and iterations, we expect to observe every exception type at least once.

Taking into account recent changes in our CI and the nature of this test, I would suggest changing it to be more robust, but keep the original intent: iterate until all exception types are observed OR really many iterations passed (to address flakyness), but add a large enough timeout so CI won't hang for four hours

Copy link
Collaborator Author

@dkhalanskyjb dkhalanskyjb Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the rationale, I don't understand the need to count iterations, as opposed to just doing this, with something like withTimeout wrapping everything:

Suggested change
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0 || loopsToMake-- > 0) {
while (eCnt == 0 || e1Cnt == 0 || e2Cnt == 0) {

Copy link
Member

@qwwdfsad qwwdfsad Jun 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about iterations for fast debugging locally and a large enough timeout for slow CI, but this should work as well, up to you

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think not having iterations, in this case, is more semantically clear. Most other stress tests perform many iterations so that a problematic condition has more chances to arise. Here, it's not the case: we want to wait for a success condition. Waiting for a given number of iterations obscures this, whereas the concept of a timeout is clear and self-evident.

@dkhalanskyjb dkhalanskyjb requested a review from qwwdfsad June 21, 2022 10:51
@dkhalanskyjb dkhalanskyjb merged commit 11daf82 into master Jun 22, 2022
@dkhalanskyjb dkhalanskyjb deleted the fix-flaky-test branch June 22, 2022 08:29
mpetrov pushed a commit to mpetrov/kotlinx.coroutines that referenced this pull request Jun 23, 2022
pablobaxter pushed a commit to pablobaxter/kotlinx.coroutines that referenced this pull request Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants