sync: TestWaitGroupMisuse2 test is flaky

[kostya-sh]

Using tip (ca91de7) TestWaitGroupMisuse2 test fails approximately 19 times out of 20 on my PC:

ok      strings 0.104s
--- FAIL: TestWaitGroupMisuse2 (1.41s)
    waitgroup_test.go:110: Should panic
    waitgroup_test.go:80: Unexpected panic: <nil>
FAIL
FAIL    sync    1.558s
ok      sync/atomic 0.

[bradfitz]

Could you describe your PC?

[bradfitz]

How many CPUs? Which? Which OS, OS version, arch (32-bit or 64-bit)?

Dmitry, maybe 1e6 should be bigger in the test?

/cc @dvyukov

[kostya-sh]

OS: 64-bit Debian 8.1 running on VMWare player 7.1 (host OS Windows 8).

Hardware: 4 CPUs (Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz)

[kostya-sh]

I've just tested it under Windows 8 OS (the same hardware). The test is still flaky though success rate is higher.

[dvyukov]

@kostya-sh What is the iteration count in the test at which it stops being flaky?

[bradfitz]

And how long does the test take to run at said iteration count?

[marete]

It fails reliably on my Linux laptop in an LXC container running Ubuntu 14.10 (3/3 times):

ok strings 0.156s
--- FAIL: TestWaitGroupMisuse2 (3.25s)
waitgroup_test.go:110: Should panic
waitgroup_test.go:80: Unexpected panic:
FAIL
FAIL sync 3.410s
ok sync/atomic 0.897s

My CPU's details (this is a Nehalem class mobile CPU):

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 37
Model name: Intel(R) Core(TM) i5 CPU M 460 @ 2.53GHz
Stepping: 5
CPU MHz: 2528.000
CPU max MHz: 2528.0000
CPU min MHz: 1197.0000
BogoMIPS: 5056.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 3072K
NUMA node0 CPU(s): 0-3

My OS's details:

Ubuntu Linux 14.10 (x86_64), running kernel 4.0.6 (Linus' mainline) in an unprivileged LXC container within an Ubuntu 15.04 (x86_64) host:

marebri@utopic:~/devel/go.git/src$ uname -a
Linux utopic 4.0.6 #1 SMP Wed Jun 24 01:11:35 EAT 2015 x86_64 x86_64 x86_64 GNU/Linux

Other information:

I reproduced this on the Ubuntu 15.04 (x86_64) host (failed 1/3 times):

ok strings 0.179s
--- FAIL: TestWaitGroupMisuse2 (3.32s)
waitgroup_test.go:110: Should panic
waitgroup_test.go:80: Unexpected panic:
FAIL
FAIL sync 3.493s
ok sync/atomic 1.018s

vs the passing 2 tests:

ok strings 0.236s
ok sync 2.478s
ok sync/atomic 0.821s
ok syscall 0.131s

ok strings 0.312s
ok sync 0.452s
ok sync/atomic 1.025s
ok syscall 0.151s

EDIT: Both host and LXC container are on the current git tip: d0ed87d

[mikioh]

We can see this on the dragonfly buildbot: http://build.golang.org/log/fd18334c684f5cec5c7d4f939c39f26ec7c30741 by 03a48eb.

[kostya-sh]

Even with 8e7 iterations the test is still flaky on x86_64 Debian VMWare VM. It takes about 100 seconds to fail with this number of iterations.

It could take any time between 0.5 sec to 50 seconds for the test to succeed.

[iworker]

I reproduced this on Ubuntu 15.04 vivid (4 times)

--- FAIL: TestWaitGroupMisuse2 (2.77s)
    waitgroup_test.go:110: Should panic
    waitgroup_test.go:80: Unexpected panic: <nil>
FAIL
FAIL    sync    2.917s
....
2015/06/29 13:50:33 Failed: exit status 1

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz
Stepping:              4
CPU MHz:               1207.269
CPU max MHz:           3900.0000
CPU min MHz:           1200.0000
BogoMIPS:              7400.11
Virtualisation:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              10240K
NUMA node0 CPU(s):     0-7

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 15.04
Release:    15.04
Codename:   vivid

Linux *** 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

After all 1e6 iterations there was no one expected panic.

[xoba]

i've also seen the failure on a m3.xlarge aws ec2 instance (4 vCPU, 15 GiB), running ami-ee793a86 (ubuntu), golang commit a76c1a5:

ok cmd/nm 1.763s
ok cmd/objdump 4.847s
ok cmd/pack 3.671s
ok cmd/pprof/internal/profile 0.030s
ok cmd/vet 4.370s

GOMAXPROCS=2 runtime -cpu=1,2,4

ok runtime 36.327s

sync -cpu=10

--- FAIL: TestWaitGroupMisuse2 (1.58s)
waitgroup_test.go:110: Should panic
waitgroup_test.go:80: Unexpected panic:
FAIL
FAIL sync 1.682s
2015/06/29 10:28:14 Failed: exit status 1

[iworker]

@dvyukov with iteration count 1e9 this test didn't fails:

ok      sync    6.109s

But I think it's not good choice to increase iterations count. Maybe better to introduce some better way to get panic with more probability. Actually, maybe iterations count must depends on CPU's count. For example, 8 CPU's – 1e9, 2 CPU's – 1e6.