Flaky Windows Test

[AlexanderWells-diamond]

While working on PR #170 to add Python3.13 to Windows, I found that there seems to be a flaky test on this platform only. An example failure in CI can be seen here.

I attempted to fix this in 12b657a, which I think has reduced the failure rate but I'm not certain.

The failing test is p4p.test.test_gw.TestHighLevelGetHoldOff.test_get_holdoff. It seems that every test after that also fails due to some state not being cleaned up.

In case the logs get cleaned up, here's the relevant part:

Complete Logs

   ======================================================================
FAIL: test_get_holdoff (p4p.test.test_gw.TestHighLevelGetHoldOff.test_get_holdoff)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\utils.py", line 104, in wrapper
    return meth(*args, **kws)
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\test_gw.py", line 579, in test_get_holdoff
    self.assertTrue(Vmin==N and
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^
                    Vmax<=N+1 and
                    ^^^^^^^^^^^^^
                    T2-T1 >= self.getholdoff and
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    T2-T0 >= (Vmax - Vmin + 1) * self.getholdoff,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    "%.3f, %.3f : %d %s"%(T1-T0, T2-T0, N, Vs))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: False is not true : 4.032, 5.032 : 5 [5, 5, 5]

======================================================================
FAIL: test_get_mask (p4p.test.test_gw.TestHighLevelGetHoldOff.test_get_mask)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\test_gw.py", line 272, in setUp
    super(TestHighLevel, self).setUp()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\utils.py", line 112, in setUp
    super(RefTestCase, self).setUp()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\utils.py", line 54, in setUp
    self.fail('Leftovers from previous test: %s = %d'%(mustzero, self.__before[mustzero]))
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Leftovers from previous test: ClientContextImpl = 1

======================================================================
FAIL: test_mon (p4p.test.test_gw.TestHighLevelGetHoldOff.test_mon)
Setup a monitor through the GW
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\test_gw.py", line 272, in setUp
    super(TestHighLevel, self).setUp()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\utils.py", line 112, in setUp
    super(RefTestCase, self).setUp()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\hostedtoolcache\windows\Python\3.13.5\x64\Lib\site-packages\p4p\test\utils.py", line 54, in setUp
    self.fail('Leftovers from previous test: %s = %d'%(mustzero, self.__before[mustzero]))
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Leftovers from previous test: ClientContextImpl = 1

[mdavidsaver]

I have seen this issue before. I don't think it signals an actual bug with the GET holdoff feature, but have not had time to dig into this test case again.

I attempted to fix this in 12b657a

Increasing getholdoff for this test is unlikely to be a "fix".

It might be worth splitting this one big test condition into 4 separate conditions as long as the full set of inputs is printed for each.

It seems that every test after that also fails due to some state not being cleaned up.

This is known and expected. In order to detect resource leaks in C extension objects, some per-class instance counters are checked both before and after each case. This works great to detect leaks from tests which would otherwise pass. However, when a test fails, a bunch of objects become referenced in Exception frames, which escape the lifetime of that case. This leaves a bit of a mess, the exact details of which depend on the test runner used.