-
Notifications
You must be signed in to change notification settings - Fork 7.1k
Modify expected value and threshold for retinanet unit-test. #2812
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2812 +/- ##
=======================================
Coverage 73.26% 73.26%
=======================================
Files 99 99
Lines 8778 8778
Branches 1387 1387
=======================================
Hits 6431 6431
Misses 1920 1920
Partials 427 427 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks a lot !
Let's just wait until CircleCI gets back to normal before merging this PR.
Test failures seem related (and happen only on GPU). Maybe there are duplicate detections somewhere and due to floating point errors we get a slightly off prediction? If debugging / fixing this is hard, we can skip the numerical tests for retinanet on the GPU (while performing the other tests), and re-enable those later? |
@fmassa I agree. This is likely due to floating point errors, possibly made worse from the fact that all weights and input data are random. I'll have another look, hopefully we won't need to disable the GPU tests. |
7ba16e0
to
5aee174
Compare
Unfortunately there is numerical and sort instability on this unit-test across platforms. Given that currently on master we don't sufficiently test retinanet, I think it's worth putting at least some tests in. As a workaround I disable some of the cuda checks but I plan to revisit this once I have a proper GPU setup. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, thanks a lot @datumbox !
…#2812) * Modify expected value and threshold for retinanet unit-test. * Disable tests on GPU Co-authored-by: Francisco Massa <[email protected]>
…#2812) * Modify expected value and threshold for retinanet unit-test. * Disable tests on GPU Co-authored-by: Francisco Massa <[email protected]>
Updating the score_thresh=0.013 for RetinaNet returns non-empty results. This fixes #2810.