-
Notifications
You must be signed in to change notification settings - Fork 11
Add basic polygon metrics to nucleus repo #185
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This pull request has been linked to: |
Edit: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very clear code to read 🙂
I've requested changes since we need to be sure that Python 3.6 users see a warning message to upgrade. Until then I'm not comfortable rolling this out since I know that python3.6 is default in some widely used distros.
4a72a74
to
684b833
Compare
Moved the new dependencies to dev only, with the Python requirement applied specifically to those dependencies. |
def __init__( | ||
self, | ||
enforce_label_match: bool = False, | ||
iou_threshold: float = 0.0, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to have defaults for the IOU and confidence thresholds? Those seem like things we'd want the user to specify.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My thinking is that we can expose information about what defaults are used in Model CI in two ways:
- Currently, we can keep these defaults in code so that users who want to see what the values are can find them directly here as defaults.
- Once the UI shows the source code of evaluation functions, we can get rid of the defaults here and instead show them in the source code (e.g.
PolygonIOU(enforce_label_match=True, iou_threshold=0.2, confidence_threshold=0.5)
).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case, I'll add a TODO here in polygon_metrics.py
to remove the defaults once we can surface these defaults more directly to the user.
def __init__( | ||
self, | ||
enforce_label_match: bool = False, | ||
iou_threshold: float = 0.5, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here. Also wondering why this is 0.5
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea is that in Nucleus and the most commonly reported metrics, it's usually 0.5
by default. Also if we're exposing these evaluation functions to users, we would want users to know the default values that are actually used in Model CI.
ce4d0e3
to
d581fa9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍 🔥
These tests are failing on master as well so they shouldn't block you. Hopefully I'll have a fix today and then you can simply rebase on top.
Co-authored-by: Gunnar Atli Thoroddsen <[email protected]>
Rebased on top of master ... |
Adds IoU, precision, and recall metrics to the Nucleus repository. Creates framework to support more general metrics.
Note: adding the
Shapely
package also adds a dependency on thegeos
library, which can be installed in MacOS usingbrew install geos
.Design doc here: https://docs.google.com/document/d/1uOjZQ9wRMdpsOaYN_w6eTqPvQu0Nwtl5wRNj0HkL0EU/edit#heading=h.raendxyxrl42
ch318548
ch303550