Fixes for use with Pandas 1.2.1 #171
Conversation
|
@frreiss there is still one more error, it has to do with TensorArray repr() with floating point values, and is similar to pandas-dev/pandas#38391. I need to get back on that for a fix, but I'll try for a workaround first. We can merge this first if ok with you. |
| """ | ||
| if isinstance(item, Span) and \ | ||
| item.begin == Span.NULL_OFFSET_VALUE: | ||
| return Span.NULL_OFFSET_VALUE in self._begins |
There was a problem hiding this comment.
This is needed because otherwise the default is to check all values for equality, and that will fail if there is a NULL value with Span.target_text=None
There was a problem hiding this comment.
Do you know how Pandas defines the result of NA == NA for other nullable types? Should that expression evaluate True, False, or NA? It could go either way.
There was a problem hiding this comment.
Looks like the current Pandas policy is "¯_(ツ)_/¯"; see the warning on https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html
There was a problem hiding this comment.
Yeah, good question. The test failure related to this was checking assert na_value in data_missing which expects that na_value == na_value to be True. Seems like it would be different for np.nan though..
| f"objects to a TokenSpanArray. Found an " | ||
| f"object of type {type(s)}" | ||
| ) | ||
| if tokens is None and not (s.begin == TokenSpan.NULL_OFFSET_VALUE and s.tokens.target_text is None): |
There was a problem hiding this comment.
This gets a little awkward dealing with with a NULL value with target_text=None, but should be ok for now
There was a problem hiding this comment.
This case will work better once TokenSpanArray allows multiple target texts/tokenization pairs. Still some design work to be done on that front, methinks.
| """ | ||
| if isinstance(item, Span) and \ | ||
| item.begin == Span.NULL_OFFSET_VALUE: | ||
| return Span.NULL_OFFSET_VALUE in self._begins |
There was a problem hiding this comment.
Do you know how Pandas defines the result of NA == NA for other nullable types? Should that expression evaluate True, False, or NA? It could go either way.
| f"objects to a TokenSpanArray. Found an " | ||
| f"object of type {type(s)}" | ||
| ) | ||
| if tokens is None and not (s.begin == TokenSpan.NULL_OFFSET_VALUE and s.tokens.target_text is None): |
There was a problem hiding this comment.
This case will work better once TokenSpanArray allows multiple target texts/tokenization pairs. Still some design work to be done on that front, methinks.
|
Going to go ahead and merge, will try to fix the TensorArray issue separately |
2 participants
Fixed
_from_sequenceto handle np.nan, added__contains__to check for null values with text set toNone.Closes #168