Improve docs for sentence segmentation #86
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Seems like those two functions got mixed up when the documentation was written. `unicode_sentences` is the one that filters for alphanumeric characters.
Manishearth
reviewed
Oct 4, 2020
| /// | ||
| /// assert_eq!(&us1[..], b); | ||
| /// ``` | ||
| fn unicode_sentences<'a>(&'a self) -> UnicodeSentences<'a>; |
There was a problem hiding this comment.
Can we add some text explaining what this function does?
There was a problem hiding this comment.
I'm afraid I don't understand. Do you mean I should expand on
Returns an iterator over substrings of
selfseparated on
UAX#29 sentence boundaries.Here, "sentences" are just those substrings which, after splitting on
UAX#29 sentence boundaries, contain any alphanumeric characters. That is, the
substring must contain at least one character with the
Alphabetic
property, or with
General_Category=Number.
above the example?
(I'm wondering whether the way the diff is rendered might be confusing you.)
There was a problem hiding this comment.
oh, sorry, this was absolutely diff rendering stuff 😄
|
Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It looks like the doc comments for
unicode_sentencesandsplit_sentence_boundsgot mixed up. I swapped them so that each function is described correctly.I also added examples illustrating the difference between the three functions for segmenting sentences.