add unicode case folding for char/str

[thestinger]

[kud1ing]

For reference:

"Case mapping or case conversion is a process whereby strings are converted to a particular form—uppercase, lowercase, or titlecase—possibly for display to the user. Case folding is primarily used for caseless comparison of text [...] As a result, case-folded text should be used solely for internal processing and generally should not be stored or displayed to the end user."

http://www.unicode.org/faq/casemap_charprop.html

[pzol]

Char upper/lower is done in #12561

For str, what would be more appropriate - an iterator or a method with the converted string?
An iterator would allow also case-insensitive comparisons, something like this:

struct LowerChars<'a> {
  chars: std::str::Chars<'a>
}

fn lower<'a>(s: &'a str) -> LowerChars<'a> {
  LowerChars { chars: s.chars() }
}

impl<'a> Iterator<char> for LowerChars<'a> {
  fn next(&mut self) -> Option<char> {
    self.chars.next().map(|c| c.to_lowercase())
  }
}

#[test]
fn test_to_uppercase(){
  let sl = "foobär";
  let su = "FOOBÄR";

  let mut z = lower(sl).zip(lower(su));
  assert!(z.all(|(x, y)| x == y ));

  let greek = "στιγμας".chars().map(|c| c.to_uppercase()).collect::<~str>();
  // fail!(greek);
  assert_eq!(greek.as_slice(), "ΣΤΙΓΜΑΣ");
}

[Kimundi]

@pzol: The normalization functions already return Iterators, so I'd choose them for case folding too.
Collecting into a actual string is easy then.
You could also provide none one-to-one case folding that way.

[pzol]

Kimundi so I'd suggest

pub trait StrSlice<'a> {
...
    fn lower_chars<'a>(s: &'a str) -> LowerChars<'a>;
    fn upper_chars<'a>(s: &'a str) -> UpperChars<'a>;
}

[Kimundi]

Yeah, that would fit in nicely

[pzol]

Now, for char I only implemented common and simple case folding. That is, one char is always one char. for THIS however, multi-codepoint mapping could be done, i.e. one codepoint can become two codepoints.

However, for case insensitive comparison afaik only ICU gets it right.
CaseInsensitiveString.equals gets the right answers where regular Java fails; for example:

• "flour and water" & "FLOUR AND WATER"
• "efficient" & "EFFICIENT"
• "poſt" & "post"
• "weiß" & "WEIẞ"
• "tschüß" & "TSCHÜSS"
• "ᾲ στο διάολο" & "Ὰͅ Στο Διάολο"
• "ᾲ στο διάολο" & "ᾺΙ ΣΤΟ ΔΙΆΟΛΟ"

So in other words for str a method like case_insensitive_compare would be appropriate.
Question is, should Equiv for str consider case folding?

[Kimundi]

The Equiv trait only exists to allow uniform comparison between ~str, &str etc, I don't think it should ignore case. That is, it exists to compare different types on identical content.

[pzol]

what about this:

pub trait StrSlice<'a> {
...
    fn equals_ignore_case(&self, needle: &str) -> bool;
    fn starts_with_ignore_case(&self, needle: &str) -> bool;
    fn ends_with_ignore_case(&self, needle: &str) -> bool;
// or 
    fn equals_ignore_case(&self, needle: &str) -> bool;
...
// or
  fn compare(&self, needle: &str, ignore_case: bool) -> bool;
...
}

I like the last one best.

[Kimundi]

Not sure... Adding a bunch of method combinations smells like a missed abstraction that could be integrated somehow. I'd add at most a eq_ignore_case function for now.

[pzol]

This is also relevant http://www.w3.org/International/wiki/Case_folding

If my understanding is right, then a proper folding might require providing a language context in order to be correct. My suggestion would be start with a normal case folding with this table

http://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt