Merge pull request #12 from sparsetech/feat/to-latin

Implement Cyrillic to Latin conversion

Author: tindzk

README.md

@@ -2,7 +2,7 @@
 [![Build Status](http://ci.sparse.tech/api/badges/sparsetech/translit-scala/status.svg)](http://ci.sparse.tech/sparsetech/translit-scala)
 [![Maven Central](https://img.shields.io/maven-central/v/tech.sparse/translit-scala_2.12.svg)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22tech.sparse%22%20AND%20a%3A%22translit-scala_2.12%22)
 
-translit-scala is a transliteration library for Scala and Scala.js. It implements transliteration rules for Slavic languages. It supports converting texts from the Latin to the Cyrillic alphabet.
+translit-scala is a transliteration library for Scala and Scala.js. It implements transliteration rules for Slavic languages. It supports converting texts from the Latin to the Cyrillic alphabet and vice-versa.
 
 ## Compatibility
 | Back end   | Scala versions |
@@ -52,7 +52,7 @@ We decompose letters in their Latin transliteration more consistently than Natio
 * Volodymyr (Володимир)
 * blyz'ko (близько)
 
-The Latin letter *y* is also the phonetic basis of four letters in the Slavic alphabet: я, є, ї, ю. They get transliterated accordingly:
+The Latin letter *y* forms the phonetic basis of four letters (iotated vowels) in the Ukrainian alphabet: я, є, ї, ю. They get transliterated accordingly:
 
 * ya → я
 * ye → є
@@ -63,20 +63,23 @@ Unlike National 2010, we always use the same transliteration regardless of the p
 
 The accented counterpart of и is й and is represented by a separate letter, *j*.
 
-*Example:* Zhurs'kyj (Згурський)
+*Example:* Zgurs'kyj (Згурський)
 
 #### Soft Signs and Apostrophes
 The second change to National 2010 is that we try to restore soft signs and apostrophes:
 
 * Ukrayins'kyj (Український), malen'kyj (маленький)
 * m'yaso (м'ясо), matir'yu (матір'ю)
 
+In National 2010, *g* gets mapped to *ґ* which is phonetically accurate, though the letter *ґ* is fairly uncommon in Ukrainian. Therefore, we represent *ґ* by the bi-gram *g'*.
+
 This feature is experimental and can be disabled by setting `apostrophes` to `false`.
 
 #### Convenience mappings
 Another modification was to provide the following mappings:
 
 * c → ц
+* h → х
 * q → щ
 * w → ш
 * x → ж
@@ -91,9 +94,8 @@ Note that these mappings are phonetically inaccurate. However, using them still
 * Another advantage is the proximity on the English keyboard layout:
     * *q* and *w* are located next to each other; *ш* and *щ* characters are phonetically close
     * *z* and *x* are located next to each other; *з* and *ж* characters are phonetically close
-
-#### Precedence
-The replacement patterns are applied sequentially by traversing the input character-by-character. In some cases, a rule spanning multiple characters should not be applied. An example is the word: схильність. The transliteration of *сх* corresponds to two separate letters *s* and *h*, which would map to *ш*. To prevent this, one can place a vertical bar between the two characters. The full transliteration then looks as follows: *s|hyl'nist*
+* *h* is mapped to *х* since it is a common letter, *kh* is only needed in case *h* is ambiguous
+	* An example is the word: схильність. The transliteration of *сх* corresponds to two separate letters *s* and *h*, which would map to *ш*. To prevent this, one can use the bi-gram *kh* instead to represent *х*. The full transliteration then looks as follows: *skhyl'nist*
 
 ## Russian
 The Russian rules are similar to the Ukrainian ones.
@@ -103,13 +105,7 @@ Some differences are:
 * *i* corresponds to *и*, whereas *y* to *ы*
 * Russian distinguishes between soft and hard signs. It does not have apostrophes. The following mappings are used:
   * Soft sign: *'* for ь
-  * Hard sign: *"* for ъ
-
-### Precedence
-As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules from being applied.
-
-* красивые: krasivy|e
-* сходить: s|hodit
+  * Hard sign: *`* for ъ
 
 ### Mapping
 | Latin | Cyrillic |
@@ -121,7 +117,7 @@ As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules
 | e     | е        |
 | f     | ф        |
 | g     | г        |
-| h     | х        |
+| h, kh | х        |
 | i     | и        |
 | j     | й        |
 | k     | к        |
@@ -141,7 +137,7 @@ As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules
 | y     | ы        |
 | z     | з        |
 | '     | ь        |
-| "     | ъ        |
+| \`    | ъ        |
 | ch    | ч        |
 | sh    | ш        |
 | ya    | я        |
@@ -151,15 +147,30 @@ As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules
 | yu    | ю        |
 | shch  | щ        |
 
-#### Examples
-| Russian | Transliterated |
-|---------|----------------|
-| Привет  | Privet         |
-| Съел    | S"el           |
-| Щётка   | Shchyotka      |
-| Льдина  | L'dina         |
+### Examples
+| Russian  | Transliterated |
+|----------|----------------|
+| Привет   | Privet         |
+| Съел     | S\`el          |
+| Щётка    | Shchyotka      |
+| Льдина   | L'dina         |
+| красивые | krasivye       |
+| сходить  | skhodit'       |
+
+## Internals
+The replacement patterns are applied sequentially by traversing the input character-by-character. The functions `latinToCyrillicIncremental` and `cyrillicToLatinIncremental` take the left context which is needed by some rules, for example to determine the correct case of soft/hard signs. The result of the functions indicates the number of characters to remove on the right as well as their string replacement.
+
+```scala
+def latinToCyrillicIncremental(
+  latin: String, cyrillic: String, append: Char
+): (Int, String)
+
+def cyrillicToLatinIncremental(
+  cyrillic: String, letter: Char
+): (Int, String)
+```
 
-### Credits
+## Credits
 The rules and examples were adapted from the following libraries:
 
 * [translit-english-ukrainian](https://github.com/MarkovSergii/translit-english-ukrainian)

shared/src/main/scala/translit/Helpers.scala

@@ -1,6 +1,9 @@
 package translit
 
 object Helpers {
+  def applyCase(str: String, isUpper: Boolean): String =
+    if (isUpper) str(0).toUpper + str.tail else str
+
   def restoreCaseAll(str: String, cyrillic: Char): Char =
     if (str.forall(_.isUpper)) cyrillic.toUpper else cyrillic
 

shared/src/main/scala/translit/Language.scala

@@ -15,6 +15,8 @@ trait Language {
     latin: String, cyrillic: String, append: Char
   ): (Int, String)
 
+  def cyrillicToLatinIncremental(cyrillic: String, letter: Char): (Int, String)
+
   def latinToCyrillic(text: String): String = {
     val result = new StringBuilder(text.length)
     var offset = 0
@@ -29,4 +31,19 @@ trait Language {
 
     result.mkString
   }
+
+  def cyrillicToLatin(text: String): String = {
+    val result = new StringBuilder(text.length * 2)
+    var offset = 0
+
+    while (offset < text.length) {
+      val (length, c) = cyrillicToLatinIncremental(
+        text.take(offset), text(offset))
+      if (length < 0) result.setLength(result.length + length)
+      result.append(c)
+      offset += 1
+    }
+
+    result.mkString
+  }
 }

shared/src/main/scala/translit/Noop.scala

@@ -4,5 +4,9 @@ object Noop extends translit.Language {
   override def latinToCyrillicIncremental(
     latin: String, cyrillic: String, append: Char
   ): (Int, String) = (0, append.toString)
+
+  override def cyrillicToLatinIncremental(
+   cyrillic: String, letter: Char
+  ): (Int, String) = (0, letter.toString)
 }
 

shared/src/main/scala/translit/Russian.scala

@@ -29,9 +29,13 @@ object Russian extends Language {
     'w' -> 'ш',
     'x' -> 'ж',
     'y' -> 'ы',
-    'z' -> 'з',
+    'z' -> 'з'
+  )
+
+  // Infer case from previous character
+  val uniGramsSpecial = Map(
     '\'' -> 'ь',
-    '"' -> 'ъ'
+    '`' -> 'ъ'
   )
 
   val biGrams = Map(
@@ -42,9 +46,7 @@ object Russian extends Language {
     "zh" -> 'ж',
     "yo" -> 'ё',
     "yu" -> 'ю',
-
-    "y|" -> 'ы',  // красивые, выучил
-    "s|" -> 'с'   // сходить
+    "kh" -> 'х'
   )
 
   val triGrams = Map[String, Char]()
@@ -53,31 +55,98 @@ object Russian extends Language {
     "shch" -> 'щ'
   )
 
+  val uniGramsInv = uniGrams.toList.map(_.swap).toMap
+  val uniGramsSpecialInv = uniGramsSpecial.toList.map(_.swap).toMap
+  val biGramsInv = biGrams.toList.map(_.swap).toMap
+  val triGramsInv = triGrams.toList.map(_.swap).toMap
+  val fourGramsInv = fourGrams.toList.map(_.swap).toMap
+
+  // y after m/n/r/t/v will be rendered as ы unless it is iotated
+  val yLetters = Set("my", "ny", "ry", "ty", "vy")
+
+  // If the y is iotated, render it as я, ё or ю
+  val iotatedLetters = Set("ya", "yo", "yu")
+
   override def latinToCyrillicIncremental(
     latin: String, cyrillic: String, append: Char
   ): (Int, String) = {
     val text = latin + append
     val ofs = text.length
-    if (ofs >= 4 &&
-        fourGrams.contains(text.substring(ofs - 4, ofs).toLowerCase)) {
-      val chars = text.substring(ofs - 4, ofs)
-      val cyrillic = fourGrams(chars.toLowerCase)
-      (-2, restoreCaseFirst(chars, cyrillic).toString)
-    } else if (ofs >= 3 &&
-      triGrams.contains(text.substring(ofs - 3, ofs).toLowerCase)) {
-      val chars = text.substring(ofs - 3, ofs)
-      val cyrillic = triGrams(chars.toLowerCase)
-      (-2, restoreCaseFirst(chars, cyrillic).toString)
-    } else if (ofs >= 2 &&
-               biGrams.contains(text.substring(ofs - 2, ofs).toLowerCase)) {
-      val chars = text.substring(ofs - 2, ofs)
-      val cyrillic = biGrams(chars.toLowerCase)
-      (-1, restoreCaseFirst(chars, cyrillic).toString)
-    } else if (uniGrams.contains(text(ofs - 1).toLower)) {
-      val cyrillic = uniGrams(text(ofs - 1).toLower)
-      (0, (if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic).toString)
-    } else {
-      (0, text(ofs - 1).toString)
+    val result =
+      if (ofs >= 4 &&
+          fourGrams.contains(text.substring(ofs - 4, ofs).toLowerCase)) {
+        val chars = text.substring(ofs - 4, ofs)
+        val cyrillic = fourGrams(chars.toLowerCase)
+        (-2, restoreCaseFirst(chars, cyrillic).toString)
+      } else if (ofs >= 3
+        && yLetters.contains(text.substring(ofs - 3, ofs - 1).toLowerCase)
+        && !iotatedLetters.contains(text.substring(ofs - 2, ofs).toLowerCase)
+      ) {
+        val cyrillic = uniGrams.getOrElse(text(ofs - 1).toLower, text(ofs - 1))
+        (0, (if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic).toString)
+      } else if (ofs >= 3 &&
+                 triGrams.contains(text.substring(ofs - 3, ofs).toLowerCase)) {
+        val chars = text.substring(ofs - 3, ofs)
+        val cyrillic = triGrams(chars.toLowerCase)
+        (-2, restoreCaseFirst(chars, cyrillic).toString)
+      } else if (ofs >= 2 &&
+                 biGrams.contains(text.substring(ofs - 2, ofs).toLowerCase)) {
+        val chars = text.substring(ofs - 2, ofs)
+        val cyrillic = biGrams(chars.toLowerCase)
+        (-1, restoreCaseFirst(chars, cyrillic).toString)
+      } else if (uniGrams.contains(text(ofs - 1).toLower)) {
+        val cyrillic = uniGrams(text(ofs - 1).toLower)
+        (0, (if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic).toString)
+      } else if (ofs >= 2 && uniGramsSpecial.contains(text(ofs - 1))) {
+        val result =
+          if (ofs >= 3 && text(ofs - 2).isUpper && text(ofs - 3).isUpper)
+            uniGramsSpecial(text(ofs - 1)).toUpper
+          else uniGramsSpecial(text(ofs - 1))
+        (0, result.toString)
+      } else {
+        (0, text(ofs - 1).toString)
+      }
+
+    if (ofs >= 3 && uniGramsSpecial.contains(text(ofs - 2))) {
+      val (l, r) = (text(ofs - 3), text(ofs - 1))
+      val letter = uniGramsSpecial(text(ofs - 2))
+      val replace = if (l.isUpper && r.isUpper) letter.toUpper else letter
+      val cyrillicOfs = cyrillic.length - 1
+
+      if (replace == cyrillic(cyrillicOfs)) result
+      else {
+        val updated = replace + cyrillic.substring(
+          cyrillicOfs + 1, cyrillic.length + result._1)
+        (-updated.length + result._1, updated + result._2)
+      }
+    } else result
+  }
+
+  private def toLatin(letter: Char): String = {
+    val isUpper = letter.isUpper
+    val letterLc = letter.toLower
+    fourGramsInv.get(letterLc).map(applyCase(_, isUpper))
+      .orElse(triGramsInv.get(letterLc).map(applyCase(_, isUpper)))
+      .orElse(biGramsInv.get(letterLc).map(applyCase(_, isUpper)))
+      .orElse(uniGramsInv.get(letterLc).map(x => applyCase(x.toString, isUpper)))
+      .orElse(uniGramsSpecialInv.get(letterLc).map(x => applyCase(x.toString, isUpper)))
+      .getOrElse(letter.toString)
+  }
+
+  override def cyrillicToLatinIncremental(
+    cyrillic: String, letter: Char
+  ): (Int, String) = {
+    val current = toLatin(letter)
+
+    val changeCase =
+      letter.isUpper &&
+      (cyrillic.length == 1 || cyrillic.lastOption.exists(_.isUpper))
+
+    if (!changeCase) (0, current)
+    else {
+      val mapped = toLatin(cyrillic.last)
+      val rest = mapped.tail
+      (-rest.length, rest.toUpperCase + current.toUpperCase)
     }
   }
 }