Implement Cyrillic to Latin conversion

tindzk · tindzk · commit 656b4989e59c · 2019-08-05T10:16:54.000+03:00
In the process of implementing the reverse direction, several limitations in the rules were addressed. The adapted rules were tested on a small Wikipedia text corpus and a decrease in the error rate was witnessed. Notable changes include the removal of the vertical bar (|) for precedence. Instead, the bi-gram "kh" was introduced. "ъ" is now mapped onto ` instead of " since double quotes are commonly used in Slavic texts. The rules now also handle different cases of capital letters. Closes #2.
diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 [![Build Status](http://ci.sparse.tech/api/badges/sparsetech/translit-scala/status.svg)](http://ci.sparse.tech/sparsetech/translit-scala)
 [![Maven Central](https://img.shields.io/maven-central/v/tech.sparse/translit-scala_2.12.svg)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22tech.sparse%22%20AND%20a%3A%22translit-scala_2.12%22)
 
-translit-scala is a transliteration library for Scala and Scala.js. It implements transliteration rules for Slavic languages. It supports converting texts from the Latin to the Cyrillic alphabet.
+translit-scala is a transliteration library for Scala and Scala.js. It implements transliteration rules for Slavic languages. It supports converting texts from the Latin to the Cyrillic alphabet and vice-versa.
 
 ## Compatibility
 | Back end   | Scala versions |
@@ -52,7 +52,7 @@ We decompose letters in their Latin transliteration more consistently than Natio
 * Volodymyr (Володимир)
 * blyz'ko (близько)
 
-The Latin letter *y* is also the phonetic basis of four letters in the Slavic alphabet: я, є, ї, ю. They get transliterated accordingly:
+The Latin letter *y* is also the phonetic basis of four letters (iotated vowels) in the Ukrainian alphabet: я, є, ї, ю. They get transliterated accordingly:
 
 * ya → я
 * ye → є
@@ -63,20 +63,23 @@ Unlike National 2010, we always use the same transliteration regardless of the p
 
 The accented counterpart of и is й and is represented by a separate letter, *j*.
 
-*Example:* Zhurs'kyj (Згурський)
+*Example:* Zgurs'kyj (Згурський)
 
 #### Soft Signs and Apostrophes
 The second change to National 2010 is that we try to restore soft signs and apostrophes:
 
 * Ukrayins'kyj (Український), malen'kyj (маленький)
 * m'yaso (м'ясо), matir'yu (матір'ю)
 
+In National 2010, *g* is mapped to *ґ* which is phonetically accurate, though the letter is fairly uncommon in Ukrainian. Therefore, *ґ* is represented by *g'*.
+
 This feature is experimental and can be disabled by setting `apostrophes` to `false`.
 
 #### Convenience mappings
 Another modification was to provide the following mappings:
 
 * c → ц
+* h → х
 * q → щ
 * w → ш
 * x → ж
@@ -91,9 +94,7 @@ Note that these mappings are phonetically inaccurate. However, using them still
 * Another advantage is the proximity on the English keyboard layout:
     * *q* and *w* are located next to each other; *ш* and *щ* characters are phonetically close
     * *z* and *x* are located next to each other; *з* and *ж* characters are phonetically close
-
-#### Precedence
-The replacement patterns are applied sequentially by traversing the input character-by-character. In some cases, a rule spanning multiple characters should not be applied. An example is the word: схильність. The transliteration of *сх* corresponds to two separate letters *s* and *h*, which would map to *ш*. To prevent this, one can place a vertical bar between the two characters. The full transliteration then looks as follows: *s|hyl'nist*
+* *h* is mapped to *х* since it is a common letter, *kh* is only needed in case *h* is ambiguous
 
 ## Russian
 The Russian rules are similar to the Ukrainian ones.
@@ -103,13 +104,7 @@ Some differences are:
 * *i* corresponds to *и*, whereas *y* to *ы*
 * Russian distinguishes between soft and hard signs. It does not have apostrophes. The following mappings are used:
   * Soft sign: *'* for ь
-  * Hard sign: *"* for ъ
-
-### Precedence
-As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules from being applied.
-
-* красивые: krasivy|e
-* сходить: s|hodit
+  * Hard sign: *`* for ъ
 
 ### Mapping
 | Latin | Cyrillic |
@@ -121,7 +116,7 @@ As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules
 | e     | е        |
 | f     | ф        |
 | g     | г        |
-| h     | х        |
+| h, kh | х        |
 | i     | и        |
 | j     | й        |
 | k     | к        |
@@ -141,7 +136,7 @@ As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules
 | y     | ы        |
 | z     | з        |
 | '     | ь        |
-| "     | ъ        |
+| \`    | ъ        |
 | ch    | ч        |
 | sh    | ш        |
 | ya    | я        |
@@ -151,15 +146,20 @@ As with the Ukrainian rules, a vertical bar can be placed to avoid certain rules
 | yu    | ю        |
 | shch  | щ        |
 
-#### Examples
-| Russian | Transliterated |
-|---------|----------------|
-| Привет  | Privet         |
-| Съел    | S"el           |
-| Щётка   | Shchyotka      |
-| Льдина  | L'dina         |
+### Examples
+| Russian  | Transliterated |
+|----------|----------------|
+| Привет   | Privet         |
+| Съел     | S\`el          |
+| Щётка    | Shchyotka      |
+| Льдина   | L'dina         |
+| красивые | krasivye       |
+| сходить  | skhodit'       |
+
+## Internals
+The replacement patterns are applied sequentially by traversing the input character-by-character. The functions `latinToCyrillicIncremental` and `cyrillicToLatinIncremental` take the left context which is needed for some rules. The result indicates the number of characters to remove and a replacement string.
 
-### Credits
+## Credits
 The rules and examples were adapted from the following libraries:
 
 * [translit-english-ukrainian](https://github.com/MarkovSergii/translit-english-ukrainian)
diff --git a/shared/src/main/scala/translit/Helpers.scala b/shared/src/main/scala/translit/Helpers.scala
@@ -1,6 +1,9 @@
 package translit
 
 object Helpers {
+  def applyCase(str: String, isUpper: Boolean): String =
+    if (isUpper) str(0).toUpper + str.tail else str
+
   def restoreCaseAll(str: String, cyrillic: Char): Char =
     if (str.forall(_.isUpper)) cyrillic.toUpper else cyrillic
 
diff --git a/shared/src/main/scala/translit/Language.scala b/shared/src/main/scala/translit/Language.scala
@@ -15,6 +15,8 @@ trait Language {
     latin: String, cyrillic: String, append: Char
   ): (Int, String)
 
+  def cyrillicToLatinIncremental(cyrillic: String, letter: Char): (Int, String)
+
   def latinToCyrillic(text: String): String = {
     val result = new StringBuilder(text.length)
     var offset = 0
@@ -29,4 +31,19 @@ trait Language {
 
     result.mkString
   }
+
+  def cyrillicToLatin(text: String): String = {
+    val result = new StringBuilder(text.length * 2)
+    var offset = 0
+
+    while (offset < text.length) {
+      val (length, c) = cyrillicToLatinIncremental(
+        text.take(offset), text(offset))
+      if (length < 0) result.setLength(result.length + length)
+      result.append(c)
+      offset += 1
+    }
+
+    result.mkString
+  }
 }
diff --git a/shared/src/main/scala/translit/Noop.scala b/shared/src/main/scala/translit/Noop.scala
@@ -4,5 +4,9 @@ object Noop extends translit.Language {
   override def latinToCyrillicIncremental(
     latin: String, cyrillic: String, append: Char
   ): (Int, String) = (0, append.toString)
+
+  override def cyrillicToLatinIncremental(
+   cyrillic: String, letter: Char
+  ): (Int, String) = (0, letter.toString)
 }
 
diff --git a/shared/src/main/scala/translit/Russian.scala b/shared/src/main/scala/translit/Russian.scala
@@ -29,9 +29,13 @@ object Russian extends Language {
     'w' -> 'ш',
     'x' -> 'ж',
     'y' -> 'ы',
-    'z' -> 'з',
+    'z' -> 'з'
+  )
+
+  // Infer case from previous character
+  val uniGramsSpecial = Map(
     '\'' -> 'ь',
-    '"' -> 'ъ'
+    '`' -> 'ъ'
   )
 
   val biGrams = Map(
@@ -42,9 +46,7 @@ object Russian extends Language {
     "zh" -> 'ж',
     "yo" -> 'ё',
     "yu" -> 'ю',
-
-    "y|" -> 'ы',  // красивые, выучил
-    "s|" -> 'с'   // сходить
+    "kh" -> 'х'
   )
 
   val triGrams = Map[String, Char]()
@@ -53,31 +55,98 @@ object Russian extends Language {
     "shch" -> 'щ'
   )
 
+  val uniGramsInv = uniGrams.toList.map(_.swap).toMap
+  val uniGramsSpecialInv = uniGramsSpecial.toList.map(_.swap).toMap
+  val biGramsInv = biGrams.toList.map(_.swap).toMap
+  val triGramsInv = triGrams.toList.map(_.swap).toMap
+  val fourGramsInv = fourGrams.toList.map(_.swap).toMap
+
+  // y after m/n/r/t/v will be rendered as ы unless it is iotated
+  val yLetters = Set("my", "ny", "ry", "ty", "vy")
+
+  // If the y is iotated, render it as я, ё or ю
+  val iotatedLetters = Set("ya", "yo", "yu")
+
   override def latinToCyrillicIncremental(
     latin: String, cyrillic: String, append: Char
   ): (Int, String) = {
     val text = latin + append
     val ofs = text.length
-    if (ofs >= 4 &&
-        fourGrams.contains(text.substring(ofs - 4, ofs).toLowerCase)) {
-      val chars = text.substring(ofs - 4, ofs)
-      val cyrillic = fourGrams(chars.toLowerCase)
-      (-2, restoreCaseFirst(chars, cyrillic).toString)
-    } else if (ofs >= 3 &&
-      triGrams.contains(text.substring(ofs - 3, ofs).toLowerCase)) {
-      val chars = text.substring(ofs - 3, ofs)
-      val cyrillic = triGrams(chars.toLowerCase)
-      (-2, restoreCaseFirst(chars, cyrillic).toString)
-    } else if (ofs >= 2 &&
-               biGrams.contains(text.substring(ofs - 2, ofs).toLowerCase)) {
-      val chars = text.substring(ofs - 2, ofs)
-      val cyrillic = biGrams(chars.toLowerCase)
-      (-1, restoreCaseFirst(chars, cyrillic).toString)
-    } else if (uniGrams.contains(text(ofs - 1).toLower)) {
-      val cyrillic = uniGrams(text(ofs - 1).toLower)
-      (0, (if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic).toString)
-    } else {
-      (0, text(ofs - 1).toString)
+    val result =
+      if (ofs >= 4 &&
+          fourGrams.contains(text.substring(ofs - 4, ofs).toLowerCase)) {
+        val chars = text.substring(ofs - 4, ofs)
+        val cyrillic = fourGrams(chars.toLowerCase)
+        (-2, restoreCaseFirst(chars, cyrillic).toString)
+      } else if (ofs >= 3
+        && yLetters.contains(text.substring(ofs - 3, ofs - 1).toLowerCase)
+        && !iotatedLetters.contains(text.substring(ofs - 2, ofs).toLowerCase)
+      ) {
+        val cyrillic = uniGrams.getOrElse(text(ofs - 1).toLower, text(ofs - 1))
+        (0, (if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic).toString)
+      } else if (ofs >= 3 &&
+                 triGrams.contains(text.substring(ofs - 3, ofs).toLowerCase)) {
+        val chars = text.substring(ofs - 3, ofs)
+        val cyrillic = triGrams(chars.toLowerCase)
+        (-2, restoreCaseFirst(chars, cyrillic).toString)
+      } else if (ofs >= 2 &&
+                 biGrams.contains(text.substring(ofs - 2, ofs).toLowerCase)) {
+        val chars = text.substring(ofs - 2, ofs)
+        val cyrillic = biGrams(chars.toLowerCase)
+        (-1, restoreCaseFirst(chars, cyrillic).toString)
+      } else if (uniGrams.contains(text(ofs - 1).toLower)) {
+        val cyrillic = uniGrams(text(ofs - 1).toLower)
+        (0, (if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic).toString)
+      } else if (ofs >= 2 && uniGramsSpecial.contains(text(ofs - 1))) {
+        val result =
+          if (ofs >= 3 && text(ofs - 2).isUpper && text(ofs - 3).isUpper)
+            uniGramsSpecial(text(ofs - 1)).toUpper
+          else uniGramsSpecial(text(ofs - 1))
+        (0, result.toString)
+      } else {
+        (0, text(ofs - 1).toString)
+      }
+
+    if (ofs >= 3 && uniGramsSpecial.contains(text(ofs - 2))) {
+      val (l, r) = (text(ofs - 3), text(ofs - 1))
+      val letter = uniGramsSpecial(text(ofs - 2))
+      val replace = if (l.isUpper && r.isUpper) letter.toUpper else letter
+      val cyrillicOfs = cyrillic.length - 1
+
+      if (replace == cyrillic(cyrillicOfs)) result
+      else {
+        val updated = replace + cyrillic.substring(
+          cyrillicOfs + 1, cyrillic.length + result._1)
+        (-updated.length + result._1, updated + result._2)
+      }
+    } else result
+  }
+
+  private def toLatin(letter: Char): String = {
+    val isUpper = letter.isUpper
+    val letterLc = letter.toLower
+    fourGramsInv.get(letterLc).map(applyCase(_, isUpper))
+      .orElse(triGramsInv.get(letterLc).map(applyCase(_, isUpper)))
+      .orElse(biGramsInv.get(letterLc).map(applyCase(_, isUpper)))
+      .orElse(uniGramsInv.get(letterLc).map(x => applyCase(x.toString, isUpper)))
+      .orElse(uniGramsSpecialInv.get(letterLc).map(x => applyCase(x.toString, isUpper)))
+      .getOrElse(letter.toString)
+  }
+
+  override def cyrillicToLatinIncremental(
+    cyrillic: String, letter: Char
+  ): (Int, String) = {
+    val current = toLatin(letter)
+
+    val changeCase =
+      letter.isUpper &&
+      (cyrillic.length == 1 || cyrillic.lastOption.exists(_.isUpper))
+
+    if (!changeCase) (0, current)
+    else {
+      val mapped = toLatin(cyrillic.last)
+      val rest = mapped.tail
+      (-rest.length, rest.toUpperCase + current.toUpperCase)
     }
   }
 }
diff --git a/shared/src/main/scala/translit/Ukrainian.scala b/shared/src/main/scala/translit/Ukrainian.scala
@@ -14,7 +14,6 @@ class Ukrainian(apostrophes: Boolean) extends Language {
     'e' -> 'е',
     'f' -> 'ф',
     'g' -> 'г',
-    'h' -> 'х',
     'i' -> 'і',
     'j' -> 'й',
     'k' -> 'к',
@@ -34,6 +33,7 @@ class Ukrainian(apostrophes: Boolean) extends Language {
     // Mappings for more convenient typing. Allows us to cover every letter of
     // the Latin alphabet
     'c' -> 'ц',
+    'h' -> 'х',
     'q' -> 'щ',
     'w' -> 'ш',
     'x' -> 'ж'
@@ -52,8 +52,7 @@ class Ukrainian(apostrophes: Boolean) extends Language {
     "ts" -> 'ц',
     "zh" -> 'ж',
 
-    // With the vertical bar, transliteration can be disabled.
-    "s|" -> 'с'
+    "kh" -> 'х'
   )
 
   val triGrams = Map[String, Char]()
@@ -62,6 +61,15 @@ class Ukrainian(apostrophes: Boolean) extends Language {
     "shch" -> 'щ'
   )
 
+  val uniGramsInv = uniGrams.toList.map(_.swap).toMap
+  val uniGramsSpecialInv = Map(
+    'ь' -> '\'',
+    '\'' -> '\''
+  )
+  val biGramsInv = biGrams.toList.map(_.swap).toMap
+  val triGramsInv = triGrams.toList.map(_.swap).toMap
+  val fourGramsInv = fourGrams.toList.map(_.swap).toMap
+
   val apostrophePatterns = Set(
     ('b', "ya"),
     ('b', "ye"),
@@ -157,6 +165,37 @@ class Ukrainian(apostrophes: Boolean) extends Language {
       }
     } else result
   }
+
+  private def toLatin(letter: Char): String = {
+    val isUpper = letter.isUpper
+    val letterLc = letter.toLower
+    fourGramsInv.get(letterLc).map(applyCase(_, isUpper))
+      .orElse(triGramsInv.get(letterLc).map(applyCase(_, isUpper)))
+      .orElse(biGramsInv.get(letterLc).map(applyCase(_, isUpper)))
+      .orElse(uniGramsInv.get(letterLc).map(x => applyCase(x.toString, isUpper)))
+      .orElse(uniGramsSpecialInv.get(letterLc).map(x => applyCase(x.toString, isUpper)))
+      .getOrElse(letter.toString)
+  }
+
+  override def cyrillicToLatinIncremental(
+    cyrillic: String, letter: Char
+  ): (Int, String) = {
+    val current = toLatin(letter)
+
+    val changeCase =
+      letter.isUpper && {
+        val withoutApostrophes = cyrillic.filter(_ != '\'')
+        withoutApostrophes.length == 1 ||
+        withoutApostrophes.lastOption.exists(_.isUpper)
+      }
+
+    if (!changeCase) (0, current)
+    else {
+      val mapped = toLatin(cyrillic.last)
+      val rest = mapped.tail
+      (-rest.length, rest.toUpperCase + current.toUpperCase)
+    }
+  }
 }
 
 object Ukrainian extends Ukrainian(apostrophes = true)
diff --git a/shared/src/test/scala/translit/RussianSpec.scala b/shared/src/test/scala/translit/RussianSpec.scala
diff --git a/shared/src/test/scala/translit/UkrainianSpec.scala b/shared/src/test/scala/translit/UkrainianSpec.scala

Original file line number	Diff line number	Diff line change
`@@ -4,5 +4,9 @@ object Noop extends translit.Language {`
`4`	`4`	`override def latinToCyrillicIncremental(`
`5`	`5`	`latin: String, cyrillic: String, append: Char`
`6`	`6`	`): (Int, String) = (0, append.toString)`
	`7`	`+`
	`8`	`+ override def cyrillicToLatinIncremental(`
	`9`	`+ cyrillic: String, letter: Char`
	`10`	`+ ): (Int, String) = (0, letter.toString)`
`7`	`11`	`}`
`8`	`12`