Merge pull request #13 from sparsetech/issue/3

Evaluate performance on 1M+ Wikipedia word corpora

Author: tindzk

.gitignore

@@ -1,5 +1,3 @@
-.bloop/
-
 *.class
 *.log
 
@@ -17,4 +15,7 @@ project/plugins/project/
 # Scala-IDE specific
 .scala_dependencies
 .worksheet
-.idea
+
+.bloop/
+/.idea/
+/.metals/

README.md

@@ -4,6 +4,19 @@
 
 translit-scala is a transliteration library for Scala and Scala.js. It implements transliteration rules for Slavic languages. It supports converting texts from the Latin to the Cyrillic alphabet and vice-versa.
 
+## Features
+* Supported languages
+	* Russian
+	* Ukrainian
+* Incremental transliteration
+* Transliterations reversible with an accuracy of 99.99%
+* Transliterations optimised for typing and reading
+	* Only letters from the US keyboard are used
+	* All common letters can be typed with a single keystroke
+	* Convenience shortcuts are provided
+* Cross-platform support (JVM, Scala.js)
+* Zero dependencies
+
 ## Compatibility
 | Back end   | Scala versions |
 |:-----------|:---------------|
@@ -19,14 +32,7 @@ libraryDependencies += "tech.sparse" %%% "translit-scala" % "0.1.1"  // JavaScri
 ## Examples
 ```scala
 translit.Ukrainian.latinToCyrillic("Kyyiv")  // Київ
-translit.Russian.latinToCyrillic("pal'ma")   // пальма
-```
-
-The transliteration to Cyrillic also restores soft signs (ь) and apostrophes ('):
-
-```scala
-translit.Ukrainian.latinToCyrillic("Mar'yana")  // Мар'яна
-translit.Ukrainian.latinToCyrillic("p'yat'")    // п'ять
+translit.Russian.latinToCyrillic("pal`ma")   // пальма
 ```
 
 ## Ukrainian
@@ -35,7 +41,7 @@ There have been several attempts to standardise transliteration rules. For examp
 * Ukrayins'kyy pravopys (BGN/PCGN 1965)
 * Ukrains'kyi pravopys (National 1996)
 * Ukrainskyi pravopys ([National 2010](http://zakon1.rada.gov.ua/laws/show/55-2010-%D0%BF))
-* Ukrayins'kyj pravopys (*translit-scala*)
+* Ukrayins\`kyj pravopys (*translit-scala*)
 
 Furthermore, there are language-specific transliterations, e.g. in German and French, that use the spelling conventions of the respective language (*sch* in German instead of *sh* in English).
 
@@ -50,7 +56,7 @@ Our transliteration was initially based on National 2010, but modified in the pr
 We decompose letters in their Latin transliteration more consistently than National 2010. The letter и always gets transcribed as *y*:
 
 * Volodymyr (Володимир)
-* blyz'ko (близько)
+* blyz\`ko (близько)
 
 The Latin letter *y* forms the phonetic basis of four letters (iotated vowels) in the Ukrainian alphabet: я, є, ї, ю. They get transliterated accordingly:
 
@@ -63,18 +69,16 @@ Unlike National 2010, we always use the same transliteration regardless of the p
 
 The accented counterpart of и is й and is represented by a separate letter, *j*.
 
-*Example:* Zgurs'kyj (Згурський)
+*Example:* Zgurs\`kyj (Згурський)
 
 #### Soft Signs and Apostrophes
-The second change to National 2010 is that we try to restore soft signs and apostrophes:
+The second change to National 2010 is that we retain soft signs (ь) and apostrophes ('):
 
-* Ukrayins'kyj (Український), malen'kyj (маленький)
+* Ukrayins\`kyj (Український), malen\`kyj (маленький)
 * m'yaso (м'ясо), matir'yu (матір'ю)
 
 In National 2010, *g* gets mapped to *ґ* which is phonetically accurate, though the letter *ґ* is fairly uncommon in Ukrainian. Therefore, we represent *ґ* by the bi-gram *g'*.
 
-This feature is experimental and can be disabled by setting `apostrophes` to `false`.
-
 #### Convenience mappings
 Another modification was to provide the following mappings:
 
@@ -95,17 +99,24 @@ Note that these mappings are phonetically inaccurate. However, using them still
     * *q* and *w* are located next to each other; *ш* and *щ* characters are phonetically close
     * *z* and *x* are located next to each other; *з* and *ж* characters are phonetically close
 * *h* is mapped to *х* since it is a common letter, *kh* is only needed in case *h* is ambiguous
-	* An example is the word: схильність. The transliteration of *сх* corresponds to two separate letters *s* and *h*, which would map to *ш*. To prevent this, one can use the bi-gram *kh* instead to represent *х*. The full transliteration then looks as follows: *skhyl'nist*
+	* An example is the word: схильність. The transliteration of *сх* corresponds to two separate letters *s* and *h*, which would map to *ш*. To prevent this, one can use the bi-gram *kh* instead to represent *х*. The full transliteration then looks as follows: *skhyl\`nist*
+
+#### Escaping
+You can insert a backslash (\\) if a replacement rule should not be applied, for example:
+
+* stranstvy\e -> странствие
+* pidtry\annya -> підтриання
+* Bash\charshiyi -> Башчаршії
 
 ## Russian
 The Russian rules are similar to the Ukrainian ones.
 
 Some differences are:
 
 * *i* corresponds to *и*, whereas *y* to *ы*
-* Russian distinguishes between soft and hard signs. It does not have apostrophes. The following mappings are used:
-  * Soft sign: *'* for ь
-  * Hard sign: *`* for ъ
+* Russian distinguishes between soft and hard signs. Apostrophes only appear in foreign names. The following mappings are used:
+  * Soft sign: *`* for ь
+  * Hard sign: *~* for ъ
 
 ### Mapping
 | Latin | Cyrillic |
@@ -136,8 +147,8 @@ Some differences are:
 | x     | ж        |
 | y     | ы        |
 | z     | з        |
-| '     | ь        |
-| \`    | ъ        |
+| \`    | ь        |
+| ~     | ъ        |
 | ch    | ч        |
 | sh    | ш        |
 | ya    | я        |
@@ -151,11 +162,11 @@ Some differences are:
 | Russian  | Transliterated |
 |----------|----------------|
 | Привет   | Privet         |
-| Съел     | S\`el          |
+| Съел     | S~el           |
 | Щётка    | Shchyotka      |
-| Льдина   | L'dina         |
-| красивые | krasivye       |
-| сходить  | skhodit'       |
+| Льдина   | L\`dina        |
+| красивые | krasivy\e      |
+| сходить  | skhodit\`      |
 
 ## Internals
 The replacement patterns are applied sequentially by traversing the input character-by-character. The functions `latinToCyrillicIncremental` and `cyrillicToLatinIncremental` take the left context which is needed by some rules, for example to determine the correct case of soft/hard signs. The result of the functions indicates the number of characters to remove on the right as well as their string replacement.
@@ -170,8 +181,16 @@ def cyrillicToLatinIncremental(
 ): (Int, String)
 ```
 
+## Performance
+The test suite evaluates whether transliterations are reversible. The accuracy is calculated on words extracted from Wikipedia article dumps for all supported languages. Words are transliterated to Latin and then back to Cyrillic. A word counts as correct if the result of the reversed transliteration matches the original.
+
+| Language  | Total     | Correct   | Accuracy |
+|-----------|-----------|-----------|----------|
+| Ukrainian | 1,811,772 | 1,811,661 | 99.99%   |
+| Russian   | 1,529,184 | 1,529,043 | 99.99%   |
+
 ## Credits
-The rules and examples were adapted from the following libraries:
+The rules and examples were adapted from the following libraries and websites:
 
 * [translit-english-ukrainian](https://github.com/MarkovSergii/translit-english-ukrainian)
 * [translit-ua](https://github.com/dchaplinsky/translit-ua)

build.sbt

@@ -50,6 +50,12 @@ lazy val translit = crossProject.in(file("."))
     /* Use io.js for faster compilation of test cases */
     scalaJSStage in Global := FastOptStage
   )
+  .jvmSettings(
+    libraryDependencies ++= Seq(
+      "com.github.pathikrit" %%% "better-files" % "3.8.0" % "test",
+      "org.scalaj" %%% "scalaj-http" % "2.4.2" % "test"
+    )
+  )
 
 lazy val js  = translit.js
 lazy val jvm = translit.jvm

build.toml

@@ -19,9 +19,19 @@ root    = "shared"
 sources = ["shared/src/main/scala"]
 targets = ["js", "jvm"]
 
+[module.translit.jvm]
+root = "jvm"
+
 [module.translit.test]
 sources   = ["shared/src/test/scala"]
 targets   = ["js", "jvm"]
 scalaDeps = [
   ["org.scalatest", "scalatest", "3.0.8"]
 ]
+
+[module.translit.test.jvm]
+sources   = ["jvm/src/test/scala"]
+scalaDeps = [
+  ["com.github.pathikrit", "better-files", "3.8.0"],
+  ["org.scalaj", "scalaj-http", "2.4.2"]
+]

jvm/src/test/scala/translit/CorpusSpec.scala

@@ -0,0 +1,67 @@
+package translit
+
+import java.nio.charset.Charset
+
+import better.files._
+import org.scalatest.FunSuite
+import scalaj.http.Http
+
+class CorpusSpec extends FunSuite {
+  def getDump(name: String): File = {
+    val file = File(s"/tmp/$name")
+    if (file.exists) file
+    else {
+      println(s"Downloading $name...")
+      val http = Http(s"http://builds.sparse.tech/wiki/$name").asBytes
+      assert(http.isSuccess)
+      file.writeByteArray(http.body)
+    }
+  }
+
+  def evaluate(dump: String, language: Language): (Long, Long) = {
+    val sentences = getDump(dump)
+
+    var correct = 0L
+    var total   = 0L
+
+    sentences.gzipInputStream().foreach { gzip =>
+      val words = gzip.lines(Charset.forName("UTF-8"))
+      val latin = (('a' to 'z') ++ ('A' to 'Z')).toSet
+
+      words.filter(!_.exists(latin.contains)).foreach { original =>
+        val latin    = language.cyrillicToLatin(original)
+        val cyrillic = language.latinToCyrillic(latin)
+        if (original == cyrillic) {
+          correct += 1
+        } else {
+          println(s"Mismatch: $original vs $cyrillic")
+        }
+
+        total += 1
+      }
+    }
+
+    (correct, total)
+  }
+
+  def check(perf: (Long, Long), expectedMinimum: Double): Unit = {
+    val (correct, total) = perf
+    val accuracy = correct.toDouble / total
+
+    println(s"Correct : $correct")
+    println(s"Total   : $total")
+    println(s"Accuracy: $accuracy")
+
+    assert(accuracy > expectedMinimum)
+  }
+
+  test("Russian") {
+    val perf = evaluate("tokens-ru-1710634.txt.gz", Russian)
+    check(perf, 0.999907)
+  }
+
+  test("Ukrainian") {
+    val perf = evaluate("tokens-uk-2000041.txt.gz", Ukrainian)
+    check(perf, 0.999938)
+  }
+}

shared/src/main/scala/translit/Russian.scala

@@ -34,16 +34,16 @@ object Russian extends Language {
 
   // Infer case from previous character
   val uniGramsSpecial = Map(
-    '\'' -> 'ь',
-    '`' -> 'ъ'
+    '`' -> 'ь',
+    '~' -> 'ъ'
   )
 
   val biGrams = Map(
     "ch" -> 'ч',
     "sh" -> 'ш',
+    "zh" -> 'ж',
     "ya" -> 'я',
     "ye" -> 'э',
-    "zh" -> 'ж',
     "yo" -> 'ё',
     "yu" -> 'ю',
     "kh" -> 'х'
@@ -55,6 +55,15 @@ object Russian extends Language {
     "shch" -> 'щ'
   )
 
+  val escape = Map(
+    "ya" -> "ыа",
+    "ye" -> "ые",
+    "yo" -> "ыо",
+    "yu" -> "ыу",
+    "shch" -> "шч"
+  )
+  val escapeCharacter = '\\'
+
   val uniGramsInv = uniGrams.toList.map(_.swap).toMap
   val uniGramsSpecialInv = uniGramsSpecial.toList.map(_.swap).toMap
   val biGramsInv = biGrams.toList.map(_.swap).toMap
@@ -64,20 +73,32 @@ object Russian extends Language {
   // y after m/n/r/t/v will be rendered as ы unless it is iotated
   val yLetters = Set("my", "ny", "ry", "ty", "vy")
 
-  // If the y is iotated, render it as я, ё or ю
-  val iotatedLetters = Set("ya", "yo", "yu")
+  // If the y is iotated, render it as я, э, ё or ю
+  val iotatedLetters = Set("ya", "ye", "yo", "yu")
 
   override def latinToCyrillicIncremental(
     latin: String, cyrillic: String, append: Char
   ): (Int, String) = {
     val text = latin + append
     val ofs = text.length
     val result =
-      if (ofs >= 4 &&
+      if (ofs >= 5 && text.takeRight(5).toLowerCase == "sh\\ch") {
+        val cyrillic = 'ч'
+        val result   = if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic
+        (-2, result.toString)
+      } else if (ofs >= 4 &&
           fourGrams.contains(text.substring(ofs - 4, ofs).toLowerCase)) {
         val chars = text.substring(ofs - 4, ofs)
         val cyrillic = fourGrams(chars.toLowerCase)
         (-2, restoreCaseFirst(chars, cyrillic).toString)
+      } else if (ofs >= 3 &&
+        escape.keySet.contains(
+          (text(ofs - 3).toString + text(ofs - 1)).toLowerCase
+        ) && text(ofs - 2) == escapeCharacter
+      ) {
+        val cyrillic = uniGrams.getOrElse(text(ofs - 1).toLower, text(ofs - 1))
+        val result = if (text(ofs - 1).isUpper) cyrillic.toUpper else cyrillic
+        (-1, result.toString)
       } else if (ofs >= 3
         && yLetters.contains(text.substring(ofs - 3, ofs - 1).toLowerCase)
         && !iotatedLetters.contains(text.substring(ofs - 2, ofs).toLowerCase)
@@ -109,8 +130,10 @@ object Russian extends Language {
 
     if (ofs >= 3 && uniGramsSpecial.contains(text(ofs - 2))) {
       val (l, r) = (text(ofs - 3), text(ofs - 1))
-      val letter = uniGramsSpecial(text(ofs - 2))
-      val replace = if (l.isUpper && r.isUpper) letter.toUpper else letter
+
+      val letter      = uniGramsSpecial(text(ofs - 2))
+      val replace     =
+        if (l.isUpper && r.isUpper) letter.toUpper else letter
       val cyrillicOfs = cyrillic.length - 1
 
       if (replace == cyrillic(cyrillicOfs)) result
@@ -136,17 +159,23 @@ object Russian extends Language {
   override def cyrillicToLatinIncremental(
     cyrillic: String, letter: Char
   ): (Int, String) = {
-    val current = toLatin(letter)
-
-    val changeCase =
-      letter.isUpper &&
-      (cyrillic.length == 1 || cyrillic.lastOption.exists(_.isUpper))
+    val current  = toLatin(letter)
+    val toEscape = cyrillic.lastOption
+      .map(_.toLower.toString + letter.toLower)
+      .exists(escape.values.toList.contains)
 
-    if (!changeCase) (0, current)
+    if (toEscape) (0, escapeCharacter + current)
     else {
-      val mapped = toLatin(cyrillic.last)
-      val rest = mapped.tail
-      (-rest.length, rest.toUpperCase + current.toUpperCase)
+      val changeCase =
+        letter.isUpper &&
+        (cyrillic.length == 1 || cyrillic.lastOption.exists(_.isUpper))
+
+      if (!changeCase) (0, current)
+      else {
+        val mapped = toLatin(cyrillic.last)
+        val rest   = mapped.tail
+        (-rest.length, rest.toUpperCase + current.toUpperCase)
+      }
     }
   }
 }