implement CESU-8

[ahorek]

https://bugs.ruby-lang.org/issues/15931

jruby/jruby#6464

[headius]

Something in here is not right and causes failures in JRuby. I attempted to use a master snapshot in a test CI on JRuby and ended up with the following errors:

1) Failure:
TestEncodingConverter#test_xml_hasharg [/home/runner/work/jruby/jruby/test/mri/ruby/test_econv.rb:843]:
<"\x22\x[26](https://github.com/jruby/jruby/runs/5572750596?check_suite_focus=true#step:7:26)\x61\x6D\x70\x3B\e\x24\x42\x24\x26\e\x28\x42\x26\x23\x78\x32\x36\x36\x31\x3B\x26\x61\x6D\x70\x3B\x26\x71\x75\x6F\x74\x3B\x[27](https://github.com/jruby/jruby/runs/5572750596?check_suite_focus=true#step:7:27)\x22"> expected but was
<"\x22\x26\x61\x6D\x70\x3B\e\x24\x42\x24\x26\e\x[28](https://github.com/jruby/jruby/runs/5572750596?check_suite_focus=true#step:7:28)\x42\x26\x23\x78\x32\x36\x36\x[31](https://github.com/jruby/jruby/runs/5572750596?check_suite_focus=true#step:7:31)\x3B\x26\x61\x6D\x70\x3B\x26\x71\x75\x6F\x74\x3B\x26\x61\x70\x6F\x73\x3B\x22">.

  2) Failure:
TestIO_M17N#test_w_xml_attr [/home/runner/work/jruby/jruby/test/mri/ruby/test_io_m17n.rb:2049]:
<"\"&amp;&lt;&gt;&quot;'\xE4\xB8\x82\xE3\x81\x82\n" + "\""> expected but was
<"\"&amp;&lt;&gt;&quot;&apos;\xE4\xB8\x82\xE3\x81\x82\n" + "\"">.

[headius]

I will revert for now and we can try to figure out what broke. It's possible the tests are bad or not up-to-date for CESU-8.

[headius]

Ah automatic revert is not possible due to other changes that have landed (like my toUnsigned changes). We should not release until we can address the failures.

[ahorek]

this test is definitely outdated https://bugs.ruby-lang.org/issues/16922

  2) Failure:
TestIO_M17N#test_w_xml_attr [/home/runner/work/jruby/jruby/test/mri/ruby/test_io_m17n.rb:2049]:
<"\"&amp;&lt;&gt;&quot;'\xE4\xB8\x82\xE3\x81\x82\n" + "\""> expected but was
<"\"&amp;&lt;&gt;&quot;&apos;\xE4\xB8\x82\xE3\x81\x82\n" + "\"">.

not sure about the second one, I'll take a look

  1) Failure:
TestEncodingConverter#test_xml_hasharg [/home/runner/work/jruby/jruby/test/mri/ruby/test_econv.rb:843]:
<"\x22\x26\x61\x6D\x70\x3B\e\x24\x42\x24\x26\e\x28\x42\x26\x23\x78\x32\x36\x36\x31\x3B\x26\x61\x6D\x70\x3B\x26\x71\x75\x6F\x74\x3B\x27\x22"> expected but was
<"\x22\x26\x61\x6D\x70\x3B\e\x24\x42\x24\x26\e\x28\x42\x26\x23\x78\x32\x36\x36\x31\x3B\x26\x61\x6D\x70\x3B\x26\x71\x75\x6F\x74\x3B\x26\x61\x70\x6F\x73\x3B\x22">.

[headius]

Ok as I suspected. I have started a PR to update the test on master and requested that the change and test patches be backported to CRuby 2.6 (though they probably won't do it).

jruby/jruby#7142

https://bugs.ruby-lang.org/issues/16922