UTF-8 strings are broken, when not save sketch before compiling

[BlackBrix]

see this thread in forum:
--> http://forum.arduino.cc/index.php?topic=360932.msg2498388#msg2498388

---

I use 1.6.6 in a portable version on an USB-Stick (Win7 / Win 10).

I disabled the option "save when verifying or uploading"

If you are working on a file (ino, h, cpp) with UTF-8 chars in string-declarations within this file
(see "extended example" from here
--> http://forum.arduino.cc/index.php?topic=360932.msg2488750#msg2488750 ),
and you change something (it's enough to add a space char in a string),
and you DO NOT save this file before compiling and uploading,
the serial output of this UTF-8 chars (via UTF-8 capable Terminal) will be with errors.

If you save this file before compiling and uploading,
(either manually Ctrl+S or by enabling the option "save when verifying or uploading")
the serial output of this UTF-8 chars (via UTF-8 capable Terminal) will work correctly.

...

so it seems that when the IDE is compiling "from cache" (or from temp. build -folder)
there's no proper conversion or handling of UTF-8 chars on Windows OS.
The process of saving the sketch before (to a file on HDD / USB-Stick) seems to lead to a proper conversion or handling of any UTF-8 chars then ...
....

other users can reproduce this error as well
--> http://forum.arduino.cc/index.php?topic=360932.msg2499702#msg2499702

[Koepel]

It seems to happen only in Windows (so far I was not able to get this in linux), and only with UTF-8 characters, and only when a file is not saved. The temperary file in the build.... folder is not UTF-8.

[lmihalkovic]

see #4259 for more encoding issues

[BlackBrix]

"Federico Fissore" left arduino and now all "his" issues here are in the state "no one assigned" ?

[lmihalkovic]

Yes it seems that's how github works. I'm sure the company is organizing itself for the after. Arduino Create also seems to be nearing general availability.

[BlackBrix]

does someone work on this issue ?

[lmihalkovic]

I think the vast majority of entries are not looked into by anyone.

[BlackBrix]

any news on this ?

[per1234]

Steps to reproduce:

• File > New
• Paste the following code into the new sketch (leaving it in an unsaved state):

void setup() {
  Serial.begin(9600);
  while (!Serial) {}
  Serial.println( "¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ­ ® ¯ ° ± ² ³ ´ µ ¶ · ¸ ¹ º » ¼ ½ ¾ ¿");
}

void loop() {}

• Sketch > Upload

• Tools > Serial Monitor

• Serial Monitor displays (incorrectly):
  

• File > Save (now the sketch is in the saved state)

• Sketch > Upload

• Tools > Serial Monitor

• Serial Monitor displays (correctly):
  

Issue still occurs with Arduino IDE 1.8.3, Windows 7 64 bit

[PaulStoffregen]

I tested just now. I was not able to reproduce this problem on Macintosh or Linux 64 bit.

It does happen with Windows 10.

[PaulStoffregen]

This issue might be related to line 584 in arduino-core/src/processing/app/legacy/PApplet.java

      OutputStreamWriter osw = new OutputStreamWriter(output, "UTF-8");

Apparently OutputStreamWriter expects "UTF8", not "UTF-8" with the extra minus sign.

https://docs.oracle.com/javase/tutorial/i18n/text/stream.html

There's also another "UTF-8" usage on line 265 which might be worth fixing.

[PaulStoffregen]

Nope, just tried editing line 584, but it doesn't fix this issue. :(