Arduino 1.5.7 - Always does a Full Re-Compile

[mpflaga]

It appears that feature from: ARDUINO 1.0.1 - 2012.05.21 is broke.

• Compilation has been speeded up by only compiling changed files. (All files are recompiled when a new board is selected.) (Paul Stoffregen) http://code.google.com/p/arduino/issues/detail?id=638

Is no longer working, as Verify appears (at least on windows 7) to always do a full (aka clean) compile rather than a re-compile of only changed files.

Not that this was/is working in 1.5.6r2, and still failing in nightly build "ARDUINO 1.5.8 BETA - not yet released" dated Aug 23 2014 5:18am

---

I would also point out that this feature always had a problem if the path to the core library directory had any white spaces. Hence the windows installer placing the install by default into the "C:\Program Files (x86)\Arduino." manifested this same symptom. Where a simple work around was to relocate the path as to not have any white spaces, such as "c:\arduino.".

---

Whereas now the problem is chronic and occurs regardless of paths directory.

[willie68]

I have the same problem
Windows 7,
Arduino Path without spaces, D:\Sprachen\arduino-1.5.7
Sketchbook path: E:\Dropbox\Elektronik\Arduino\projekte\152VO_Uhr\sketchbook
all lib's, sources, everything will be recompiled on every upload...

The 1.5.6 compile only changes.

[matthijskooijman]

Could either of you provide the full compiler output for the first and second verify runs after startup?

I'm not seeing this problem here on linux, so I suspect the problem is Windows-specific. Presumably it fails to get the timestamps of files for some reason, or the timestamp of some file is off (far into the future perhaps)?

[mpflaga]

In short they are identical. There are now timestamps in the compiler output. I have included 7zip bundle of both the first and 2nd build of a blank sketch at https://dl.dropboxusercontent.com/u/99286313/CompileFullOutputOf1stAnd2nd.7z. Where the stdout.txt shows they are identical. The 2nd is appended with the first.

[mpflaga]

a variation of this problem has always existed. In that if the path to the IDE's directory had any white spaces the same would symptom would occur. I don't recall exactly but do recall that I tracked it down to possibly the file object in the java for opening and returning the status of the files was not likely reporting back correctly with white spaces. Where I am not skilled enough fix. but believe that it was not properly quoting the directory name. This smells similiar in that it is about that same file open and status always reporting what ever it is looking for does not exist and wanting to make a clean build.

[PaulStoffregen]

The incorrect handling of spaces was entirely my fault, from 4 years ago when I first wrote this speedup. I mainly use Linux, where spares are rarely used in filenames. Then I tested on Mac and Windows. On Windows, I copy to c:\arduino, because I'm mainly using Linux and that's simpler to copy files over.

I designed the code to "fail safe". There are numerous checks. If any condition fails, the file is recompiled. Only if all pass is the compilation skipped and the old .o file reused.

The problem with spaces involved my incorrect parsing of the .d dependency files created by gcc. The official gcc documentation doesn't mention any special character handling. But when people reported no speedup on Windows, eventually I figured out the issue was the space in "Program Files". By looking at the actual .d files, it turns out gcc adds a backslash in front of the space. I didn't know this, so my original code didn't translate "\ " into " " before comparing strings. Remember, it's designed to "fail safe" if anything doesn't match up.

I fixed this years ago in Teensyduino (where this code was originally developed, years before the Arduino folks accepted my contribution). I don't know if the space parsing ever got fixed in Arduino's version.

Anyway, the point you should take away from this is the code performs MANY checks. It's designed to "fail safe". If any of the files don't appear to exist, or their modification times can't be read (by the Java JRE), or the modification times appear older on either the .o files OR the .d files, or if the .d file isn't exactly the right (expected) format, or anything at all doesn't match up, then it returns false to the compile code, indicating the source needs to be recompiled.

If you're not familiar with the code, it could seem the old spaces parsing bug and whatever has recently failed might be related. The effect is the same. Well, I am familiar with this code. I wrote it. My point is the code is extremely cautious. There are MANY possible ways it can return false, causing an unnecessary recompile. It was intentionally designed with caution. Even with such an approach, it took about 2 years and even a video demo begging people to beta test with regular (non-Teensy) Arduino boards before this was accepted as a contribution to Arduino.

I know you just want your wait time reduced. I feel your pain. I've gone to great lengths to speed things up (just try a Teensy3 someday to see how fast things can be.....) But the last thing anyone wants is incorrectly compiled code, which is why the checks are so cautious. Even though the failure will look similar, unnecessarily recompiling files, there are MANY possible reasons such a bug could happen.

[mpflaga]

Thanks for the history. I think it is key. Noting that 1.5.7 took the leap of using a later version of gcc, as the windows build employes 4.8.1 as opposed to the older 4.3.2. Your insight about the intermediate files lends me to suspect the new gcc may be altering such files in new ways.

I have found your improvement very productive as I play nightly with code that has gotten quite large. I have only once seen an issue requiring a complete rebuild. Where I was messing with changing libraries on the fly. So this was understandable. Otherwise, it has been perfect when not skipped.

[willie68]

Any news on this? I'll find the Version 1.5.7 nearly unusable, because of this issue...

[matthijskooijman]

Hmm, regarding the space escaping @PaulStoffregen Paul mentioned, it seems I fixed something related to that in a recent commit: 726f2ba

Given that the original report says it used to work without spaces in the path, but stopped working somewhere after 1.5.6, I suspect that my change has actually broken things for people (at least on Windows, I've tested this on Linux).

@mpflaga, @willie68, could you post the contents of a .d file generated on Windows?

[matthijskooijman]

Hmm, I'm wondering if \ is used as a path separator as well as or instead of an escape character on Windows...

[willie68]

OK, HardwareSerial.cpp.d First compile Timestamp 02.10.2014 15:58 (German system)

C:\Users\W8858~1.KLA\AppData\Local\Temp\build7047330185438250531.tmp\HardwareSerial.cpp.o: \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.cpp \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\binary.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WCharacter.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WString.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Stream.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Printable.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/new.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\USBAPI.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial_private.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\wiring_private.h

Second compile 2.10.2014 16:01

C:\Users\W8858~1.KLA\AppData\Local\Temp\build7047330185438250531.tmp\HardwareSerial.cpp.o: \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.cpp \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\binary.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WCharacter.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WString.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Stream.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Printable.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/new.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\USBAPI.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial_private.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\wiring_private.h

[mpflaga]

Above I reference a link to drop box that has a 7zip of all the temp files from a 1st and 2nd build. which includes the core.a along with the *.d files that are created. Is there new build with your above mentioned change, as to test. I see the commit, but believe I would need to build a runtime exe, as to test.

[matthijskooijman]

Right, so it uses both \ and / as a path separator (e.g. in E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h) as well as an escape character (at the end of a line).

That certainly explains why my change broke things. I wonder how a space in a pathname is encoded (and even more interestingly, I wonder how a space at the start of a filename looks). I'm trying to fire up my virtualbox to test for myself, but it's not quite working yet. Care to do another test for me?

Could you rename your arduino folder to "E:\Test\ Test\Arduino Test" (so there's a space at the start of the second folder and a space inside of the third folder). If you can also manage to put a backspace inside a folder name, that would be even better, but I think Windows does not allow this.

[matthijskooijman]

Is there new build with your above mentioned change, as to test. I see the commit, but believe I would need to build a runtime exe, as to test.

That commit has been available in the nightlies for a while and is actually the cause of your problem, not the solution :-)

It helps to allow spaces on Linux but, as I see now, it only worsens the situation on Windows.

[willie68]

Windows doesn't allow backspace in names and spaces before the name.
Spaces are Escaped with leading \ "\152VO Uhr" -> "\152VO\ Uhr"

C:\Users\W8858~1.KLA\AppData\Local\Temp\build7929098494296034026.tmp\Dogm\Dogm.cpp.o: \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm\Dogm.cpp \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/Dogm.h \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/utility/dogm128.h \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/utility/dogmpgm.h \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/utility/dogmfont.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/WString.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Printable.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/new.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/binary.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/WCharacter.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/HardwareSerial.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Stream.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/USBAPI.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h

[mpflaga]

Actually it will if you use the dos command mkdir " foo". The Explorer GUI shell removes leading spaces. Note that mkdir " foo\bar" creates a dir of " foo" and a sub directory of "bar". At least on Windows 7.
C:\projects\Arduino>cd " test"
C:\projects\Arduino\ test>mkdir "arduino test"
C:\projects\Arduino\ test>cd "arduino test
C:\projects\Arduino\ test\arduino test>mkdir "foo\bar"
C:\projects\Arduino\ test\arduino test>cd foo
C:\projects\Arduino\ test\arduino test\foo>cd bar
C:\projects\Arduino\ test\arduino test\foo\bar>