Move built-in examples to dedicated repository

[per1234]

A dedicated repository for the examples will make it easier for them to be built in to arbitrary development tools. It will make it easier to use them for compilation testing of boards platforms. It provides a dedicated location for issue reports and pull requests that are specific to the examples. It continues the work done by moving the AVR and SAM boards platforms and built-in libraries towards making the arduino/Arduino repository solely a place to host the GUI code of the Arduino IDE.

In addition to the example sketches, I also moved the "examples-formatter" script and configuration file to the examples repository:
https://github.com/arduino/arduino-examples

I have updated the build system to pull the examples from the dedicated repository, using the same approach as done for the bundled libraries.

One of the unit tests was using the Blink sketch from the examples, so I have moved that sketch into app/testdata/sketches and updated the test accordingly.

I have updated documentation to reflect the change, including following the precedent of what was done for the boards packages by adding a readme to the now empty build/shared/examples folder.

All Submissions:

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

[matthijskooijman]

This PR broke the build for me. I get:

untar-unzip-checksum:                                                          
     [echo] Testing checksum of "arduino-examples-1.9.0.zip"               
                                                                               
unzip:                                                                         
                                                                               
BUILD FAILED 
/home/matthijs/docs/Electronics/Arduino/Arduino/build/build.xml:136: The following error occurred while executing this line:
/home/matthijs/docs/Electronics/Arduino/Arduino/build/build.xml:663: The following error occurred while executing this line:
/home/matthijs/docs/Electronics/Arduino/Arduino/build/build.xml:214: The following error occurred while executing this line:
/home/matthijs/docs/Electronics/Arduino/Arduino/build/build.xml:252: /home/matthijs/docs/Electronics/Arduino/Arduino/build/shared/arduino-examples-1.9.0/examples does not exist.

The .zip file is downloaded properly, but it looks like it is not unzipped. I suspect this is because the unzip target checks for the presence of the $final_folder, which is linux/work/examples in this case, and skips the unzip if the folder is already available. That folder is available from an earlier build before this PR, so running ant clean fixes this. However, I suspect the same issue could happen again after an examples version bump. Not sure if this is unique to examples, or can happen with the other downloaded stuff too, though.

Note sure what the right fix here is, but

[cmaglie]

The build works with no problems for me...

The folder that is checked is not build/linux/work/examples but build/shared/arduino-examples-1.9.0 (and later we copy the examples folder inside that one).
I managed to reproduce the error you posted above by making an empty build/shared/arduino-examples-1.9.0 so the unzip target does not unpack the archive thinking that is already unpacked. Honestly, I don't know how you end up in this situation, looks really weird, handling this situation means that we should check that the content of the zip is actually unpacked and not just look if the top level folder exists... IMHO it's not worth it, considering that we use the same unzip target for all the other archives.

[matthijskooijman]

If i do rm -rf build/shared/arduino-examples-1.9.0/ and then run ant, I get the same error again.

The folder that is checked is not build/linux/work/examples but build/shared/arduino-examples-1.9.0 (and later we copy the examples folder inside that one).

Nope, it does check build/linux/work/examples.

The unzip rule checks for the ..._installed property:

Arduino/build/build.xml

Line 939 in 7a9234c

<target name="unzip" depends="untar-unzip-checksum" unless="${archive_file}_installed">

Which gets set if the final_folder exists:

Arduino/build/build.xml

Line 887 in 7a9234c

<available file="${final_folder}" property="${archive_file}_installed" />

And final_folder is build/linux/work/examples:

Arduino/build/build.xml

Line 239 in 7a9234c

<property name="BUILT-IN-EXAMPLES-FINAL-PATH" value="${target.path}/examples" />

Hence, if I do rm -rf build/linux/work/examples (removing the final_folder), the build works again, because now unzip works again.

Other uses of unzip

For libraries, the final_folder is the unpacked directory, which is then moved into another place. This means that these will re-unpack and move on every build.

Arduino/build/build.xml

Lines 295 to 301 in 7a9234c

	<antcall target="unzip">
	<param name="archive_file" value="@{foldername}-@{version}.zip"/>
	<param name="archive_url" value="https://github.com/@{githubuser}/@{name}/archive/@{tag}.zip"/>
	<param name="final_folder" value="${target.path}/libraries/@{name}-@{version}"/>
	<param name="dest_folder" value="${target.path}/libraries"/>
	</antcall>
	<move file="${target.path}/libraries/@{name}-@{version}" tofile="${target.path}/libraries/@{foldername}"/>

For the Wifi updater tool, AVR core, docs, appbundler, ... the unzip puts it in the final location right away, without further copy or move. I think this means when these get a version bump, the new version is downloaded, but not unpacked and otherwise ignored.

Arduino/build/build.xml

Lines 308 to 313 in 7a9234c

	<antcall target="untar-bz2">
	<param name="archive_file" value="avr-${AVRCORE-VERSION}.tar.bz2"/>
	<param name="archive_url" value="https://downloads.arduino.cc/cores/avr-${AVRCORE-VERSION}.tar.bz2"/>
	<param name="final_folder" value="${target.path}/hardware/arduino/avr"/>
	<param name="dest_folder" value="${target.path}/hardware/arduino"/>
	</antcall>

For the java libraries (libastylej, liblistserials), the final_folder is also the unpacked directory, which is then copied into another place. This means that these will not be unpacked, but copied again on every build.

Arduino/build/build.xml

Lines 493 to 499 in 7a9234c

	<antcall target="unzip">
	<param name="archive_file" value="./libastylej-2.05.1-5.zip" />
	<param name="archive_url" value="https://downloads.arduino.cc/libastylej-2.05.1-5.zip" />
	<param name="final_folder" value="${staging_folder}/libastylej-2.05.1" />
	<param name="dest_folder" value="${staging_folder}" />
	</antcall>
	<copy file="macosx/libastylej-2.05.1/libastylej.jnilib" todir="macosx/work/${staging_hardware_folder}/../lib/" />

There's some more uses, which I didn't check.

So even though those other uses do have their problems (notably unpacking and copying when not really needed), they do not have the same problem as the examples, that the build breaks when updating to a new version.

[per1234]

Hi @matthijskooijman. I apologize if my change to the build system caused you some difficulties.

I've been trying to reproduce the issue, but have not been able to.

Here's what I've done:

git clone -b investigate-examples-build-issue --depth 3 https://github.com/per1234/Arduino
cd Arduino/build
# checkout the ref from before the change to the build system
git checkout 2fd59442311b88d126772deda7c993f6c974cbbe
ant dist
ant run
# checkout the ref for the change to the build system, which gets built-in examples version 1.9.0 from the dedicated repo
git checkout 7a9234c30461c728441aeab393763b39fc4fce64
ant clean build start
# checkout the ref that "updates" the built-in examples version to 1.8.13
git checkout 706029a61d1feae5f2f07bb76220a8f1d83c1fe1
ant clean build start

Every build works as expected for me.

Would you mind sharing a set of steps I can use to reproduce the bug so I can investigate this further?

[matthijskooijman]

Hi @matthijskooijman. I apologize if my change to the build system caused you some difficulties.

No worries, that's just how progress works :-)

In your steps, you run ant clean in between, which indeed makes the build work, the problem only exists when you don't run clean. Of course, maybe it is sane to run clean whenever you change the git revision anyway, so this might be a non-issue, but given that most of the build works properly incrementally, it would be nice if this also worked. OTOH, if we say "whenever you pull, you should also clean", it might be worthwile to also fix the other things I mentioned above (files that are unpacked or copied again on every build, fixing those would speed up incremental builds after small java-only changes significantly).

Here's a modified version of your commands without clean, that shows the breakage I meant:

git clone -b investigate-examples-build-issue --depth 3 https://github.com/per1234/Arduino
cd Arduino/build
# checkout the ref from before the change to the build system
git checkout 2fd59442311b88d126772deda7c993f6c974cbbe
ant build # works
# checkout the ref for the change to the build system, which gets built-in examples version 1.9.0 from the dedicated repo
git checkout 7a9234c30461c728441aeab393763b39fc4fce64
ant build # fails
ant clean build # to fix
# checkout the ref that "updates" the built-in examples version to 1.8.13
git checkout 706029a61d1feae5f2f07bb76220a8f1d83c1fe1
ant build # fails

I'm not 100% sure about the last failure (I forgot the build before the last checkout and just did a clean initially, so I ended up with a succesful build at first and had go back and remove the 1.8.13 stuff, but I think that would also happen with the above steps).