Enable mpy-cross compilation of some/all __init__.py files in libraries

[kevinjwalters]

The current build system intentionally does not apply mpy-cross to __init__.py to produce a compiled __init__.mpy. I noticed this whilst checking the latest adafruit_midi library in the bundle as it blows up one of my programs with a MemoryError - I had previously been manually compiling all the .py files.

@sommersoft pointed me to the relevant line in the build system, this is an intentional omission.

IGNORE_PY = ["setup.py", "conf.py", "__init__.py"]

In the absence of comments or other external information I'll speculate the purpose of this was to disable compilation for:

• ./setup.py - pointless as it's not part of the library code
• ./docs/conf.py - pointless (ditto)
• __init__.py - not sure here, this could be because many are 0 bytes and mpy-cross will make a bigger (!) .mpy - there's some additional special casing hard-coded in too.

There's some additional comments on this in adafruit/Adafruit_CircuitPython_MIDI#3 (comment).

If my understanding of the functionality and exclusions here is correct this would be better implemetented as a fully pathed exclusion of ./setup.py and ./docs/conf.py and an extra list of unpathed files to exclude from compilation if 0 bytes, or possibly a general rule of do not mpy-cross compile if less than or equal to XX bytes?

[kevinjwalters]

Quick survey of non-zero length __init__.py files from adafruit-circuitpython-bundle-4.x-mpy-20190425 bundle (first one will get smaller with adafruit/Adafruit_CircuitPython_MIDI#13):

adafruit_midi/__init__.py                           9186
adafruit_imageload/__init__.py                      2249
adafruit_bluefruit_connect/__in...                  1930
adafruit_ble/__init__.py                            1819
adafruit_hid/__init__.py                            1380
adafruit_thermal_printer/__init...                   982
adafruit_tinylora/__init__.py                        177
adafruit_rgb_display/__init__.py                     106
adafruit_apds9960/__init__.py                         43

[sommersoft]

So, after some deeper diving, I found that I pointed to the wrong block previously. The part that precludes __init__.py from mpy-cross is here, and looks like so:

 if (not mpy_cross or
     os.stat(full_path).st_size == 0 or
     filename.endswith("__init__.py")):
                output_file = os.path.join(output_directory, filename)
                shutil.copyfile(temp_file.name, output_file)

This basically forces a straight file copy of the .py. Also, the above only applies to packaged libraries. With the os.stat().st_size == 0 condition, I think it's safe to remove the __init__.py condition. I'll run some iterations. @kevinjwalters, I'll put up a version of a midi bundle in a few, if you wouldn't mind testing.

The previously cited IGNORE_PY = ["setup.py", "conf.py", "__init__.py"] only applies to un-packaged libraries, so can probably remain.

[sommersoft]

@kevinjwalters here's a whole bundle, with __init__.pys, mpy'd.

test-4.x-mpy-2d7b358.zip

[kevinjwalters]

That bundle looks and works fine, I tried it on two CPXs I've borrowed with 4.0.0 rc1.

I had used the manually compiled __init__.py for several weeks beforehand on various 4.0.0 betas without incident, btw.

[kevinjwalters]

What's approximate ETA for getting this fix out there and what's likely to be the first bundle produced with this?

I ask because I'd like to know whether I need to include an interim step in a Learn guide to source adafruit_midi library from somewhere with compiled mpy before the official bundle arrives.

[kevinjwalters]

On a loosely-related note, the adafruit_midi library does't currently appear on https://circuitpython.readthedocs.io/en/latest/docs/drivers.html#adafruit-circuitpython-library-bundle - the page says its source is https://github.com/adafruit/Adafruit_CircuitPython_Bundle/blob/master/docs/index.rst which appears to use some sort of search/generated includes. How does the library get added to this page?

[kevinjwalters]

BTW, there's a tiny chance this was to deal with micropython/micropython#2680 logged by @tdicola? That was quickly fixed. The rollout of mpy support seems rather phased across different hardware platforms so I'm not sure when/if/how that impacted CircuitPython?

[sommersoft]

What's approximate ETA for getting this fix out there and what's likely to be the first bundle produced with this?

With PyCon going on this week, I can't say for sure. I could merge the PR, but I'd rather have it reviewed before that.

On a loosely-related note, the adafruit_midi library does't currently appear on RTD

The entries are in the /docs/drivers.rst file. I can do up a PR real quick.

[sommersoft]

Actually, it looks like RTD was never setup for adafruit_midi; I can only find usb_midi from the core. I'll put up an issue for that.

RTD Issue

[sommersoft]

PR #31 didn't work out well upon deployments to PyPI. So, it was reverted. Reopening this so it doesn't get lost...