Bazel build integration

[sixolet]

I've been playing with getting Mypy to integrate with bazel for typechecking. I've got a basic version of it working. I intend to describe it here for discussion of the general approach, and then post a series of PRs for individual things it needs as I clean them up, if the general approach seems right.

The general approach: Make a mypy_library and mypy_binary macro that puts a typechecking aspect on the python files it contains. The aspect calculates the mypy cache files; give mypy a mode where it stops considering itself responsible for calculating freshness of dependencies, and instead let bazel handle that.

Specific PRs I expect to make:

• A --cache-hermetically option to stop writing mtimes
• A --cache-mapping option to specify a json file containing .data.json and .meta.json file locations by module id, both for cache files to read and new cache files to write (bazel likes being super specific about things like this; it seems cleaner to specify to mypy than to rely on mypy's cache directory layout). This option would also short-circuit the process of looking for source and checking it for freshness for the covered modules; just believe the cache files bazel is handing you.
• Possibly a --module-without-init option (or some name, I don't know what it should be called) to allow mypy to believe things are modules without an __init__.py file. Bazel for some reason doesn't care whether you have them in your python source when you build. I can possibly hack around this by touching that file myself in the build directory in the bazel rule, instead.
• A --ignore-module-prefix option to understand that certain parts of the module path are already accounted for, and all modules within the prefix to ignore should be treated as if they have empty __init__.py.
• An actual Skylark implementation of the python typechecking macros, and a BUILD file for mypy.

That would be enough to get it working (and I have a prototype of that, which I need to split into separate PRs and clean up a lot if it seems an ok approach). To make it awesomer:

• Speed up mypy's startup time. There might be low-hanging fruit?
• Precompute the cache files for builtins and typeshed, so we never need to go looking for source there.
• Implement some dependency pruning, so you don't even need to load all the cache files all the time.

I'm submitting this description before I'm ready to submit PRs deliberately, so I am not super attached to any one approach. Suggestions?

[emmatyping]

--module-without-init

I don't think this is a good idea. The issue sounds more to be with Bazel than Mypy, and I think it makes more sense to modify the build script until Bazel fixes this.

Precompute the cache files for builtins and typeshed, so we never need to go looking for source there.

I really, really like this idea. The typeshed information should be immutable therefore this makes a lot of sense. If someone is using a custom typeshed directory, it might get a bit more complicated, but in the general case and for releases, this will be really nice. Mypy currently spends 1.5 seconds on an empty file, so I do hope there are ways to improve startup time.

[JukkaL]

Some thoughts below. Note that this our internal team priority is to speed up mypy though a daemon mode instead of a build tool like bazel. Our review bandwidth may be lower than normal until the end of year, so there's a risk that bigger PRs in particular will be slow to review. It's better if we can first reach agreement on the design of new features at an abstract level before looking at PRs, as design reviews are much less work (and can save also implementation work in case the design gets iterated on a lot).

Speed up mypy's startup time. There might be low-hanging fruit?

I have a few ideas that may be helpful:

• Make sure you don't compile the .pyc/.pyo files during each mypy run.
• Remove type annotations in the released mypy version. Initializing annotations takes some time. After the recent optimizations the potential savings might be marginal, though.
• Profile the time needed to evaluate the top level of each module in mypy, and see if there are things that are slower than expected. Named tuples are known to slow down program startup, for example.
• See if a lot of code gets imported that isn't actually needed for most runs, and see if they can be lazily imported. Running code coverage might be an easy way to do this.

The non-bazel answer is to run mypy as a daemon and to use a small client that starts up very quickly. However, this probably is not a good match for bazel (though I've heard that it's possible). A related idea is to somehow keep a fully initialized mypy process running and just fork it whenever a fresh run is needed. This would provide isolation between mypy runs.

Precompute the cache files for builtins and typeshed, so we never need to go looking for source there.

This can't happen in mypy itself, since the cache files depend on mypy options such as platform and target Python version. However, it may make sense to have a build artifact in your bazel setup for typeshed with bundled cache files, and to use it instead of the standard typeshed though --custom-typeshed-dir.

A --cache-hermetically option to stop writing mtimes

Before adding a large number of command-line options, I'd like to see the output of mypy --help cleaned up so that it only shows the most commonly used options. We could point to the docs for information about other options, or perhaps we could have a separate help option for displaying verbose help with all options.

Implement some dependency pruning, so you don't even need to load all the cache files all the time.

Something like this could be pretty useful. Can you create a separate issue for this? This would be useful outside bazel as well.

[gvanrossum]

Sorry for the slow response!

Regarding --module-without-init: this is separately being proposed as --namespace-packages in a different issue: #2773. I'm thinking you'd like it to apply to Python 2 too? If so, please add a note there (the other org asking for this, Instagram, are on Python 3).

I'm not so worried about the proliferating number of command line flags -- at least, I don't think you need to pay for it before you can add new flags.

I think a --cache-hermetically flag would be pretty simple and self-contained, so go ahead and send us a PR for that. Do understand that it would only suppress the "mtime", not the option values contained in the meta files. (Actually it might also suppress the "path" field? Because the hash is the only things that matters.)

I'm not sure what the benefits of your propose --cache-mapping would be. IIUC this would be a dict giving e.g.

{"mypy.build": [".mypy_cache/3.6/mypy/build.meta.json", ".mypy_cache/3.6/mypy/build.data.json"],
 ...}

The contents of such a file (IIUC) should be entirely predictable given the values of --cache-dir and --python-version. But perhaps you care more about the "just believe the cache" part of the flag? Or perhaps that could be folded into --cache-hermetically? Hm, now I'm beginning to doubt my understanding of your proposed workflow.

I'm not sure what --ignore-module-prefix is supposed to do. What exactly do you mean by "module path"? The search path used to find module files, or the pathname of the module file?

[JukkaL]

I'm not so worried about the proliferating number of command line flags -- at least, I don't think you need to pay for it before you can add new flags.

Yeah, this should be up to the core team, since we are the ones who've added (or accepted) all the existing command-line options :-)

I have some vague ideas about improving mypy's startup time and cache deserialization speed, but I'm not yet sure if these are practical and when these will happen. Lazily loading only necessary dependent cache files of indirectly imported modules might be some relatively low-hanging fruit?

[JukkaL]

I said earlier:

Our review bandwidth may be lower than normal until the end of year

This may not actually need to be the case. I'm cautiously optimistic.

[dgoldstein0]

any plan to actually do any of this work?

btw we've also seen the bazel-python problem around init.py and discussed it a bunch internally. My quick skim of the --namespace-packages proposal makes it sound like it'll solve our problem if we get py2 support for it.

Anyhow I hadn't looked into the internals of mypy at all and this proposal seems to cover that pretty well. From a bazel POV though, what we need is a way to invoke mypy in two (new?) modes:

• give it a set of python and types files (perhaps .pyi) and generate a new type file output - so we can implement a build step that summarizes the types of a library, for it's dependencies to use
• give it a set of python and types files of dependencies and typecheck the python - so we can typecheck a library, based on it's dependency libraries types

in addition to all the above stuff about making mypy hermetic and care less about existence of init files.

[dgoldstein0]

and to clarify - if mypy has those two modes, I'd be willing to take a stab at writing the skylark to invoke mypy within bazel

[gvanrossum]

Would you be willing to also take a stab at implementing those two modes? We're pretty booked up with other perf work already.

[dgoldstein0]

unfortunately I have no familiarity with the mypy codebase so I'm not comfortable committing to anything like that. I'm also pretty busy with my actual day job, but if you convince me it's not too hard then maybe I'd look into it. But jumping into a new codebase I have no confidence this would be quick.

…

On Wed, Nov 1, 2017 at 7:37 PM Guido van Rossum ***@***.***> wrote: Would you be willing to also take a stab at implementing those two modes? We're pretty booked up with other perf work already. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3912 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBnd8Ya4Adc3BJYCzug710MxIBdr0arks5sySrngaJpZM4PLGh4> .

[JukkaL]

give it a set of python and types files (perhaps .pyi) and generate a new type file output - so we can implement a build step that summarizes the types of a library, for it's dependencies to use

Using the existing JSON serialized cache files for this would be both faster and less work to implement, at the cost of not having an easily human-readable representation.

The mypy startup performance was again brought up by @sixolet last week (offline). Here pruning dependencies (either by not loading some dependencies at all or by loading them lazily) would likely be the low-hanging fruit. Other possible optimizations are harder:

• Speed up importing the mypy implementation and dependencies. Pruning internal mypy dependencies might help a bit, and Python 3.7 might also improve this a little. Larger improvements will likely require somewhat drastic measures -- but these are still within the realm of possibility.
• Make deserialization of JSON cache files faster. Again, this is non-trivial but conceptually it should be possible to make this much faster.

[sixolet]

I made a proof of concept a couple months ago. I still plan to turn it into real PRs as soon as I dig myself out of all my free time going to baby care :P

…

On Thu, Nov 2, 2017 at 4:58 AM, Jukka Lehtosalo ***@***.***> wrote: give it a set of python and types files (perhaps .pyi) and generate a new type file output - so we can implement a build step that summarizes the types of a library, for it's dependencies to use Using the existing JSON serialized cache files for this would be both faster and less work to implement, at the cost of not having an easily human-readable representation. The mypy startup performance was again brought up by @sixolet <https://github.com/sixolet> last week (offline). Here pruning dependencies (either by not loading some dependencies at all or by loading them lazily) would likely be the low-hanging fruit. Other possible optimizations are harder: - Speed up importing the mypy implementation and dependencies. Pruning internal mypy dependencies might help a bit, and Python 3.7 might also improve this a little. Larger improvements will likely require somewhat drastic measures -- but these are still within the realm of possibility. - Make deserialization of JSON cache files faster. Again, this is non-trivial but conceptually it should be possible to make this much faster. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3912 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjs4Xaygd4iq0Dh_FUt22dVRFbErhx1ks5sya58gaJpZM4PLGh4> .

[gvanrossum]

I am trying to wrap my head around this. I'm still reading up on bazel but I think this is going to be fun!

[kamahen]

A few thoughts from someone who has done something similar ...

The mypy cache could be specified as one of the outputs in the mypy aspect's action (and input(s) to other rules). Then Bazel would take care of it for you and re-run only what's necessary.

Bazel allows only one rule for creating an output. But your BUILD files probably have some rules that specify the same source files as other rules. There are a few solutions to this, but probably the easiest is to add the rule name to the output file (to make it unique) and have the Skylark create a mapping file that is input to mypy to map between the Bazel label and the actual file. You can see an example of this in pytype/imports_map_loader.py.
Another way would be similar to how Java rules work with .class files.

@gvanrossum - trust me, this quickly becomes not-fun. However, the documentation is much better now than when I did this, and Skylark has far fewer bugs now (but be warned -- it is not Python).