Initial version of API module

[JdeH]

Hi,

On invitation of gvanrossum I've designed a simplified API for mypy, meant to enable easy use of mypy as module rather than as stand-alone program.

The main goals in designing this API were to keep everything simple and explicit. Not all functionality of mypy is currently accessible via the API. Still this initial version at least helped me to indeed use mypy as a module in a straightforward way.

The API is non-intrusive in this sense that for this initial version deliberately no modifications were made to the rest of mypy. For easy reference, below there is a brief description of the API, that's also part of the source code:

# This module contains an API for using mypy as a module inside another (host) application,
# rather than from the command line or as a separate process. Examples of such host applications
# are IDE's and command line build- or (pre)compilation-tools. Once this API is stable and
# flexible enough, it deserves consideration to make the command line version of mypy just another
# host application.
#
# Being an interface, this module attempts to be thin, stable and self-explanatory. Since the guts
# of mypy are bound to change, it doesn't depend too much upon them. Rather it presents an
# external view of mypy, using a very limited number of domain bound, hence presumably relativey
# stable concepts.
#
# More specific, it exports:
#
#   -   A singleton object named type_validator, representing mypy to the host application. This
#       object features two methods:
#
#       -   Method set_options, which allows setting those options of mypy which are meant for
#           production use. Its argument list makes clear which they are and which defaults they
#           have.
#
#       -   Method validate, which receives a list of strings, denoting source file paths of top
#           level modules. These top level modules and the modules they import are checked
#           recursively. Method validate returns a polymorphic list containing objects whose class
#           derives from ValidationMessage.
#
#   -   Class ValidationMessage. This class facilitates the use of its subclasses in a
#       polymorphic, but still typed, list. In most situations there's no need to use this
#       baseclass directly. Objects of its subclasses represent messages that the validator wants
#       to deliver to the user via the host. Such objects are in binary form, granting the host
#       the freedom to convert them to any suitable format.
#
#       -   Class ValidationRemark is a subclass of ValidationMessage. It is the baseclass of all
#           ValidationMessage's that do not represent an error.
#
#       -   Class ValidationError is also a subclass of ValidationMessage. It is the  baseclass
#           of all errors encountered during a valiation run. In most situations there's no need
#           to use this baseclass directly. There is no separate warning class, rather objects of
#           some subclasses of ValidationError can have an severity attribute.
#
#           -   Class StaticTypingError is a subclass of ValidationError. Its instances represent
#               static typing inconsistencies found by mypy. Finding objects of this class is what
#               mypy is about.
#
#           -   Class CompilationError is a subclass of ValidationError. Its instances represent
#               any other problem encountered by mypy. Currently this category isn't subdivided
#               any further. Its derived classes, which are currently unused, suggest that in the
#               future such a subdivision may be useful.
#
#               -   Class SyntaxError is a subclass of CompilationError. Its instances represent
#                   syntax errors encountered by mypy in the code under scrutiny. It is there for
#                   future use. While in the end the Python interpreter will catch any syntax
#                   errors, if mypy already knows about them, a second parse is redundant and can
#                   be avoided.
#
#               -   Class InternalError is a subclass of CompilationError. Its instances represent
#                   errors due to malfunction of mypy itself. It is there for future use.

I've tested the correct functioning of the API in the Transcrypt Python to JavaScript translator. While this test is by no means exhaustive, the API indeed provided very easy integration of mypy in Transcrypt. As an example here's the code of the Transcrypt module that uses the new mypy API and subsequently logs messages in a proprietary, hierarchical format, per file and, within that, per line.

Activating mypy is simply done in the following line (settings are just the defaults):

validationMessages = api.type_validator.validate ([sourcePath])

Everything mypy wants the world to know is in a polymorphic list returned by the validator, in a simple, non-textual format, documented explicitly in the API module, suitable for further processing. No exceptions to be caught, no special treatment or required dissection of CompileErrors, no need to read through other modules of mypy. The ValidationMessage class hierarchy can be expanded at will as mypy evolves, not changing anything to the API.

The list of ValidationMessage's is used by the surrounding code of Transcrypt as follows:

(Note that while the code below is formatted according to non-PEP standards since it coexists with C++ and JS code, the API itself follows PEP)

import traceback
from mypy import api
from org.transcrypt import utils

def run (sourcePath):
    try:
        utils.log (True, 'Performing static type validation on application: {}\n', sourcePath)

        validationMessages = api.type_validator.validate ([sourcePath])

        if validationMessages:
            oldFileName = ''
            for validationMessage in validationMessages:
                if isinstance (validationMessage, api.StaticTypingError):
                    if validationMessage.file_name != oldFileName:
                        utils.log (True, '\tFile {}\n', validationMessage.file_name)
                        oldFileName = validationMessage.file_name
                    utils.log (True, '\t\tLine {}: {}\n', validationMessage.line_nr, validationMessage.description)
                elif isinstance (validationError, api.CompilationError):
                    utils.log (True, '\t{}'.format (message))
                else:
                    util.log (True, '\tUnknown error')
            utils.log (True, '\n')
    except Exception as exception:  # Just to be sure, not required by API
        utils.log (False, traceback.format_exc ())
        raise exception

Kind regards
Jacques de Hooge

[gvanrossum]

Thanks for giving it a shot. In my personal opinion this API is too complex and too constraining for mypy's implementation.

I'd like to propose something simpler, closer to the command line. You'd pass it a list of files (or possibly files and directories) plus an Options object (see mypy/options.py) and it returns a possibly-empty list of tuples describing errors with the same format as returned by render_messages() in mypy/errors.py.

What would you think of that? It's less likely to be stable forever, but I think it's too early to commit to an API that's entirely future-proof. So I'd like to give you something that's just short of having to parse the errors from a subprocess.

PS. If you submit an implementation of that, could you please use docstrings instead of long comments?

[JdeH]

Hi, thanks for the feedback

While in the long run I think an error class hierarchy is a good idea, because it makes it easy catch specific errors first and then have a wider net for the more general ones, it may be too specific and restricting now. So I'll take a look at using tuples as you propose, as well as passing in an Options object as a whole, rather than separate options.

Can you clarify a bit more what you mean by 'closer to the command line'? Since you propose to use an Options object, it's my understanding that you do not want to use the command line switch syntax to pass in options. Or do I misunderstand you here? Which aspect of the API would you like to be closer to the command line?

About comments versus docstrings: I will adapt them.

[gvanrossum]

I just meant that the Options object is a pretty good representation of
most of the flags that can be passed through the command line; I don't
think that specifying the options as strings would be better. But if you
prefer passing a list of flags that must be parsed using argparse that's
fine with me too.

Regarding syntax highlighters, it's not just your syntax highlighter that
matters. And perhaps you could switch to PyCharm? The community edition is
free!

On Mon, Sep 12, 2016 at 11:09 PM, Jacques de Hooge <[email protected]

wrote:

Hi, thanks for the feedback

While in the long run I think an error class hierarchy is a good idea,
because it makes it easy catch specific errors first and then have a wider
net for the more general ones, it may be too specific and restricting now.
So I'll take a look at using tuples as you propose, as well as passing in
an Options object as a whole, rather than separate options.

Can you clarify a bit more what you mean 'closer to the command line'?
Since you propose to use an Options object, it's my understanding that you
do not want to use the command line switch syntax to pass in options. Or do
I misunderstand you here? Which aspect of the API would you like to be
closer to the command line?

About the docstrings: The syntax highlighter of Notepad++ (yes, I develop
on Windows) makes them look exactly like normal strings, which I find
confusing. But I guess a syntax highlighter should not determine the way I
code... So I'll adapt them as well.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#2114 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMo8O0eTfZFnGrRK3W3MKDPbz_KnIks5qpj4qgaJpZM4J40Xw
.

--Guido van Rossum (python.org/~guido)

[gvanrossum]

This seems to be lingering. I'm closing it now, since we're not going with this version. If you have a new version, please just open a new PR. Thanks!

[JdeH]

OK, I am rather busy these days. But it's still on my to do list.

[gvanrossum]

No problem. Hope to hear from you soon!