Question: usage as a library

[bchurchill]

I'm developing a static analysis for python code that depends on type information. I'd really like to use an existing type analysis instead of building my own! Is it possible to use pytype as a library?

The best case scenario for me would be to annotate every node of an AST with some kind of overapproximation of the type information (e.g. a list of possible types, a type with a hole in it, etc.). For example, I would like to write code like this:

my_ast = ast.parse(some_python_code)
typed_ast = pytype.analyze(my_ast)
...
class MyVisitor(ast.NodeVisitor):

  def generic_visit(self,node):
      print("This node has type " + str(node.type))

Is something like this possible?

[martindemello]

pytype-as-a-library is a work in progress, and not documented much, but check out tools/traces, tools/annotate_ast and tools/xref for some examples of leveraging pytype for type analysis.

since pytype itself works with bytecode rather than the AST, the primary mechanism for exporting types for reuse is a series of "opcode traces", consisting of an opcode and some pytype data associated with it (typically the pytype object that represents the opcode's argument, e.g. (opcode=MAKE_FUNCTION, data=<abstract.InterpreterFunction>). the data points into a live pytype vm at the time of generation, and can either be converted into abstract types (the approach taken by tools/traces) or used as is, with the vm preserved so the live objects stay valid (the approach taken by tools/xref).

i'm afraid that right now this is all new (and incomplete) enough that poking through the source code is the best way to get started, but it's definitely usable. in particular, https://github.com/google/pytype/blob/master/pytype/tools/traces/traces_test.py is worth looking at to see if it does the kind of thing you have in mind.

[bchurchill]

Thanks for the quick reply! Much appreciated. How difficult do you think it is to get AST data out from the traces? If it's not so easy, do you know of any other tools that might do an OK job at this?

[martindemello]

it's reasonably hard because it involves reconcilng the ast and bytecode views of a program. it's definitely something on our roadmap to provide as a public library, but we've only recently gotten started on it (https://github.com/google/pytype/blob/master/pytype/tools/annotate_ast/annotate_ast.py)

that said, there's a lot of code in https://github.com/google/pytype/blob/master/pytype/tools/xref/indexer.py that walks the AST and captures symbol definitions and references complete with types from pytype; it was unfortunately written to get the job done rather than to be easily reusable or generic, and so will not do much for nodes that are expressions rather than symbols, but if all you want is variable types you could consume them from the generated index.

[martindemello]

(i'm not aware of any other typechecker that does this either; facebook's pyre might do it eventually; i know some fb people at pycon were talking about it. there's an open issue for mypy: python/mypy#4868)

[KevinHock]

Just wanted to say, thank you both for the interesting discussion.

[bchurchill]

Thanks for the help!

[kamahen]

I have some code that generates Kythe facts for a program and its imports, and can easily be adapted to other output formats. It does some type inferencing although not as complete as pytype (I'll eventually incorporate pytype's output). So, eventually, this should give the exact token information, and be able to incorporate of all of pytype's analysis.
It's "90%" done (with the usual caveats). It's currently written in a combination of Python and Prolog; I intend to eventually convert it to be all in Python. (It's a an open-source hobby project.)