Add some initial tests for the fast parser

[ddfisher]

Primarily tests #1600. New fixes to the fast parser will have tests added here.

[ddfisher]

I'm really confused by this CI error. import mypy.fastparse fails on Python 3.3, but only when run from the runtests script (as opposed to directly with scripts/myunit) and only when the import statement is not at the top level.

[gvanrossum]

Weird. I recommend adding print() calls showing sys.path and os.getcwd() to see where you are. Another theory: typed_ast wasn't actually successfully built and installed in 3.3 and that's causing it. Is it installed from a Linux wheel? What Python version was present when you built the wheels?

[ddfisher]

I was able to repro locally, where I checked that typed_ast did get installed successfully, so it's not related to that. (Also the error happens well before fastparse tries to import typed_ast.)

Also: there are no Linux wheels. Are Linux wheels even supported yet?

[ddfisher]

I added sys.path and cwd printouts: sys.path remains unchanged from the top level, but the cwd changes from mypy to mypy/tmp-test-dirs/mypy-test-qhg83l -- that must be it somehow.

[ddfisher]

Hmm. The mypy directory is in my sys.path, though:
['', '/Users/ddfisher/src/mypy/lib-typing/3.2', '/Users/ddfisher/src/mypy', '/Users/ddfisher/.pyenv/versions/3.3.6/lib/python33.zip', '/Users/ddfisher/.pyenv/versions/3.3.6/lib/python3.3', '/Users/ddfisher/.pyenv/versions/3.3.6/lib/python3.3/plat-darwin', '/Users/ddfisher/.pyenv/versions/3.3.6/lib/python3.3/lib-dynload', '/Users/ddfisher/.pyenv/versions/3.3.6/lib/python3.3/site-packages']

[gvanrossum]

Maybe there's another subpackage named mypy that doesn't have fastparser?
Or maybe an old mypy is installed? Or maybe the initial mypy package is
an old one that doesn't have it?

[gvanrossum]

Try printing mypy.file.

[ddfisher]

I figured it out. It's nothing to do with fastparse in particular (and everything to do with the fact that it's imported in a function). The final import in this code will fail in Python 3.3, but succeed in Python 3.4 and 3.5:

import os
import sys

sys.path = ['', '/Users/ddfisher/src/mypy'] + sys.path

import mypy
os.chdir('tmp')
import mypy.fastparse

The sys.path modifications I make here mirror those I see when runtests.py is invoked. Removing the empty string from the beginning of sys.path fixes the problem, so I'm going to try to make that change to the test runner.

[ddfisher]

Making that change to the test runner doesn't seem to fix it. Ugh.

[ddfisher]

Python seems to be prepending an empty string to sys.path (and I think I'm filtering it out too late). Is there any way to prevent it from doing it? As far as I can tell, it only happens when you run python -m [module] (and not when you just run python [file].

[gvanrossum]

The actual rule is that it prepends the directory where the file lives, or
'' if there is no file (e.g. when reading string or using -m). I think what
happens is that the file of the mypy package ends up being a relative
path in Python 3.3. A possible fix may be to go over mypy.path and make
all entries in there absolute.

--Guido (mobile)

[ddfisher]

That works! I'll try to write a good comment about it.

[ddfisher]

... and now Travis is down.

[ddfisher]

Checks pass!

[gvanrossum]

Worth filing an issue for.

[ddfisher]

Looks like there's already one, actually: #1422

[gvanrossum]

Whew! Epic.

[ddfisher]

Thanks for the help! Definitely didn't expect this to be so hard. ._.

[gvanrossum]

Or those delayed imports are somehow executed sooner. There's something generally iffy about the way fastparse.py is handled. Fortunately that will soon be behind us!