--target doesn't work for namespace packages

[webmaven]

When doing a local install of a namespace package, the .pth file doesn't get picked up for some reason:

MAC-C08182:projects c08182$ mkdir targettest
MAC-C08182:projects c08182$ cd targettest/
MAC-C08182:targettest c08182$ pip install repoze.lru -t lib/
Downloading/unpacking repoze.lru
  Downloading repoze.lru-0.6.tar.gz
  Running setup.py (path:/private/var/folders/9w/_wt1czb57k9c5b0hgg6dfl2x7jbl85/T/pip_build_c08182/repoze.lru/setup.py) egg_info for package repoze.lru

Installing collected packages: repoze.lru
  Running setup.py install for repoze.lru

    Skipping installation of /var/folders/9w/_wt1czb57k9c5b0hgg6dfl2x7jbl85/T/tmp7uY1G8/lib/python/repoze/__init__.py (namespace package)
    Installing /var/folders/9w/_wt1czb57k9c5b0hgg6dfl2x7jbl85/T/tmp7uY1G8/lib/python/repoze.lru-0.6-py2.7-nspkg.pth
Successfully installed repoze.lru
Cleaning up...
MAC-C08182:targettest c08182$ PYTHONPATH=$(pwd)/lib
MAC-C08182:targettest c08182$ python
Python 2.7.7 |Anaconda 2.0.0 (x86_64)| (default, Jun  2 2014, 12:48:16) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import repoze.lru
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named repoze.lru
>>> 
MAC-C08182:targettest c08182$ 

However, all necessary files are in fact installed in lib/:

MAC-C08182:targettest c08182$ cd lib/
MAC-C08182:lib c08182$ ls
repoze              repoze.lru-0.6-py2.7.egg-info
repoze.lru-0.6-py2.7-nspkg.pth
MAC-C08182:lib c08182$ 

And of course the whole procedure works just fine for non-namespace packages.

Thanks to @mmerickel for setting me straight on the source of the issue.

[mmerickel]

The Python docs say only the site-packages folders relative to sys.prefix and sys.exec_prefix are searched for .pth files. I guess the question is how to make the -t option work?

[webmaven]

@mmerickel I am having difficulty locating the relevant section of the docs. Link, please?

[mmerickel]

https://docs.python.org/2/library/site.html

[webmaven]

A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above;

Darn, not a lot of ambiguity there.

Any ideas on how I can work around this limitation? App Engine inserts the lib subdirectory into sys.path (eg. https://github.com/GoogleCloudPlatform/appengine-python-flask-skeleton/blob/master/appengine_config.py#L6), but that obviously doesn't enable the .pth file.

[Ivoz]

You could manually move the namespace'd package that you want to the actual folder where it would be located if it was simply part of the overall package, in a build / deploy step.

Alternately, call addsitedir explicitly in your own sitecustomize.py file.

[webmaven]

@Ivoz site.addsitedir('lib') was exactly what I needed. Thanks!

[webmaven]

Unfortunately, site.addsitedir(path) appends the directory to the path, rather than inserting it at the beginning. This causes problems if one of the packages you're trying to use in the target directory is a newer version than one that exists earlier in the path.

I have a workaround which I am trying to simplify:

import os
import site
import sys

dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)

sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)

But that seems rather ugly to me. What would help a lot would be if a site.insertsitedir(index, path) was available (or something similar) as detailed in this issue that was brought to my attention by @gotgenes: http://bugs.python.org/issue7744

But in the absence of such a method, can the above code be improved or is there another, simpler workaround?