Running setup.py build in Windows in Admin prompt results in Access denied errors

[ellinorcrux]

Hi,

I tried using this tool in an elevated command prompt, but I got some strange "Access is denied" and "undefined reference to f" errors. See image:

If I go to the location of my "root" file, I'm able to build using the go build command in the image above.

I'm on Python 2.7.11, 32 bit on Windows 8

I built Go from source, and I'm on git hash 8f70e1f8a91db6c8a1bcc292766af95e52e8512b

I have gcc installed in MINGW32

This is what my setup.py looks like:

There are a ton of other arguments I didn't include. I have a template for setup.py that I generate for each package I make.

I'm wondering what the best approach to trying to debug this issue is. I've tried looking at the files its complaining about, but I can't find all of them. Any tips?

Thanks!

[asottile]

Interesting, I haven't actually tried using this on windows yet but I'll give it a poke when I get home -- I'd start with trying to build the examples: https://github.com/asottile/setuptools-golang-examples and see what you run into

[ellinorcrux]

I'll give that a shot. Thanks! Note that using "c-shared" option on go build is not available in Go 1.8 on Windows. Its slated for release in Go 1.9. That's why I had to build from source.

[ellinorcrux]

Ah sorry. Not Go 1.9, should be Go 1.10: golang/go#11058

[ellinorcrux]

I cloned this repository using git, then I commented back in the "setuptools-golang" dependency back in for the "imports_gh" example in the testing folder. I ended up with a very similar result:

[asottile]

Ah, so the first error is actually the meaningful one, seems we probably need to extend this to include whatever link flag that works on windows (similar to #8)

The second error is due to shutil.rmtree failing to clean readonly files on windows -- this is only hiding the previous error -- here's how I've fixed this in the past

[ellinorcrux]

Ah OK! Thank you! Are you planning to put the patch in for the link flag or is that something you would like me to help with?

[asottile]

If you'd like to take a stab at it feel free -- I'm not sure when I'll get to that (something something copious amounts of freetime 😆) -- the shutil bit is pretty easy if you want to lump them together (the shutil patch actually is probably necessary so you can debug at all)

[ellinorcrux]

I put in the shutil fix, and also a fix for converting unicode strings in the env to normal strings: ellinorcrux@728e5f2

The unicode string thing might only be an issue in py27. I'm not sure what effect it would have in py3.

The thing I'm confused about though is that it seems that the '-Wl,--unresolved-symbols=ignore-all' arguments are ignored completely. I am actually using gcc despite being on Windows.

After my fix I now get the following output:

[asottile]

heh yeah, I've hit the environ problem before as well -- they need to be "native" strings on windows (str (bytes) in python2, str (text) in python3).

So python for windows is usually compiled with msvc -- the error messages also look like msvc errors, I'm thinking we'll need to find the "allow undefined symbols" option for msvc.

[ellinorcrux]

I'm pretty certain its using gcc. I tried using "/FORCE" from here: https://msdn.microsoft.com/en-us/library/70abkas3.aspx

But I end up with this:

I'm not sure that MSVC is even supported for "--buildmode=c-shared" option yet, so I'm not sure there's any value in adding the equivalent flags in for it.

[asottile]

hmmmm... I'll have to poke at this -- perhaps I'll be able to look into this more over the holiday weekend -- do you have some directions on how I can get started with building go on windows (at least so I can reproduce what you're looking it 😆 )

[ellinorcrux]

Here's the guide to building from source: https://golang.org/doc/install/source
Take your time :) I'll keep looking into it meanwhile.
Thanks!

[ellinorcrux]

Note that some of the unit tests for the built-in libraries might fail when you install from source. This wasn't a problem for me because I wasn't planning to use those libraries for this project. The installation worked fine otherwise :)

[ellinorcrux]

I played around with this a bit more. It appears the main issue is now the "undefined reference to __imp__Py..." errors. I noticed that one of the things that was getting included was "C:\Python27\PC". This folder actually doesn't exist in my installation. I've noticed it doesn't exist in a fresh installation on another machine either, so this doesn't seem quite right. I think the missing references are meant to be in a file called libpython27.a in the C:\Python27\libs folder, but adding "-IC:\Python27\libs" to CGO_CFLAGS didn't seem to help. Not sure what to try next...

[asottile]

So the way python linking works is kind of odd. When it builds the .so files, it doesn't actually link them against libpython -- they get hooked together at runtime when imported. There's a flag that works well under linux-gcc and clang that ignores the undefined references, I think we just need ot figure out what that correct flag is for windows-gcc.

[ellinorcrux]

I tried following the instructions in this article since it looks very promising, but it didn't help either:
http://techtonik.rainforce.org/2008/01/compiling-python-extension-with-gcc.html

I ended up with "c:/mingw/bin/../lib/gcc/mingw32/5.3.0/../../../../mingw32/bin/ld.exe: cannot find -lpython27"

I added the -L and -l flags to CGO_LDFLAGS. I'm not sure exactly what the gcc call will look like, and if I faithfully reproduced it.

[ellinorcrux]

I understand what you're saying. Unfortunately I think MinGW doesn't support any such option for the linker: http://mingw.5.n7.nabble.com/ignoring-unresolved-symbols-in-shared-libraries-td14076.html because of some underlying Windows thing that I don't understand.

However in the end I did manage to get this to work by adding "-LC:\Python27\libs -lpython27 " to CGO_LDFLAGS. I mentioned this earlier as not working; that's because I had a typo 🤦‍♀️

Thank you so much for your help and this awesome utility! 👍

I will submit some suggested changes in my fork of the repo soon once I've cleaned my stuff up a little.

[asottile]

Cool! This is actually how this utility used to work before this change -- maybe we can conditionally enable that approach only on windows?

[ellinorcrux]

Ah, interesting. I've uploaded some code that worked for me, but I wasn't even thinking about pypy when I wrote it. Here's what I've got:

master...ellinorcrux:master

Feel free to use parts of it if you like, or rewrite it to better accommodate other platforms.

[asottile]

Sounds good! We can probably hook up appveyor as well to keep this from breaking in the future -- cool to see that it wasn't far off from having windows just work! Thanks again for the investigation on this 👍

[asottile]

I tried applying (iteratively) the changes necessary to get the tests to pass, my progress is here

Just wanted to confirm a few things about your setup before continuing as I'm getting into territory I don't fully understand :)

1. mingw64 (gcc): 64 bit or 32bit?
2. go: 64bit or 32bit?
3. cpython: compiled with mingw? or from python.org installers
4. cpython: 64bit or 32bit?

For me I have these setups:

1. mingw: 64bit
2. go: 64bit
3. from python.org installers (msvc)
4. 64bit

I'm getting to the following:

python2.7.14

I'm currently getting to the following build error:

    $ CGO_CFLAGS='-Ic:\python27\include -Ic:\temp\pytest-16\venv0\venv\PC' CGO_LDFLAGS='-Lc:\python27\libs -lpython27' G
OPATH='c:\temp\tmp8eqhfa' go build -buildmode=c-shared -o 'c:\temp\pip-eimzs6-build\build\lib.win-amd64-2.7\sum.pyd'
    # github.com/asottile/fake
    C:\Temp\go-build209371238\b001\_x003.o:sum.c:(.text+0x69): undefined reference to `__imp_Py_InitModule4'
    collect2.exe: error: ld returned 1 exit status

which seems to be fixable via this

Adding that define, I can successfully build, but something seems not right with the shared objects created:

        if returncode is not None:
            if proc.returncode != returncode:
                raise AssertionError(
                    '{!r} returned {} (expected {})\nout:\n{}\nerr:\n{}\n'.format(
>                       cmd, proc.returncode, returncode, out, err,
                    )
                )
E               AssertionError: (u'c:\\temp\\pytest-19\\venv0\\venv\\Scripts\\python', u'-c', u'import sum; print(sum.su
m(1, 2))') returned -1073741819 (expected 0)
E               out:
E
E               err:
E               None

python3.6.3

For whatever reason (?) I don't need to jump through the -DMS_WIN64 hoop -- but I end up at the same failure state as python2.7

[asottile]

Another useful datapoint might be which gcc (it's possible I installed the wrong one, there were a few options on threading model, etc.)

oh but I can see from your screenshot that'll just be c:/mingw/bin/gcc.exe probably -- not sure how to see the compilation options and I'm not at my windows machine again lol

[ellinorcrux]

To answer your questions, I'm using:

1. MINGW 32 bit
2. Go: Also 32 bit since I compiled it using MINGW32
3. CPython I installed from their MSVC Windows binaries. Also 32bit. I have 2.7.11

I believe there are some issues with Python 64 bit and MINGW64 which might explain the issues you're seeing.

Firstly, the "__imp_Py_InitModule4" missing reference should be in a file called "libpython27.a" or "libpython35.a" in the PythonXY/libs folder. The 64 bit version of Python does not provide this library, apparently because the MINGW64 runtime interferes with the MSVC runtime. The Python on Windows binaries are compiled using MSVC.

I could not find this well documented anywhere, but here's where I got this idea: https://stackoverflow.com/questions/7492416/building-64bit-libpython27-a-using-cygwin-dlltool (the second answer) also mentioned here: https://groups.google.com/forum/#!topic/cython-users/UX70kTr-6vk

This thread also gives more insight into the whole conflict between Python and MINGW64:
https://bugs.python.org/issue4709

Pretty frustrating. From observing the discussion on the Golang side, it looks like they're not planning on supporting MSVC anytime soon, while the Python guys don't really have a supported solution either, despite the scientific Python community having a need for it. This appears to be the current workaround: http://ascend4.org/Setting_up_a_MinGW-w64_build_environment#Setup_Python_for_compilation_of_extensions

[asottile]

ah ok, only supporting 32bit is fine for me for now -- I'll try again with 32bit

In the meantime, can you try checking out my branch and running the tests and see if they pass?

[ellinorcrux]

As for which gcc, this is what's visible from inside my MINGW32 package manager:

---

[ellinorcrux]

I'll definitely try running the tests. Thanks!

[ellinorcrux]

tox_stdout.txt
I ran tox and this is what I got. It looks like some of the tests are failing for trivial reasons, like unexpected windows line endings.

[asottile]

Nice, the other error is due to the ca file being not findable -- need to re-remember which environment variable enables that and passenv whitelist it -- this looks good and doable! Thanks again :)

[ellinorcrux]

Hey did you manage to get the unit tests working? Let me know if you want me to run them again on my machine. I know its kind of a hassle to set the environment up. Thanks for looking into this :)

[asottile]

oh shoot, lost this in my long list of TODOs, I'm leaving this email unread and I'll take a look this weekend.

Thanks for the patience and for helping double check the work :)

[asottile]

Hoooray!!!!

Anthony@AnthonysDesktop MINGW64 ~/Desktop/git/setuptools-golang (windows)
$ py.test tests --maxfail 1
============================= test session starts =============================
platform win32 -- Python 2.7.14, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: C:\Users\Anthony\Desktop\git\setuptools-golang, inifile:
collected 9 items

tests\setuptools_golang_test.py .........                                [100%]

========================= 9 passed in 105.07 seconds ==========================

now to work out the tox environment variable and then try and hook it up with CI!

[asottile]

Try it out!

Just released v1.2.0

[ellinorcrux]

Awesome thanks again!