macOS wheel cannot be installed without admin permissions

[cbrnr]

I just tried to pip install pyreadstat, but installation failed because:

ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/zlib1.dll'

I know that I could use the --user flag, but I don't think this should be necessary. Why is the package trying to copy zlib1.dll to /usr/local (which doesn't have group write permissions for normal users)? First, this is probably a Windows lib and not needed on macOS. Second, libs should go into /usr/local/lib, which by default has group write permission for regular users.

[cbrnr]

Edit: Despite of this error, the package is still installed correctly. So it really looks like zlib1.dll is not required on macOS. I don't have a lot of experience with setup.py, but it looks like this can be changed there (https://github.com/Roche/pyreadstat/blob/master/setup.py#L70-L95).

[ofajardo]

hi

Thanks for the report. I fail to reproduce what you are seeing. Please provide more details about your setup: what python version are you using? where is it installed? are you using the system python or anaconda/user installed?

Assuming you are using the system python, I fail to reproduce the issue, because I get a permission denied to python site-packages in /Library/Python, which for me makes full sense, you certainly need sudo in order to write there. But that means the package should fail to be installed, which you say is not your case. So it means to me you are not using the system python (which anyway is a bad practice btw), so what are you using? Maybe you are not using the system python, but whatever you are using you still installed as root?

It is indeed possible to exclude the dlls for mac. However I need them on linux, because I build the source distribution package on linux, but then this is used on windows when building the conda packages, so I cannot get rid of them even if not necessary on linux. For mac I was maintained the same configuration as for other systems to make it less complicated, copying the dlls should not disturb as it should be written to the package location and not /usr/local/zlib1.dll (that happens to me in all systems). I would like to understand that better why are you getting that message, but right now I can't reproduce it.

[cbrnr]

This error occurs when I use Python (I tried with the latest version 3.7) installed via Homebrew (which is located in /usr/local). With Anaconda, everything seems to work as expected. However, I noticed that zlib1.dll and libiconv-2.dll are copied into the Anaconda root, which is user-writeable (but I guess this is not the location you intended to install these files).

[ofajardo]

OK, that makes a bit more sense.

Regarding Homebrew my guess is you may have a bit of chaos regarding permissions, with some folders being writtable by your user and some not, only by root (/usr/local). Look this one.

https://stackoverflow.com/questions/52949531/could-not-install-packages-due-to-an-environmenterror-errno-13

Regarding Anaconda, yes that's expected, I cannot decide where exactly is python going to copy the dlls, so probably is fine.

[cbrnr]

My permissions in /usr/local are fine, the root itself should not be group writeable. I don't think that this is a good location to put the files, regardless of which Python distribution is used.

If possible, try to move these files into the package folder, e.g. $PYTHON_ROOT/lib/python3.7/site-packages/pyreadstat/ - this should also be the preferred location on Linux systems. I think this SO answer shows how to include additional files in a package.

[ofajardo]

yeah, I already have data_files, even on windows it is copying the dlls to the top level. Strange, I remember in previous versions it was not like that. package_data for some reason is not effective at all.
Anyway, I was already thinking on the another suggestion, so I changed the manifest and removed the data for mac and linux. On linux I can see now that the dlls are not copied. I assume should be the same for mac.
Would you be able to test wheter this modification solves your issue? It is on the dev branch. You would need to install directly from the repo (you would need a compiler and cython potentially).

[cbrnr]

I installed with pip install git+https://github.com/Roche/pyreadstat@dev and I received no error, so I guess this change worked.

[ofajardo]

Thanks for checking! Yes I think it should be good now.

[cbrnr]

Looking forward to the next release (note that the dev branch has version 0.2.4, which should probably be changed for the release). Also let me know if I can help with integrating your reader into Pandas.

[ofajardo]

yeah, I will bump the version and make a release soon. Just giving a bit of time for the other two open issues to see if there is a quick fix to include in the next release.
Regarding pandas, yes any help is appreciated. As you suggested the interesting thing here will be how to add the metadata to the pandas data frame, I guess subclassing it? The other thing I did not understood is who is going to actually do it, us (on the pyreadstat side) or them (the pandas guys).

[cbrnr]

I would simply try to return a normal dataframe which contains as much information as possible. That is, I would use categorical columns if applicable. I think the plan is that we make a PR and they help us get it into the right shape. I am not super familiar with pandas development, but I could try to come up with a basic PR which can then be further refined. I'm just not sure when I'll have the time, so if you have time to work on this earlier let me know and feel free to propose the initial PR.

[ofajardo]

there is extra information that cannot be accommodated in a normal data frame. For instance in SPSS you have (short) name for the columns, but in addition you can have a (long) label. As far as I remember in pandas you can have only column names, so you can get one or the other. Then you have missing values that can have all sort of labels as well. Maybe one way would be to have flags like get me column names, or column labels, etc, but this could generate lots of flags; so maybe only the most important could be included. But that means the pandas function will be more limited that pyreadstat.

[cbrnr]

I think in a first step the function should return the most sensible values, i.e. drop the labels. Later on, we can discuss if and how this additional meta information could be handled. I'm working on a simple PR right now, so we can continue our discussion there (will submit soon).

[ofajardo]

OK, sounds good. In that case just read with pyreadstat, return the dataframe and discard the metadata object. That should be enough.

[cbrnr]

This is the PR: pandas-dev/pandas#26537