BLD: tslib dependencies not fully specified

[jreback]

    '_libs.tslib': {'pyxfile': '_libs/tslib',
                    'pxdfiles': ['_libs/src/util'],
                    'depends': tseries_depends,
                    'sources': np_datetime_sources},

is not complete, in fact this directly depends on

(pandas) bash-3.2$ ls pandas/_libs/tslibs/*.pxd
pandas/_libs/tslibs/conversion.pxd      pandas/_libs/tslibs/nattype.pxd         pandas/_libs/tslibs/timedeltas.pxd
pandas/_libs/tslibs/frequencies.pxd     pandas/_libs/tslibs/np_datetime.pxd     pandas/_libs/tslibs/timezones.pxd
(pandas) bash-3.2$ ls pandas/_libs/tslib*.pxd
pandas/_libs/tslib.pxd

not really sure of the exact rule, but I think anything you cimport from a .pxd must be listed as a dep.
Now rebuilding from clean just works, but rebuilding from a change in one part of the structure will only sometimes work.

[jreback]

@jbrockmendel need to fix this before any other changes.

[jbrockmendel]

but rebuilding from a change in one part of the structure will only sometimes work.

Do you have an example for how this fails? My understanding is that cython checks the timestamps on other cython files for determining if dependencies have changed.

[jreback]

try making a change on a .pxd (e.g moving something), them try to make, it will fail because it can't load the object.

[jreback]

nothing to do with time stamps, rather the build process doesn't know about deps. that it is depending, IOW tslib depends on tslibs/converstion.pxd, but this is not indicated anywhere (to rebuild tslib).

[gfyoung]

try making a change on a .pxd (e.g moving something), them try to make, it will fail because it can't load the object

I noticed this yesterday when I tried to build pandas from source in an existing environment. Some Cython files didn't get compiled again as needed (because dependencies were not specified), causing a statement like import pandas to fail.

[jbrockmendel]

nothing to do with time stamps, rather the build process doesn't know about deps

OK I'll try to grok this.

Cython clearly knows something about up-to-dateness:

$python setup.py build_ext --inplace
running build_ext
skipping 'pandas/io/sas/sas.c' Cython extension (up-to-date)
skipping 'pandas/_libs/properties.c' Cython extension (up-to-date)
skipping 'pandas/_libs/groupby.c' Cython extension (up-to-date)
[...]

[gfyoung]

@jbrockmendel : That's actually us 😄 . We compare timestamps on all files and dependencies to determine whether they need updating.

[jbrockmendel]

try making a change on a .pxd (e.g moving something), them try to make, it will fail because it can't load the object.

OK, I just took an existing branch, went to conversion.pxd and made a whitespace edit, then ran setup.py build_ext:

$python setup.py build_ext --inplace
[...]
skipping 'pandas/_libs/tslibs/frequencies.c' Cython extension (up-to-date)
skipping 'pandas/_libs/hashtable.c' Cython extension (up-to-date)
cythoning pandas/_libs/tslibs/conversion.pyx to pandas/_libs/tslibs/conversion.c
building 'pandas._libs.tslibs.conversion' extension
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/usr/local/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c pandas/_libs/tslibs/conversion.c -o build/temp.macosx-10.12-x86_64-2.7/pandas/_libs/tslibs/conversion.o -Wno-unused-function
[...]
skipping 'pandas/_libs/interval.c' Cython extension (up-to-date)
[...]
$python
>>> import pandas as pd
>>>

So far so good. So I guess you're referring to something less benign than a whitespace change? If I comment out convert_to_tsobject in conversion.pxd then I'll get an error at import-time. Ideally this would raise at build-time, which I guess is what you're getting at here.

The cython folks are at least aware of this; lack of discussion suggests it isn't a high priority.

There is a cython option that would allow us to do without pxd files entirely in some cases, but I'm a bit wary of it: cython/cython#1866

Pretty sure I've seen this on the cython mailing list, will try to track that down.

[jbrockmendel]

That's actually us 😄 . We compare timestamps on all files to determine whether they need updating.

Hah. Good to know.

[gfyoung]

If I comment out convert_to_tsobject in conversion.pxd then I'll get an error at import-time. Ideally this would raise at build-time, which I guess is what you're getting at here.

Yes, that's exactly what happened to me yesterday. I wouldn't worry too much about anything Cython-issue-related, as that's beyond our control from a pandas perspective. The best thing would be to try to correct the dependencies on our end so that we don't get import errors.