PERF: perf regressions vs 0.14.0

[jreback]

We have many + perf fixes in 0.14.1 yeh!

but need to look at these (resamples could be spurious)

dataframe_resample_mean_string               |   4.0483 |   2.4170 |   1.6749 |
dataframe_resample_max_numpy                 |   3.3394 |   1.7230 |   1.9381 |
dataframe_resample_max_string                |   3.3154 |   1.7047 |   1.9448 |
dataframe_resample_min_numpy                 |   3.3020 |   1.6843 |   1.9604 |
dataframe_resample_min_string                |   3.2830 |   1.6587 |   1.9793 |
timeseries_timestamp_downsample_mean         |  11.4803 |   4.0847 |   2.8106 |
timeseries_period_downsample_mean            |  29.7631 |  10.3510 |   2.8754 |
datetimeindex_normalize                      |  77.5570 |   3.0633 |  25.3183 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [272dae5] : BUG: doc example in groupby.rst (GH7559 / GH7628)
Base   [da0f7ae] : RLS: 0.14.0 final

[jreback]

cc @sinhrks

[cpcloud]

There are some pretty hefty hits for DatetimeIndex-ed repring, so much so that I have to KeyboardInterrupt the repring. I'll see if I can't track this down. It looks like there's some unnecessary vectorization of timezone manipulation.

[jreback]

can you post your example?

[cpcloud]

In [1]: period = 1.0 / 2048 * 1e9

In [2]: freq = pd.datetools.Nano(period)

In [3]: df = DataFrame({'a': np.random.randn(6e6)}, index=pd.date_range('now', periods=6e6, freq=freq, tz='EST'))

In [4]: df.a  # takes a very long time

[cpcloud]

hm maybe repr-ing is not the issue repr(df.a) finishes in a timely manner.

[jreback]

what does ipython actually do after executing, maybe it caches it or something?

[jreback]

the answer is in the format, while I ctrl-c....its in trim_zeros (its prob trimming on the entire frame and not on the displayed portion)

[cpcloud]

hm when i control-c it's in a np.vectorize-d function. vectorize is not really that much faster than just map + lambda which i think is the issue

[cpcloud]

okay maybe not ... just cython'd that and it didn't improve the situation

[cpcloud]

that is where the bottleneck is though

[cpcloud]

In [4]: df.a
> /home/phillip/Documents/code/py/pandas/pandas/tseries/frequencies.py(674)__init__()
    673             import ipdb; ipdb.set_trace()
--> 674             self.values = tslib.tz_convert_safe_dst(self.values, 'UTC',
    675                                                     index.tz)

ipdb> n
> /home/phillip/Documents/code/py/pandas/pandas/tseries/frequencies.py(675)__init__()
    674             self.values = tslib.tz_convert_safe_dst(self.values, 'UTC',
--> 675                                                     index.tz)
    676             #self.values = np.vectorize(f)(self.values)

ipdb> n  # bueller ..... bueller ..... anybody

[jreback]

yes...that is the same problem as in the vbench above

[cpcloud]

ok

[cpcloud]

what does bcdc6d9 actually fix? i don't see any docs on it

[jreback]

If you put the commented code back this fixes the entire problem: https://github.com/pydata/pandas/blob/master/pandas/tseries/frequencies.py#L675

[cpcloud]

yep

[jreback]

can you figure out where that was changed?

[cpcloud]

the commit i just put up is where it was changed

[cpcloud]

git blame -L 671,677 pandas/tseries/frequencies.py

[jreback]

cc @sinhrks bcdc6d9

[jreback]

partially closd by #7652

@cpcloud this also fixes the display issue from above (though no idea how to actully test it)

[jreback]

cc @sinhrks. Seems min/max are doing a vair amount of computation on an Index w/o missing values. This is why the resamples are now slower. Can you take a look?

[sinhrks]

Yes, will check.

[sinhrks]

It looks offset ops get also little slower. I'll take a look.

-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_ctor_dtindex_Millix2                   |   2.3377 |   1.8246 |   1.2812 |
frame_ctor_dtindex_YearEndx2                 |   2.7140 |   2.1173 |   1.2818 |
frame_ctor_dtindex_Microx2                   |   2.3347 |   1.8183 |   1.2840 |
frame_ctor_dtindex_Hourx2                    |   2.3340 |   1.8170 |   1.2846 |
frame_ctor_dtindex_Microx1                   |   2.3340 |   1.8164 |   1.2850 |
frame_ctor_dtindex_Dayx1                     |   2.3523 |   1.8146 |   1.2963 |
...
timeseries_day_incr                          |   0.0190 |   0.0054 |   3.5147 |
timeseries_day_apply                         |   0.0173 |   0.0033 |   5.1905 |