'infer_freq' does not work with DST transition

[broessli]

Consider the following index crossing a DST transition:

In [1]: import pandas as pd;pd.__version__
Out[1]: '0.15.0'
In [2]: index = pd.date_range(pd.Timestamp("2014-10-25 03:00", tz="Europe/Paris"), periods=10, freq="3H")

In [3]: index
Out[3]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2014-10-25 03:00:00+02:00, ..., 2014-10-26 05:00:00+01:00]
Length: 10, Freq: 3H, Timezone: Europe/Paris

Pandas cannot infer the index frequency:

In [4]: pd.infer_freq(index) is None
Out[4]: True

If we convert the index to UTC, the frequency can be inferred:

In [5]: pd.infer_freq(index.tz_convert("UTC"))
Out[5]: '3H'

If the index does not cross a DST boundary, the frequency can be inferred as well:

In [6]: index = pd.date_range(pd.Timestamp("2014-10-25 03:00", tz="Europe/Paris"), periods=6, freq="3H")

In [7]: index
Out[7]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2014-10-25 03:00:00+02:00, ..., 2014-10-25 18:00:00+02:00]
Length: 6, Freq: 3H, Timezone: Europe/Paris

In [8]: pd.infer_freq(index)
Out[8]: '3H'

[jreback]

cc @rockg
cc @ischwabacher

I am not sure this is a bug. The frequency inferer is looking for a constant difference between each of the times (essentially diffing them). So the freq by-definition is different around DST transitions. So None is correct (it is not a regular frequency). Though that may be incorrect (e.g. maybe it should be inferring with UTC?)

[rockg]

I think this is a bug. In terms of hours, there is a constant difference. The code is converting the underlying UTC values to the local timezone so that week/month frequencies are calculated in local time (for example, 2010-12-25 00:00 is a Saturday in Paris but a Friday in UTC) but when that is done across timezones that shift becomes broken. The issue is in tseries.frequencies._FrequencyInferer.

    def __init__(self, index, warn=True):
        self.index = index
        self.values = np.asarray(index).view('i8')

        if hasattr(index,'tz'):
            if index.tz is not None:
                self.values = tslib.tz_convert(self.values, 'UTC', index.tz)

Using the above example, we have:

index.asi8
[1414198800000000000 1414209600000000000 1414220400000000000
 1414231200000000000 1414242000000000000 1414252800000000000
 1414263600000000000 1414274400000000000 1414285200000000000
 1414296000000000000]
np.diff(index.asi8)
[10800000000000 10800000000000 10800000000000 10800000000000 10800000000000
 10800000000000 10800000000000 10800000000000 10800000000000]
self.values
[1414206000000000000 1414216800000000000 1414227600000000000
 1414238400000000000 1414249200000000000 1414260000000000000
 1414270800000000000 1414281600000000000 1414288800000000000
 1414299600000000000]
np.diff(self.values)
[10800000000000 10800000000000 10800000000000 10800000000000 10800000000000
 10800000000000 10800000000000  7200000000000 10800000000000]

Hopefully this is enough for somebody to go off on. It's not immediately obvious what the fix is (besides treating the sub-daily and daily+ offsets differently--not converting in the former and converting in the latter).

[ischwabacher]

I agree that infer_freq should be able to recover the frequency given to date_range since the elapsed time between successive times in the index is constant.

[MartinLinden]

I would also consider this a bug as the user is looking for the regular elapsed time between two timestamps.
Could we reopen this issue?

[jreback]

pls open a new issue with an example / this is 5 years old