Unexpected decoded time in xarray >= 0.10.1

[JanisGailis]

Problem description

Given the original time dimension:

ds = xr.open_mfdataset("C:\\Users\\janis\\.cate\\data_stores\\local\\local.SST_should_fail\\*.nc", decode_cf=False)

<xarray.DataArray 'time' (time: 32)>
array([788961600, 789048000, 789134400, 789220800, 789307200, 789393600,
       789480000, 789566400, 789652800, 789739200, 789825600, 789912000,
       789998400, 790084800, 790171200, 790257600, 790344000, 790430400,
       790516800, 790603200, 790689600, 790776000, 790862400, 790948800,
       791035200, 791121600, 791208000, 791294400, 791380800, 791467200,
       791553600, 791640000], dtype=int64)
Coordinates:
  * time     (time) int64 788961600 789048000 789134400 789220800 789307200 ...
Attributes:
    standard_name:  time
    axis:           T
    comment:        
    bounds:         time_bnds
    long_name:      reference time of sst file
    _ChunkSizes:    1
    units:          seconds since 1981-01-01
    calendar:       gregorian

Produces this decoded time dimension with xarray >= 0.10.1:

ds = xr.open_mfdataset("C:\\Users\\janis\\.cate\\data_stores\\local\\local.SST_should_fail\\*.nc", decode_cf=True)

<xarray.DataArray 'time' (time: 32)>
array(['1981-01-01T00:00:00.627867648', '1980-12-31T23:59:58.770774016',
       '1981-01-01T00:00:01.208647680', '1980-12-31T23:59:59.351554048',
       '1981-01-01T00:00:01.789427712', '1980-12-31T23:59:59.932334080',
       '1980-12-31T23:59:58.075240448', '1981-01-01T00:00:00.513114112',
       '1980-12-31T23:59:58.656020480', '1981-01-01T00:00:01.093894144',
       '1980-12-31T23:59:59.236800512', '1981-01-01T00:00:01.674674176',
       '1980-12-31T23:59:59.817580544', '1980-12-31T23:59:57.960486912',
       '1981-01-01T00:00:00.398360576', '1980-12-31T23:59:58.541266944',
       '1981-01-01T00:00:00.979140608', '1980-12-31T23:59:59.122046976',
       '1981-01-01T00:00:01.559920640', '1980-12-31T23:59:59.702827008',
       '1981-01-01T00:00:02.140700672', '1981-01-01T00:00:00.283607040',
       '1980-12-31T23:59:58.426513408', '1981-01-01T00:00:00.864387072',
       '1980-12-31T23:59:59.007293440', '1981-01-01T00:00:01.445167104',
       '1980-12-31T23:59:59.588073472', '1981-01-01T00:00:02.025947136',
       '1981-01-01T00:00:00.168853504', '1980-12-31T23:59:58.311759872',
       '1981-01-01T00:00:00.749633536', '1980-12-31T23:59:58.892539904'],
      dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 1981-01-01T00:00:00.627867648 ...
Attributes:
    standard_name:  time
    axis:           T
    comment:        
    bounds:         time_bnds
    long_name:      reference time of sst file
    _ChunkSizes:    1

Expected Output

With xarray == 0.10.0 the output is as expected:

ds = xr.open_mfdataset("C:\\Users\\janis\\.cate\\data_stores\\local\\local.SST_should_fail\\*.nc",
                       decode_cf=True)

<xarray.DataArray 'time' (time: 32)>
array(['2006-01-01T12:00:00.000000000', '2006-01-02T12:00:00.000000000',
       '2006-01-03T12:00:00.000000000', '2006-01-04T12:00:00.000000000',
       '2006-01-05T12:00:00.000000000', '2006-01-06T12:00:00.000000000',
       '2006-01-07T12:00:00.000000000', '2006-01-08T12:00:00.000000000',
       '2006-01-09T12:00:00.000000000', '2006-01-10T12:00:00.000000000',
       '2006-01-11T12:00:00.000000000', '2006-01-12T12:00:00.000000000',
       '2006-01-13T12:00:00.000000000', '2006-01-14T12:00:00.000000000',
       '2006-01-15T12:00:00.000000000', '2006-01-16T12:00:00.000000000',
       '2006-01-17T12:00:00.000000000', '2006-01-18T12:00:00.000000000',
       '2006-01-19T12:00:00.000000000', '2006-01-20T12:00:00.000000000',
       '2006-01-21T12:00:00.000000000', '2006-01-22T12:00:00.000000000',
       '2006-01-23T12:00:00.000000000', '2006-01-24T12:00:00.000000000',
       '2006-01-25T12:00:00.000000000', '2006-01-26T12:00:00.000000000',
       '2006-01-27T12:00:00.000000000', '2006-01-28T12:00:00.000000000',
       '2006-01-29T12:00:00.000000000', '2006-01-30T12:00:00.000000000',
       '2006-01-31T12:00:00.000000000', '2006-02-01T12:00:00.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 2006-01-01T12:00:00 2006-01-02T12:00:00 ...
Attributes:
    standard_name:  time
    axis:           T
    comment:        
    bounds:         time_bnds
    long_name:      reference time of sst file
    _ChunkSizes:    1

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.4.final.0 python-bits: 32 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 69 Stepping 1, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None

xarray: 0.10.1
pandas: 0.22.0
numpy: 1.14.2
scipy: 0.19.1
netCDF4: 1.3.1
h5netcdf: 0.5.0
h5py: 2.7.1
Nio: None
zarr: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.17.1
distributed: 1.21.3
matplotlib: 2.2.2
cartopy: 0.16.0
seaborn: None
setuptools: 39.0.1
pip: 9.0.2
conda: None
pytest: 3.1.3
IPython: 6.2.1
sphinx: None

[JanisGailis]

#1859 seems to be related.

[JanisGailis]

Actual data can be retrieved from here:
ftp://anon-ftp.ceda.ac.uk/neodc/esacci/sst/data/lt/Analysis/L4/v01.1

[fujiisoup]

I think it is related to #1932.
Can you try xarray=0.10.2?

[JanisGailis]

Thanks for looking into this!

I did try 0.10.2, same result as 0.10.1.

[fujiisoup]

Could you try investigating minimal working example?
I tried with two of your examples, but the issue is not reproduced

In [9]: ds = xr.open_mfdataset(['Desktop/19910901120000-ESACCI-L4_GHRSST-SSTdepth-OSTIA-GLOB_LT-v02.0-fv01.1.nc', 'Desktop/19910902120000-ESACCI-L4_GHRSST-SSTdepth-OSTIA-GLOB_LT-v02.0-fv01.1.nc'], decode_cf=True)
In [11]: ds['time']
Out[11]: 
<xarray.DataArray 'time' (time: 2)>
array(['1981-01-01T00:00:00.564166656', '1980-12-31T23:59:58.707073024'],
      dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 1981-01-01T00:00:00.564166656 ...
Attributes:
    standard_name:  time
    axis:           T
    bounds:         time_bnds
    comment:        
    long_name:      reference time of sst file

[JanisGailis]

You have just reproduced the issue.

The correct datetime values are in the filenames. So, you open two files, one from 1991-09-01T12:00:00.00 and the other from 1991-09-02T12:00:00.00, but the decoded time dimension becomes:

array(['1981-01-01T00:00:00.564166656', '1980-12-31T23:59:58.707073024'], dtype='datetime64[ns]')

Which is exactly the problem I'm facing.

[fujiisoup]

Oh, thanks. I misunderstood the problem.
Related to #1803? but I did not follow that well.
Can anyone look into this issue?
The minimal working example would be
1 download from one file from here
2 Load and see xr.open_dataset(path/to/file)['time']
3 compare with netCDF4.Dataset(path/to/file)['time']

[fmaussion]

The problem comes from the fact that the times in the files are stored in int32. This mwe reproduces the error:

import xarray as xr
import numpy as np
ds = xr.Dataset()
ds['time'] = xr.DataArray([np.int32(788961600)], dims=['time'])
ds['time'].attrs['units'] = 'seconds since 1981-01-01'
xr.decode_cf(ds)['time']

This is due to #1414, I'm looking into it