Incompatibility with newest Pandas `2.0.0`

[kasra-keshavarz]

Hello Andrew,

Thanks for the wonderful metsim package. Just to let you know that metsim==2.4.1 is incompatible with Pandas=2.0.0. This is no surprise as Python's libraries are always evolving and backward compatibility is not guaranteed. Thank you!

[arbennett]

Thanks @kasra-keshavarz - do you know exactly where incompatibilities come in? Hopefully this will be easy enough to fix.

[DaveCasson]

Hi Andrew and Karsa, the incompatibilities likely come in at 0.25 (DEPR: Add Deprecated warning for timedelta with passed units M and Y) based on this pull request: pandas-dev/pandas#23264

[DaveCasson]

I had thought it had to do with specification of output frequency, however I think the incompatibility is more generic than that. I can't quite pinpoint it or would submit a PR.

Units 'M', 'Y' and 'y' do not represent unambiguous timedelta values and are not supported.
File "/home/dcasson/GitRepos/gmet_to_summa_snakemake/workflow/rules/run_metsim.smk", line 25, in __rule_generate_metsim_output
File "/home/dcasson/GitRepos/gmet_to_summa_snakemake/workflow/rules/../scripts/metsim_utils.py", line 44, in create_metsim_config
File "/home/dcasson/summa_snakemake/lib/python3.10/site-packages/metsim/metsim.py", line 222, in init
File "/home/dcasson/summa_snakemake/lib/python3.10/site-packages/metsim/metsim.py", line 530, in _get_output_times
File "/home/dcasson/summa_snakemake/lib/python3.10/site-packages/metsim/metsim.py", line 295, in met_data
File "/home/dcasson/summa_snakemake/lib/python3.10/site-packages/metsim/io.py", line 182, in read_met_data
File "/home/dcasson/summa_snakemake/lib/python3.10/site-packages/metsim/io.py", line 207, in process_nc
File "/home/dcasson/summa_snakemake/lib/python3.10/site-packages/metsim/io.py", line 290, in read_netcdf
File "pandas/_libs/tslibs/timedeltas.pyx", line 1766, in pandas._libs.tslibs.timedeltas.Timedelta.new
File "pandas/_libs/tslibs/timedeltas.pyx", line 553, in pandas._libs.tslibs.timedeltas.parse_timedelta_string
File "pandas/_libs/tslibs/timedeltas.pyx", line 694, in pandas._libs.tslibs.timedeltas.timedelta_from_spec

[arbennett]

Thanks for looking into this @DaveCasson - it seems that the first incompatibility is at this line: https://github.com/UW-Hydro/MetSim/blob/master/metsim/io.py#L290

There might be others that crop up, but this one shouldn't be terrible to fix. Basically what's going on is MetSim tries to first normalize all input timestamps to just raw days (AKA with no hour, minute, or second portions) and to do that I made it round by subtracting off ~12 hours and then doing the conversion. Basically the same idea as adding/subtracting 1/2 before converting a float to an int to guarantee rounding in the direction that you want.

[DaveCasson]

It appears only two fixes are needed:

1. Can replace where you indicated above: ds['time'] = (ds.indexes['time']
   - pd.Timedelta(hours=11, minutes=59, seconds=59)).round('D')

2. Substitute the closed variable for inclusive:
   

   MetSim/metsim/datetime.py

   Line 51 in 04a6d91

   if calendar in ['standard', 'gregorian', 'propoleptic_gregorian']:

if calendar in ['standard', 'gregorian', 'propoleptic_gregorian']:
return pd.date_range(start=start, end=end, periods=periods,
freq=freq, tz=tz, normalize=normalize, name=name,
inclusive=closed, **kwargs)

I expect this will match the expected behaviour. I will test and submit a PR.
There is now a bit of a variable mis-match (closed variable goes to inclusive), so open to other suggestion.

[DaveCasson]

The switch from closed to inclusive is documented in the pandas 1.4.0 release notes here: https://pandas.pydata.org/docs/whatsnew/v1.4.0.html

[arbennett]

Awesome, thanks for tracking this down!

[dwoodson-usbr]

Is there any update on this issue or work around?

Ran into this issue when running the MetSim tutorial.

[arbennett]

@dwoodson-usbr - I just need to do some testing and merge in @DaveCasson's PR. I will try to get to it this weekend!

[arbennett]

Fixed in #260