Implement cftime vectorization as discussed in PR #8322 #8324

antscloud · 2023-10-17T17:01:25Z

As discussed in #8322, here is the test for implementing the vectorization

Only this test seems to fail in test_coding_times.py :

xarray/xarray/tests/test_coding_times.py

Lines 1061 to 1071 in f895dc1

    
           @requires_cftime 
        
           @pytest.mark.parametrize("freq", ["U", "L", "S", "T", "H", "D"]) 
        
           def test_encode_decode_roundtrip_cftime(freq) -> None: 
        
               initial_time = cftime_range("0001", periods=1) 
        
               times = initial_time.append( 
        
                   cftime_range("0001", periods=2, freq=freq) + timedelta(days=291000 * 365) 
        
               ) 
        
               variable = Variable(["time"], times) 
        
               encoded = conventions.encode_cf_variable(variable) 
        
               decoded = conventions.decode_cf_variable("time", encoded, use_cftime=True) 
        
               assert_equal(variable, decoded)

I don't really understand why though if you have an idea

kmuehlbauer

@antscloud Nice catch! I have two suggestion to get the test suite running.

xarray/coding/times.py

headtr1ck · 2024-07-11T17:05:41Z

@antscloud are you willing to continue this? It would be a good addition.

antscloud · 2024-07-14T15:29:19Z

Hi, sorry i completely forgot about this issue, i've implemented the suggestions and all the tests appear to pass, I hope it works in the CI 🤞

headtr1ck · 2024-07-15T15:05:17Z

BENCHMARKS NOT SIGNIFICANTLY CHANGED.

I didn't check in detail if we have a test that might cover this already and if yes, what the changes are.

spencerkclark · 2024-07-20T15:02:16Z

I don't think we currently do, but I think this new benchmark should cover it: #9262.

spencerkclark

Thanks @antscloud! I think this looks good to me now. Could you add a what's new entry under a new Performance heading?

headtr1ck · 2024-07-20T17:05:20Z

| Change | Before [91a52dc] | After [3512bee] | Ratio | Benchmark (Parameter) |
|----------|----------------------|---------------------|---------|-----------------------------------------------------------|
| - | 139±0.9ms | 42.3±0.7ms | 0.3 | coding.EncodeCFDatetime.time_encode_cf_datetime('noleap') |

3 times faster, not bad

headtr1ck

Looks good to me!

xarray/coding/times.py

headtr1ck · 2024-07-22T09:55:12Z

xarray/coding/times.py

+    dates = np.atleast_1d(dates)
+
+    # Find all the None position
+    none_position = np.equal(dates, None)


mypy complains here.
Not sure if this is an issue of wrong typing of numpy, or if this is not a recommended way of calling equals. (I think the None is the problem).

Co-authored-by: Michael Niklas <[email protected]>

headtr1ck · 2024-10-14T16:34:08Z

The mypy error is still there, not sure how to effectively fix it, maybe use kwargs instead or simply type ignore it.

Otherwise this looks good and should get an entry in whats-new!

xarray/coding/times.py

kmuehlbauer · 2025-05-23T12:54:27Z

Thanks @antscloud! Still a valuable addition, which should be mentioned in whats-new.rst. @antscloud are you still around to finalize this? Otherwise I could take care of that.

xarray/coding/times.py

kmuehlbauer · 2025-05-23T13:12:15Z

Great, either I messed up the merge or this fails now, because of other concurring changes in the past time. @spencerkclark Do you spot anything suspicious here?

spencerkclark · 2025-05-24T13:59:34Z

Thanks @kmuehlbauer for reviving this—I think the test failures are a manifestation of the cftime issue you found earlier (Unidata/cftime#354). The difference here from what was implemented in #9618 is that datetime.datetime objects are no longer converted to cftime.DatetimeProlepticGregorian objects prior to being passed to cftime.date2num.

While the issue was fixed in Unidata/cftime#355, a new version of cftime has not yet been released. We could lobby for that, or raise an error in these circumstances. Independent of the cftime issue, there may be a reasonable case for raising (#10352).

kmuehlbauer · 2025-05-27T13:53:36Z

@spencerkclark Looks like this works after merging #10352. 🎉

spencerkclark

Thanks @antscloud! Sorry this took so long to push through. I went ahead and added a what's new entry on your behalf. Feel free to edit as you see fit, but otherwise I will merge on Thursday.

spencerkclark · 2025-05-27T22:27:09Z

@kmuehlbauer I'm guessing you approve now, but giving you the chance to take another look and remove your requested change (which I think is resolved).

spencerkclark · 2025-05-30T01:16:28Z

The benchmark workflow failure is unrelated, so I am going to go ahead and merge. Thanks again @antscloud!

* main: Fix performance regression in interp from pydata#9881 (pydata#10370) html repr: improve style for dropdown sections (pydata#10354) Grouper tweaks. (pydata#10362) Docs: Add links to getting help mermaid diagram (pydata#10324) Enforce ruff/flynt rules (FLY) (pydata#10375) Add missing AbstractWritableDataStore base methods and arguments (pydata#10343) Improve html repr in dark mode (Jupyterlab + Xarray docs) (pydata#10353) Pin Mypy to 1.15 (pydata#10378) use numpy dtype exposed by zarr array instead of metadata.data_type (pydata#10348) Fix doc typo for caption "Interoperability" (pydata#10374) Implement cftime vectorization as discussed in PR pydata#8322 (pydata#8324) Enforce ruff/flake8-pyi rules (PYI) (pydata#10359) Apply assorted ruff/Pylint rules (PL) / Enforce PLE rules (pydata#10366) (fix): pandas extension array repr for int64[pyarrow] (pydata#10317) Enforce ruff/flake8-implicit-str-concat rules (ISC) (pydata#10368) Enforce ruff/refurb rules (FURB) (pydata#10367) Ignore ruff/Pyflakes rule F401 more precisely (pydata#10369) Apply assorted ruff/flake8-simplify rules (SIM) (pydata#10364) Apply assorted ruff/flake8-pytest-style rules (PT) (pydata#10363) Fix "a array" misspelling (pydata#10365)

github-actions bot added the topic-cftime label Oct 17, 2023

dcherian changed the title ~~Implement vectorization as discussed in PR #8322~~ Implement cftime vectorization as discussed in PR #8322 Oct 17, 2023

Illviljan added the run-benchmark Run the ASV benchmark workflow label Oct 17, 2023

kmuehlbauer requested changes Oct 18, 2023

View reviewed changes

xarray/coding/times.py Outdated Show resolved Hide resolved

xarray/coding/times.py Show resolved Hide resolved

headtr1ck reviewed Oct 23, 2023

View reviewed changes

xarray/coding/times.py Outdated Show resolved Hide resolved

antscloud added 2 commits July 14, 2024 16:43

feat: implement as discussed in PR pydata#8322

3f6bf49

fix: Stick closer to the original function and add PR suggestions

f28fc7d

antscloud force-pushed the vectorization_of_encode_cftime branch from cff06d4 to f28fc7d Compare July 14, 2024 15:27

spencerkclark mentioned this pull request Jul 20, 2024

Add encode_cf_datetime benchmark #9262

Merged

spencerkclark reviewed Jul 20, 2024

View reviewed changes

Merge branch 'main' into vectorization_of_encode_cftime

dc2609c

headtr1ck approved these changes Jul 20, 2024

View reviewed changes

xarray/coding/times.py Outdated Show resolved Hide resolved

headtr1ck reviewed Jul 22, 2024

View reviewed changes

dcherian and others added 2 commits August 2, 2024 09:29

Update xarray/coding/times.py

06af81a

Co-authored-by: Michael Niklas <[email protected]>

Merge branch 'main' into vectorization_of_encode_cftime

05428fd

dcherian reviewed Oct 22, 2024

View reviewed changes

xarray/coding/times.py Outdated Show resolved Hide resolved

dcherian and others added 3 commits October 22, 2024 14:28

Update xarray/coding/times.py

80bb5fe

Merge branch 'main' into vectorization_of_encode_cftime

f4828a2

Merge branch 'main' into vectorization_of_encode_cftime

8be73cf

github-actions bot added the topic-documentation label May 23, 2025

kmuehlbauer reviewed May 23, 2025

View reviewed changes

xarray/coding/times.py Outdated Show resolved Hide resolved

Update xarray/coding/times.py

042638d

spencerkclark mentioned this pull request May 24, 2025

Disallow standard calendar np.datetime64 encoding prior to reform #10352

Merged

2 tasks

Merge branch 'main' into vectorization_of_encode_cftime

147ea67

dcherian requested a review from spencerkclark May 27, 2025 17:34

spencerkclark added 2 commits May 27, 2025 18:14

Add what's new entry

c83f76a

Merge branch 'main' into vectorization_of_encode_cftime

6d4dc3b

spencerkclark approved these changes May 27, 2025

View reviewed changes

spencerkclark requested a review from kmuehlbauer May 27, 2025 22:23

kmuehlbauer approved these changes May 28, 2025

View reviewed changes

spencerkclark added the plan to merge Final call for comments label May 28, 2025

Merge branch 'main' into vectorization_of_encode_cftime

c37f1f1

spencerkclark merged commit 8b961e4 into pydata:main May 30, 2025
31 of 32 checks passed

	@requires_cftime
	@pytest.mark.parametrize("freq", ["U", "L", "S", "T", "H", "D"])
	def test_encode_decode_roundtrip_cftime(freq) -> None:
	initial_time = cftime_range("0001", periods=1)
	times = initial_time.append(
	cftime_range("0001", periods=2, freq=freq) + timedelta(days=291000 * 365)
	)
	variable = Variable(["time"], times)
	encoded = conventions.encode_cf_variable(variable)
	decoded = conventions.decode_cf_variable("time", encoded, use_cftime=True)
	assert_equal(variable, decoded)

Uh oh!

Implement cftime vectorization as discussed in PR #8322 #8324

Implement cftime vectorization as discussed in PR #8322 #8324

Uh oh!

Conversation

antscloud commented Oct 17, 2023

Uh oh!

kmuehlbauer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

headtr1ck commented Jul 11, 2024

Uh oh!

antscloud commented Jul 14, 2024

Uh oh!

headtr1ck commented Jul 15, 2024

Uh oh!

spencerkclark commented Jul 20, 2024

Uh oh!

spencerkclark left a comment

Choose a reason for hiding this comment

Uh oh!

headtr1ck commented Jul 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

headtr1ck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

headtr1ck Jul 22, 2024

Choose a reason for hiding this comment

Uh oh!

headtr1ck commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kmuehlbauer commented May 23, 2025

Uh oh!

Uh oh!

kmuehlbauer commented May 23, 2025

Uh oh!

spencerkclark commented May 24, 2025

Uh oh!

kmuehlbauer commented May 27, 2025

Uh oh!

spencerkclark left a comment

Choose a reason for hiding this comment

Uh oh!

spencerkclark commented May 27, 2025

Uh oh!

spencerkclark commented May 30, 2025

Uh oh!

Uh oh!

Uh oh!

headtr1ck commented Jul 20, 2024 •

edited

Loading

headtr1ck commented Oct 14, 2024 •

edited

Loading