Skip to content

BUG: pd.options.future.no_silent_downcasting is not backward compatible #59845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 tasks done
vityas opened this issue Sep 20, 2024 · 1 comment
Open
3 tasks done
Labels
Bug Needs Info Clarification about behavior needed to assess issue PDEP6-related related to PDEP6 (not upcasting during setitem-like Series operations)

Comments

@vityas
Copy link

vityas commented Sep 20, 2024

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

with pd.option_context('future.no_silent_downcasting', True):
  pd.Series([None]).fillna(False)

Issue Description

I was not able to find a code that doesn't throw warnings for pandas >= 2.0. The example will work in pandas 2.2, but will break for earlier versions. Ahem... This is probably the first time in my Python development that a backwards incompatible change has been issued with a backwards incompatible workaround.

Expected Behavior

Not throw any warning.

i could do something like this

if pandas.__version__ >= '2.2.0':
  with pd.option_context('future.no_silent_downcasting', True):
    pd.Series([None]).fillna(False)
else:
  pd.Series([None]).fillna(False)

I suppose this code would work, but it's a bad code.

Installed Versions

pandas : 2.2.2
numpy : 1.26.3
pytz : 2024.1
dateutil : 2.8.1

@vityas vityas added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 20, 2024
@rhshadrach
Copy link
Member

Thanks for the report.

This is probably the first time in my Python development that a backwards incompatible change has been issued with a backwards incompatible workaround.

future.no_silent_downcasting was introduced so that users could adopt the future behavior of the breaking change, not to be a backwards compatible way to avoid the warning.

I suppose this code would work, but it's a bad code.

Why is it bad code? If you'd like to support multiple versions of pandas, I think you'll need to branch on the version. Though it is probably better practice to access the version via from importlib.metadata import version and not rely on string comparisons (e.g. https://packaging.pypa.io/en/latest/version.html).

@rhshadrach rhshadrach added Needs Info Clarification about behavior needed to assess issue PDEP6-related related to PDEP6 (not upcasting during setitem-like Series operations) and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 23, 2024
bikegeek added a commit to dtcenter/METdataio that referenced this issue Mar 13, 2025
bikegeek added a commit to dtcenter/METdataio that referenced this issue Mar 21, 2025
* Update unit_tests.yaml

change python version from 3.10 to 3.12, using existing requirements.txt

* Update requirements.txt

update versions that were installed for Python 3.12 conda environment

* Update unit_tests.yaml

check out the feature_update_python branch of METcalcpy to incorporate the  Python 3.12 changes

* Update requirements.txt

pyyaml should be 6.0.2

* Updating to Python 3.12 use "assert mock_foo.called" rather than "assert mock_foo.called_once_with(...)"
Refer to this: python/cpython#100690

* Update numpy to 2.2.3 to be consistent with METcalcpy and METplotpy

* Update python version

* Change ?=3.12 to =3.12 for Python version

* set minimum python version to 3.11

* Revert back to mimimum python of 3.10.4

* fix missing "

* update escape sequence in lines 432 and 436 to be compatible with Python 3.12

* in pandas to_datetime, replace errors='ignore' to errors='raise' to address the FutureWarning

* Addres FutureWarning: Downcasting behavior in the replace method.  Opt-in to the future behavior and explicitly as specified in StackOverflow: https://stackoverflow.com/questions/79226735/pandas-replace-and-downcasting-deprecation-since-version-2-2-0

* update package versions to be consistent with those specified in the requirements.txt file

* In addressing the FutureWarning, inadvertantly left the request for dtype in command.

* use the feature_update_python branch instead of the develop branch to use the correct package versions

* Explicitly cast the fill value to an int to satisfy the pandas 2.2 expectation of an int dtype

* convert the OBS_LEAD andt FCST_LEAD series to int64 to make all dtypes consistent

* set dtypes to OBS_LEAD and FCST_LEAD columns to facilitate merging

* set dtype of the FCST_LEAD and OBS_LEAD series earlier, to fix error when using pandas merge

* Revert to original code

* Revert the OBS_LEAD column to match original code.  Replicate same method for setting the FCST_LEAD column to int

* set the FCST_LEAD and OBS_LEAD columns as strings, as the fill values in the constants.py file are supposed to be strings

* Explicitly change the type of the fcst_lead and obs_lead columns prior to merging to address the FutureWarning

* Address the ValueError when merging the OBS_LEAD columns between two different dataframes where one type is an int vs object

* After merge, change the OBS_LEAD column to string values to address failing tests

* move dtype reassignment outside of try-except code and catch only exceptions from merge validation issues

* remove try-except and include changing the dtypes for the fcst_lead in both dataframes so they are consistent (and avoiding the ValueError when merging on different dtypes)

* fix indentation error

* fix syntax for verifying merge operation

* Remove the validate option on merging two dataframes

* Explicitly set the dtypes in fcst_column and obs_column for remaining merge operations

* Update requirements.txt

* Update requirements.txt

* Update requirements.txt

* Update nco_requirements.txt

updates to package versions for Python 3.12 migration

* Package versions for Python 3.10

* For testing current code for backwards compatibility with Python 3.10

* Update the names of the runs to facilitate monitoring of GHA's

* Remove the python 3.10 run to isolate issues from running tests

* Updates for Python 3.12

* Python 3.10 version for sonarqube

* Unused/ignored, removing this.

* Testing for Python 3.10

* Update unit_tests.yaml

updated the METcalcpy version from feature_update_python to the develop branch

* Update py310_unit_tests.yaml

update METcalcpy branch to develop

* Update benchmark_pandas2_2.yaml

update METcalcpy branch to develop

* Address another deprecation with m_2d_data.obs_valid

* Removing testing for Python 3.10 due to lack of backwards compatibility in Pandas: pandas-dev/pandas#59845

* Updated for Python 3.12

* Update .readthedocs.yaml

update for Python 3.12

* Update sonarqube.yaml

update METcalcpy branch to the develop branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Info Clarification about behavior needed to assess issue PDEP6-related related to PDEP6 (not upcasting during setitem-like Series operations)
Projects
None yet
Development

No branches or pull requests

2 participants