.at/.iat do not change the values to a Decimal object .loc/.iloc does

[derekwong9]

import pandas as pd
import numpy as np
from decimal import Decimal
np.random.seed(123)

df = pd.DataFrame(np.random.randint(0,5,size=(5, 4)), columns=list('ABCD'))
df.values

array([[2, 4, 2, 1],
[3, 2, 3, 1],
[1, 0, 1, 1],
[0, 0, 1, 3],
[4, 0, 0, 4]])

for col in df.columns:
    for row in df.index:
        df.at[row, col] = Decimal(str(df.at[row, col]))

df.values

array([[2, 4, 2, 1],
[3, 2, 3, 1],
[1, 0, 1, 1],
[0, 0, 1, 3],
[4, 0, 0, 4]])

for col in df.columns:
    for row in df.index:
        df.loc[row, col] = Decimal(str(df.loc[row, col]))

df.values

array([[Decimal('2'), Decimal('4'), Decimal('2'), Decimal('1')],
[Decimal('3'), Decimal('2'), Decimal('3'), Decimal('1')],
[Decimal('1'), Decimal('0'), Decimal('1'), Decimal('1')],
[Decimal('0'), Decimal('0'), Decimal('1'), Decimal('3')],
[Decimal('4'), Decimal('0'), Decimal('0'), Decimal('4')]],
dtype=object)

Problem description

I am using the decimal.Decimal package for financial calculations and to prevent float rounding errors. .astype(Decimal) does not work as it is not a numpy dtype nor a python dtype. So my only option is to loop through a large DataFrame and assign the Decimal dtype. As I am only get/set a single value and modifying it, I was trying to use .at/.iat however when I run this it does not actually change the datatype. However when I run the same loop using the less efficient .loc/.iloc it works. I have no idea why. My expected output is the array of values changed to a Decimal object.

Expected Output

array([[Decimal('2'), Decimal('4'), Decimal('2'), Decimal('1')],
       [Decimal('3'), Decimal('2'), Decimal('3'), Decimal('1')],
       [Decimal('1'), Decimal('0'), Decimal('1'), Decimal('1')],
       [Decimal('0'), Decimal('0'), Decimal('1'), Decimal('3')],
       [Decimal('4'), Decimal('0'), Decimal('0'), Decimal('4')]],
      dtype=object)

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.4
pytest: 3.7.1
pip: 18.0
setuptools: 40.2.0
Cython: 0.28.5
numpy: 1.15.0
scipy: 1.1.0
pyarrow: 0.9.0
xarray: None
IPython: 6.5.0
sphinx: 1.7.6
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.7
feather: 0.4.0
matplotlib: 2.2.3
openpyxl: 2.5.5
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.5
lxml: 4.2.4
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.10
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[TomAugspurger]

I don't think we're likely to try to support this directly.

I'd recommend making an extension type for Decimal data. The prototype in our test suite at https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/decimal/array.py may be a good starting point (or not).

[derekwong9]

No need to support directly, I am just wondering why .at/.iat does not alter the value, but loc/iloc does. since I am only doing single value get/set it should be faster with at/iat.

…

On Tue, Sep 18, 2018, 19:38 Tom Augspurger ***@***.***> wrote: I don't think we're likely to try to support this directly. I'd recommend making an extension type <http://pandas-docs.github.io/pandas-docs-travis/extending.html#extensiondtype> for Decimal data. The prototype in our test suite at https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/decimal/array.py may be a good starting point (or not). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#22740 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF595B2odT2_KpGgsDLJx0NtmbSr3q3iks5ucNszgaJpZM4Wtm27> .

[TomAugspurger]

I'm not sure; you're welcome to take a look.

…

On Tue, Sep 18, 2018 at 9:21 AM Derek Wong ***@***.***> wrote: No need to support directly, I am just wondering why .at/.iat does not alter the value, but loc/iloc does. since I am only doing single value get/set it should be faster with at/iat. On Tue, Sep 18, 2018, 19:38 Tom Augspurger ***@***.***> wrote: > I don't think we're likely to try to support this directly. > > I'd recommend making an extension type > < http://pandas-docs.github.io/pandas-docs-travis/extending.html#extensiondtype > > for Decimal data. The prototype in our test suite at > https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/decimal/array.py > may be a good starting point (or not). > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > < #22740 (comment)>, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AF595B2odT2_KpGgsDLJx0NtmbSr3q3iks5ucNszgaJpZM4Wtm27 > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#22740 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIgsgmvgaz7_0FcqzrJgqiAMeOgq4ks5ucQFOgaJpZM4Wtm27> .

[jbrockmendel]

With .loc we check blk._can_hold_element to see if the Decimal obj can be inserted without casting. _can_hold_element returns False here, so .loc casts. .at doesn't check _can_hold_element, just does a try/except (i think validate_numeric_casting) may be involved. The try/except doesn't raise so it sets inplace.

So it's not great that a) .at uses different logic then .loc and b) _can_hold_element and validate_numeric_casting dont match here