BUG: GroupBy.transform() unexpectedly coerces columns to object dtype when there is another object dtype column

[willsthompson]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

df = pd.DataFrame({
    'x': ['a', 'b', 'c', 'd'],
    'y': [5, 6, 7, 8],
    'g': [1, 2, 3, 3]
})

def myfirst(c):
    return c.iloc[0]

# I'd expect the y col dtype to remain int64 in all examples below, but it changes to object 
# when using myfirst() with transform() and there is at least one object column in the result.

df.groupby('g').transform('first').dtypes  # int64 dtype when using builtin
df.groupby('g').transform(myfirst).dtypes  # object dtype when using function
df.groupby('g').agg(myfirst).dtypes        # int64 dtype when using function with agg()
df.drop(columns=['x']).groupby('g').transform(myfirst).dtypes  # int64 dtype if object column is removed
df.groupby('g').transform(lambda c: type(myfirst(c)))['y'] == np.int64  # Confirm myfirst() is returning int64 scalars

Problem description

The current transform() behavior is inconsistent with the behavior of other reducers and with agg(). Those fall in like with my expectation to not change datatypes unless the output would otherwise prevent it (i.e. a reducer adds NaNs). However, here:

df.groupby('g').transform(myfirst).dtypes

the int64 y column is coerced into an object dtype:

x    object
y    object
dtype: object

The coercion also occurs when the y column is float or bool, but not when it is datetime64[ns].

Expected Output

x    object
y     int64
dtype: object

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.8.7.final.0
python-bits : 64
OS : Darwin
OS-release : 20.4.0
Version : Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; root:xnu-7195.101.2~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8
pandas : 1.2.4
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
pip : 20.3.4
setuptools : 49.2.1
Cython : 0.29.14
pytest : 6.2.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.3
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 2.0.0
pyxlsb : None
s3fs : None
scipy : 1.6.1
sqlalchemy : 1.3.23
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
numba : None

[phofl]

Your specific example is not a transform bug. You are getting a DataFrame into your function myfirst, where iloc casts to a Series, causing the dtype conversion. But the following causes the same behavior, which is a bug:

def myfirst(c):
    if isinstance(c, Series):
        return c.iloc[0]
    return c.iloc[[0]]

[willsthompson]

@phofl Thanks, I guess I don't quite understand these APIs. My understanding was that transform() must operate on columns, so its function always receives a Series, whereas with agg() it depends. For example, if these are true:

assert (df.groupby('g').transform(lambda c: type(myfirst(c)))['y'] == np.int64).all()
assert (df.groupby('g').agg(lambda c: type(myfirst(c)))['y'] == np.int64).all()

Can't I assume the function received a Series, since iloc[0] returned an atomic type?