apply on np.datetime col doesn't upconvert to Timestamp

[dalejung]

import pandas as pd

s = pd.Series(pd.date_range(start="2000/1/1", freq="D", periods=10))
s.apply(lambda x: x.date())

# AttributeError: 'numpy.datetime64' object has no attribute 'date'

This is an error from some unit tests. Did something major change with how datetimes are handled? I recall that np.datetime64 was autoboxed into Timestamps.

I'll update after poking around.

[dalejung]

Ah, I'm a liar. The unit test was more like

ind = pd.date_range(start="2000/1/1", freq="D", periods=10)
vals = [pd.Timestamp(str(d)) for d in ind]
df = pd.DataFrame(index=ind)
df['dates'] = vals
df.dates.apply(lambda x: x.date())

Which worked on 0.10.0 since it kept the objects around.

[dalejung]

e06334a

Found the commit. Makes sense, I'll change stuff on my end. Perhaps this should be an enhancement?

[jreback]

yes should be a mention in release notes
(it's more of a partial bug fix actually)
not sure that it always converts to time stamps, just more often
eg see applymap in frame.py (had to force conversion here, not sure of the promises that pandas makes in regards to whether the object is a time stamp (or a np.datetime64)
prob a number cases that are not well tested

[dalejung]

Yeah. It broke a couple things on my end. There should probably be more emphasis on the change since it breaks things in indirect ways.

[jreback]

do u have an example of something that works on 0.10.0 and breaks in master?

[dalejung]

see my first comment. Anything that assumes the DataFrame columns will stay objects and thus have object methods available.

[jreback]

your example fails in 0.10.0 as well, did this work in a previous version?

(as a side note, your example is a series; frames I believe work correctly)

interestingly this works:

for x in s:
    s.date()

as s is the correct type 'M8[ns]', and the iteration does on-the-fly conversion to Timestamp...simple to fix apply to do that (as was done with applymap in Frame)....but bigger question is that this conversion could be done in the constructor. maybe performance related? and/or since we are talking about a column of data (and not the index), its very hard to explicty infer the correct type (and you also want to allow somewhat arbitrary types in the container as well). The user I think should force the conversion (which is the way it is now)

@wesm any thoughts here?

[dalejung]

I think you misread my comment. The original post never worked. At the time, I hadn't realized that the issue was the object -> datetime64 conversion. Without the conversion, auto-boxing is not necessary.

I was referring to the DataFrame example which does work in 0.10.0. Also, creating a Series out of a list of object doesn't convert to datetime64 in dev. So there's a lack of consistency there.

I'm ambivalent to whether automatic object -> datetime64 is correct. It's more of a backwards compatibility issue.
#2624 also came up for me due to the change.

[wesm]

fixed this thanks