Skip to content

Problem mixing datetime64 with other types in DataFrame column #9336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
eoincondron opened this issue Jan 22, 2015 · 2 comments · Fixed by #20550
Closed

Problem mixing datetime64 with other types in DataFrame column #9336

eoincondron opened this issue Jan 22, 2015 · 2 comments · Fixed by #20550
Labels
Error Reporting Incorrect or improved errors from pandas good first issue Indexing Related to indexing on series/frames, not to indexes themselves
Milestone

Comments

@eoincondron
Copy link

Given a DataFrame with a column of type datetime64, there is a problem assigning values of other types to slice of said column:
in [148]:

import datetime as dt
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.random((3, 2)), columns = ['a', 'b' ])
df['c'] = pd.NaT
df.c

Out[148]:
0   NaT
1   NaT
2   NaT
Name: c, dtype: datetime64[ns]

It doesn't mind this:

In[185]:df.loc[0, 'c'] = pd.datetime.today()
df.c
Out[185]: 
0   2015-01-22 10:56:49.763999
1                          NaT
2                          NaT

Now try assigning values of a another type:

In [186]: df.loc[0, 'c'] = 1
AttributeError: 'int' object has no attribute 'view'

In[187]: df.loc[[0, 1], 'c'] = 'a', 'b'
AttributeError: 'tuple' object has no attribute 'view'

Now, the problem runs deeper than this. I didn't actually come across it when trying to assign mixed types to a single column. I was actually assigning values to a block of a DataFrame which included the time column. Here is an example:

In [252]: 
A = pd.DataFrame([[2, pd.datetime(2013, 1, 1)], [3, pd.datetime(2014, 1, 1)]])

In[253]: 
df.loc[[0, 1], ['a', 'c']] = A.values
AttributeError: 'list' object has no attribute 'view'

Note that the following works:

 In[254]: 
 df.loc[[0, 1], ['a', 'c']] = A

So when using the array A.values, the dtype is forced to object which then causes the problem. However, in the case where A has a different index than df, using A.values is necessary to assign the values. Maybe I need to be more careful but the error message doesn't help and it took me a long time to understand the problem. If I try to assign values to a slice of column 'c' using the corresponding slice of A.values I get a different error, which is more informative:

In[255]: 
df.loc[[0, 1],  'c'] = A.values[:, 1]
TypeError: Cannot change data-type for object array.
@jreback
Copy link
Contributor

jreback commented Jan 22, 2015

these could certainly have a trapped (and then better message).

Usually we allow things like this but coerce to object. In general though I think raising is better.
If you really really want say a mixed-dtype Series then one should construct it directly.

Welcome a pull-request to update the errors.

@jreback jreback added Error Reporting Incorrect or improved errors from pandas Indexing Related to indexing on series/frames, not to indexes themselves labels Jan 22, 2015
@jreback jreback added this to the 0.16.0 milestone Jan 22, 2015
@jreback jreback modified the milestones: 0.16.0, Next Major Release Mar 5, 2015
@jreback
Copy link
Contributor

jreback commented Aug 6, 2017

this is fixed in master if anyone would like to put up some tests.

@jreback jreback modified the milestones: 0.21.0, Next Major Release Aug 6, 2017
@jreback jreback modified the milestones: 0.21.0, Next Major Release Sep 23, 2017
@jreback jreback modified the milestones: Next Major Release, 0.23.0 Mar 30, 2018
@jreback jreback modified the milestones: 0.23.0, Next Major Release Apr 14, 2018
@jreback jreback modified the milestones: Next Major Release, 0.24.0 May 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Error Reporting Incorrect or improved errors from pandas good first issue Indexing Related to indexing on series/frames, not to indexes themselves
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants