ER: compound dtypes - DataFrame constructor/astype

[mamikonyan]

xref #9133, maybe allow a dict of dtypes to be passed as well

I trying to use the dtype argument in the DataFrame constructor to set the types of several columns, and I'm getting incorrect types. Everything works well, however, when the dtypes come from the recarray itself.

In [61]:
data = [(1,1.2), (2,2.3)]
dtype = [('a','i4'),('b','f4')]
a = np.array(data, dtype=dtype)
pd.DataFrame(a).dtypes

Out [61]:
a      int32
b    float32
dtype: object

But if I use the dtype constructor argument, I get incorrect types:

In [65]:
pd.DataFrame(data, dtype=dtype).dtypes
Out [65]:
0    object
1    object
dtype: object

The astype() member function doesn't work either:

In [75]:
pd.DataFrame(data).astype(dtype)

Truncated Traceback (Use C-c C-x to view full TB):
c:\Anaconda\lib\site-packages\pandas\core\common.pyc in take_nd(arr, indexer,  axis, out, fill_value, mask_info, allow_fill)
    491         indexer = _ensure_int64(indexer)
    492         if not allow_fill:
--> 493             dtype, fill_value = arr.dtype, arr.dtype.type()
    494             mask_info = None, False
    495         else:

TypeError: function takes exactly 1 argument (0 given)

I'm using Pandas 0.11.0 from Anaconda.
Thanks in advance.

[cpcloud]

this will be fixed, but is there a case where you need to pass the dtype? I suppose if you have the array and the dtype separate then you might want to just pass them in, but if you have that you can construct the recarray and then pass to DataFrame...

[jreback]

actually, this is not currently implemented

[jreback]

this should raise now, marking as a bug for that (it expectes a single dtype, not a compound one)

[cpcloud]

@jreback so it should raise? what's wrong with accepting a compound dtype?

[jreback]

@cpcloud nothing wrong with accepting it, but its NotImplementedError (until it is)

[mamikonyan]

OK, thanks, guys.

[jreback]

@mamikony going to reopen...thanks for noticing this....don't really have any test for this

[mamikonyan]

So, what do you think about the second issue of astype() throwing?

[jreback]

same issue, its setup to deal with a single dtype (not compound). the purpose is to coerce your data. What is your goal here?

[mamikonyan]

I had a series where the elements were compound strings,

In [130]:
pd.Series(['A:1:3.14'])

Out [130]:
0    A:1:3.14
dtype: object

So I wanted to split them into a data frame with appropriate types, but I got this garbage:

In [131]:
pd.DataFrame(_.map(lambda s: s.split(':')).tolist(), dtype=[('a','S1'),('b','i4'),('c','f4')])
Out [131]:
             0            1                  2
0  (A, 0, 0.0)  (1, 0, 0.0)  (3, 3420462, 0.0)

I'll just have to do it column by column.

[jreback]

The split creates a series with elements that are lists
The apply creates a frame from this
convert_objects with the convert_numeric=True coerces strings to numbers (if it can)

In [1]: s = pd.Series(['A:1:3.14'])

In [7]: s.str.split(':').apply(Series).convert_objects(convert_numeric=True)
Out[7]: 
   0  1     2
0  A  1  3.14

In [8]: s.str.split(':').apply(Series).convert_objects(convert_numeric=True).dtypes
Out[8]: 
0     object
1      int64
2    float64
dtype: object

[mamikonyan]

Fair enough, thanks. You can't set the dtype, but at least it guesses correctly. I need precise control of the dtype because I write it out to an HDF5 file (with Pytables and df.to_records()) and I'd like to have the proper dtype at the start. Also, I didn't know about the str namespace. Thanks.

[jreback]

@mamikony

you can certainly change it if you like (just do it one-by-one)

are you not using HDFStore? (which uses PyTables under the hood)? (and will be significantly faster than using to_records, btw), not to mention that it save indices and almost all dtypes. see here: http://pandas.pydata.org/pandas-docs/dev/io.html#hdf5-pytables