Converting to DataFrames.jl, performance

[FrancisH-C]

On a large dataset, converting from Pandas.DataFrame to DataFrames.DataFrame with constructor takes a lot of time. Not sure if it is Pandas.jl job to ensure that but I figure I should post it anyway. Working exemple in julia 1.0 :

# panda_df is a large Pandas.DataFrame
# This takes  ~ 1h
DataFrames.DataFrame(panda_df)

# This takes ~5 min (probably not the optimal solution)
names=Symbol.(Pandas.values.(Pandas.columns(panda_df)))
df=DataFrames.DataFrame()   
for i in 1:length(names)
    df[names[i]]=Array(panda_df[names[i]])
end

[malmaud]

@davidanthoff, any insight?

[malmaud]

I'm not actually sure if this involves the queryverse, though.

[davidanthoff]

This is probably because the code path that is used now iterates everything as rows during the conversion, and that is probably not the fastest option. One simple way to address this is to add support for the get_columns_copy_using_missing interface from TableTraits.jl. The only thing that would need to be done is add:

TableTraits.supports_get_columns_copy_using_missing(x::Pandas.DataFrame) = true

function TableTraits.get_columns_copy_using_missing(x::Pandas.DataFrame)
  # return a named tuple of columns here
  # A naive, untested and probably not working version might be
  names=tuple(Symbol.(Pandas.values.(Pandas.columns(x))))
  cols = (Array(x[names[i]]) for i in 1:length(cols))
  return NamedTuple{names}(tuple(cols...))
end

I'm sure I messed up the construction of the named tuple (I can never remember the syntax), but I think you probably get the idea.

Two important aspects here: the arrays returned that way need to be copies, i.e. Pandas.DataFrame should not hold on to them. And you should use Union{T,Missing} arrays if there is missing data somewhere.

DataFrames.jl supports this interface, so if those two methods were added here in Pandas.jl, then the DataFrame constructor would automatically use them.

[norci]

I have a easier solution:
use Pandas.to_csv, save the data to disk, then use CSV.read to load the data.

[FrancisH-C]

For my use case, it takes too much time to save data to disk since I'm importing my (big) files compressed.

As a general idea, it should work I guess.