ENH: Add support for Categoricals in BlockManager

[jtratner]

tl;dr - add true support for Categoricals in NDFrame.

There was an issue on the mailing list about using cut and sorting the results that brought this to mind. The issue is both that (I believe) a categorical loses its representation when you put it in a DataFrame and so the output of cut has to just be strings. I propose the following:

1. Add a CategoricalBlock (or FactorBlock) internally that can handle categoricals like those produced from cut that could share most of MI's internals, as a 2D int ndarray with an associated list of indexes for each column (again, nearly the same as MI except most ops would be working on just one 'level' and underlying could/would be 2D rather than list of Int64Index). Probably also would mean abstracting common operations to a separate mixin class.
2. Change Categorical to be a Series subclass with a SingleBlockManager that's a CategoricalBlock. This would not change its API, but it would gain Series methods.
3. Add a to_categorical method to Series (bonus points if we change convert_objects to detect if there are < Some_Max number of labels and convert object dtypes to categoricals).
4. Add a registration method to make_block so it iterates over a set of functions that either return a klass or None before falling back to ObjectBlock (so abstract existing else clause into a function and make the list of functions semi-public).

I'm going to work on this and I don't think it will be that difficult to implement, but it would make pandas more useful for representing level sets and other normalized data.

[jreback]

+1

4. just have to be careful since this is done a lot that don't sacrifice performance here on the type inference (currently it's setup for most common types first and fast path skips this anyhow)

[jtratner]

Right - could even keep existing else and then iterate over additional
functions if they are there.

[jreback]

obviously related to #4551 which is essentially the common op mixin - I may work on this

[jtratner]

Yeah, definitely similarities. I'm most interested in getting an efficient internal representation set up (and maybe changing MI to use the same setup of 2D int ndarray + list of indexes rather than list of int ndarray + list of indexes) and then building out. The nice thing is that, if you know that labelsare sorted, it's trivial to get min and max, plus value_counts are the same as for int block with a mini wrapper around result.

[jtratner]

@jseabold - anything from the statsmodels side on this? Not sure if there's anything on statsmodels wishlist for Categorical that we should keep in mind (I think you're one of the right people to ask).

[jtratner]

Heck, MI's from_arrays already uses Categorical anyways, so there's already quite a bit of overlap.

[jseabold]

This will be a welcome feature for us. We'll work to make any changes we need to support it, though I don't suspect we'll need to do anything.

[jtratner]

Yeah I wouldn't expect anything.

[jseabold]

The status quo won't change, but we will be able to catch categoricals and handle them specially now. This is only available through formulas now and if we get a DataFrame we more or less do np.asarray(df). I'll file a ticket for this.

Closed #6219 in favor of this, just noting here the large memory hit associated
with not storing catagorical string columns as a factored data structure.
Just one more reason for doing it.

[jankatins]

Closed #6242 in favor of this issue.

#6242 has some "what should work with categorials" (in the form of pseudo test code): #6242 (comment) and #6242 (comment)

"[regarding prind(df) behaviour] I would think it look the same as if you would print out Series(factor(...)), only the dtypes and so would show that it is a factor."